Shoutout to Paul Harris @rolfyone and Luca @eth2353 for their comments and review.
Tracking of CL/EL client usage is critical for understanding network resilience but there’s no reliable reporting solution.
Prior Efforts
Fingerprinting: Blockprint monitored client block proposal styles to guess which client is being used but this became inaccurate following the Ethereum Electra upgrade and has since been deprecated. [1], [2]
Crawling: MigaLabs monitors nodes on the network but this is not exclusive to validator nodes and has peer bias. [3]
Relay Data: Relays have client data but ePBS removes relays (would also need to convince them to share the data to be aggregated). [4]
Graffiti Watermark: Uses proposal graffiti field to report CL/EL client but implementation isn’t standardized, relies on there being enough room if a custom graffiti is set, and doesn’t account for multinode/DVT setups. [5], [6]
Prior Research
This reasearch post goes over the following topics and implementation challenges:
Making client data visible in proposed blocks
Improving the reliability of crawling methods
Dedicated sub-protocol for client data collection
Dedicated private voting scheme
Proposal: New 32 byte Proposal Field
A new 32 byte beacon block body data field inspired by graffiti watermarks, but as a dedicated field and with an expanded schema accounting for multinode setups like Vero, Vouch, and DVT. Would also be beneficial for monitoring zkCL proof diversity in the future.
Terminology
Field Version - Versioning for this new data field.
Setup - The type of setup being used, such as a regular simple setup, obol, ssv, vero, vouch, etc.
Threshold - The M-of-N of connected clients required to agree before casting their vote.
Client Pairs - The number of sets of client combinations. For example if running Prysm+Nethermind and Teku+Besu that is 2 client pairs.
Encoding
byte 0 [7-0] = Field version (max 2^8 = 256)
byte 1 [7-3] = Setup (max 2^5 = 32)
byte 1 [2-0] = Threshold (max 2^3 = 8)
bytes 2-15 = 1 byte per CL/EL client pair (14 pairs, max 2^4 = 16 options each)
bytes 16-31 = unused for initial implementation but allows for additional tracking in the future
In the future when zkCL proofs are introduced we can bump the client data version and use the additional bytes to gather that diversity data as well.
A few other variations were considered before narrowing down to this. You can check out this gist if you’re interested.
Reference Lists
Reference lists will be needed for the setups and client pairs that maps numerical values to the setup, CL, and EL options which will be used for byte encoding/decoding.
A value of 0 and 1 can be reserved for “undefined” and “other”.
Example
Assumptions:
Field Version: 1
Setup: Vero
Setup Reference List Mappings:
Undefined: 0
Other: 1
Simple: 2
Obol: 3
SSV: 4
Vero: 5
Vouch: 6
Threshold: 3/4
Client Pairs: TK+NM,LS+GE,NB+NM,LH+BU
CL Reference List Mappings:
Undefined: 0
Other: 1
GR: 2
LH: 3
LS: 4
NB: 5
TK: 6
PM: 7
EL Reference List Mappings:
Undefined: 0
Other: 1
BU: 2
EJ: 3
EG: 4
EX: 5
GE: 6
NB: 7
NM: 8
RH: 9
Byte Encoding:
Byte 0 = 00000001 (version)
Byte 1 = 00101011 (setup 00101 + threshold 011)
Byte 2 = 01101000 (TK 0110 + NM 1000)
Byte 3 = 01000110 (LS 0100 + GE 0110)
Byte 4 = 01011000 (NB 0101 + NM 1000)
Byte 5 = 00110010 (LH 0011 + BU 0010)
Byte 6-15 = 00000000 (undefined client pairs)
Bytes 16-31 = 00000000 (unused)
Configurations
Default to on
Allow user to disable client reporting
Are there any arguments for having the ability to opt out of EL, CL, and zkCL proof reporting separately instead of one setting that controls all?
Allow user to disable client setup
Concerns/Shortcomings
Privacy
Some may be concerned about privacy but there’s no client version info listed, there’s the option to disable it, and clients already add client info this doesn’t seem like a real issue.
With multinode clients like Vero, the risk of knowing the client you’re currently running is greatly reduced.
Data Spoofing - While it’s possible to spoof the data, given the large validator set I believe this will still be better than what we’re currently working with.
Stale Data - This implementation relies on proposals so data doesn’t provide a live view of the network (currently would take at least ~129 days to rotate through the validator set). Granted this may continue to decrease with consolidations since it was ~131 days just a couple weeks prior to writing this.
Discussion
Would love to hear any other ideas for simple ways to implement client reporting.
While this proposal may not be the best solution technically, I believe the ease of implementation and flexibility makes this worth it. Let’s gain the needed insight into our network.
If no other ideas or strong concerns arise, I would like to move forward with an EIP proposal. Would need a technical partner for this so would love to hear if anybody if interested in helping out.
Just to clarify this point, because even today there is not one single way of requiring multiple clients to agree in multi-node setups. The bare minimum requirement here is for multiple clients to agree on the attestation source checkpoint. Ideally though, agreement would already/also be required on the target checkpoint – that helps other network participants not to cast wrong source votes later on.
For implementers, the new /eth/v2/node/version Beacon API endpoint (Gloas+) provides all the necessary data (CL+EL client information) to validator clients.
I’m happy to continue helping out with this on a technical level.
Deadline for proposing EIPs to Hegotá upgrade is August 6, so need to do an EIP & get it proposed before then, otherwise will have to wait for I* upgrade. Happy to help on EIP basics (I am an associate editor, I help assign numbers).
A shorter term alternative would be to work with client teams to have the client data added to graffiti by default. This wouldn’t require a network upgrade, though it could take a while for stakers to update.
Do we really need to need this information in-protocol which adds complexity over the long run? As far as I can tell, it still relies on honest reporting. Even if a majority of validator nodes are honest and willing to report the data, the end result will still have inaccuracy. Is there no way of improving existing out-of-protocol solutions?
The amount of complexity the proposed approach adds is very low. I put together a PoC implementation in 3 CL clients, a validator client and the Dora explorer over the course of a single afternoon. The new beacon block field is not in any way consensus critical, and is very similar to the graffiti field which has been (ab)used for this purpose recently. Unless you compare this to submitting data to a centralized API, I don’t see how this could get less complex.
The proposed solution does rely on honest reporting but there is no incentive to be dishonest. The more validators report the data, and the more of them are honest, the better our insights into the network will be.
Is there no way of improving existing out-of-protocol solutions?
There is no existing solution that measures this data reliably, in a way that would let us identify how safe Ethereum is from consensus bugs like the one we have seen on Holesky that led to its demise. Nowadays, we’re practically flying blind when it comes to client usage across the network and how exposed the network is to consensus bugs.
The graffiti approach could possibly be improved upon. However, that field was not originally intended for the purpose of client usage tracking. It was meant to be customizable by validators, and we see that many validators opt not to use the field to share client information, instead setting the field to a custom value. Whereas the new proposed field has a clear purpose, and is not customizable.
I browse through a few blocks of recent proposals, and it appears that the current graffiti field is underused, i.e., there are still space to append client version info in the graffiti field.
This brings the question: if the space currently available is already underused, then is it necessary to have a new field? Though I can see that having this extra field has the advantage of standardising the client info
Would it be simpler to have all clients to append the client version info instead (like what Teku and Lighthouse currently does, not sure about other clients)
According to @ethDreamer about 40% use the graffiti watermark. Aside from the issues of it not being standardized, user graffiti overrides this data and the information obtained is limited. It doesn’t tell us how much of the network is using multinode clients or what their thresholds are. It also doesn’t provide expansion to into other reporting such as monitoring zkCL proof diversity.
Small note on the zkCL proof diversity motivation:
I don’t think such a beacon block field would ever need to carry it. During the optional-proof phase (EIP-8025), zkAttesters still run a stateful EL and use zkEVM proofs as a supplementary check, so the CL/EL pair reporting in this proposal stays accurate as-is (–> zkEVM proof diversity isn’t a network-safety variable in that phase).
Side-note: one metric that does matter then is proof supply diversity (enough distinct proof types produced per slot), and that’s already natively observable per EIP-8025: every SignedExecutionProof on a dedicated proof subnet carries a proof_type identifier (practically a combination of a guest program + zkVM) and a signed validator_index.
Once proofs become mandatory, the accepted proof set (e.g. 1-of-1) is a protocol parameter —> it’s defined by the spec, not measured. Resilience concerns shift to formal verification and the security of that proof system.
So, from what I can tell, bytes 16-31 don’t need to be reserved for future zkEVM proof diversity.
The CL/EL/multi-node motivation stands regardless, although consuming bandwidth for telemetry at the protocol level (even if only a single byte) seems rather unorthodox given that protocol designers refer to bandwidth as the ultimate bottleneck…
If this can still be spoofed or opted out of and no way to fully verify, I’m not sure what decisively better signal it provides over graffiti, especially considering this adds permanent protocol surface for every client to support.
This data can always be spoofed, no matter how we collect it. There is also no real incentive to spoof it.
Still, I believe the proposed way is a meaningful improvement over the current graffiti way.
It is much more efficient. The current graffiti way uses 1 byte for every single client code character, so that takes up at least 4 bytes for every client pair if you don’t include client version information. This proposal uses only 1 byte for a complete CL+EL client pair.
There is currently no standard way to report client usage in graffiti for multi-node setups, like Vero/Vouch/DVT. While the current graffiti way could be expanded, the inefficiency turns this into a very suboptimal option once multiple client pairs are involved.
The new field is specifically designed for this purpose. There will be no reason or easy way to put any other kind of data in there since the field content will not be customizable.
There is little reason to disable it, as opposed to the current graffiti where operators may disable it just to keep their chosen graffiti intact, prevent leaking client version information or any other reason. Having a bunch of semi-random characters appended to your chosen graffiti is not very nice, subjectively speaking. Since there is little reason to disable the functionality, this will hopefully give us a much better picture of how safe the network is from bugs in any single client, or combination of clients.
The added client maintenance burden is as minimal as could be really, see my previous comment. Other alternatives were explored, as mentioned in the proposal. I don’t see any way to report this in a privacy-friendly manner that would be anywhere as simple to implement and maintain as this proposal.
Currently relays monitor this by looking at the user-agents for incoming requests from proposers. I haven’t looked deeply at the ePBS spec, but it’s probably still possible to collect this data (and in a less permissioned) way after Glamsterdam.
If data monitoring platforms/entities set up uncompetitive builders that are included in the onchain builder registries; and clients include user agents in the header requests, then client data could be collected by anyone who wants it without including the data permanently onchain.
edit: I guess this only works for CL data, but i still think it’s a direction worth exploring.
In the same vein, these builders would see the IP addresses of proposers and then could distinguish validators from ordinary nodes, and we could get more reliable data from crawlers to learn the distribution of EL clients.
We could potentially add things like obol / vouch to the graffiti watermark, but there’d be no real way to capture the data backing them, unless rather than version it was built into a couple of bytes there…
I guess my main point is this does allow us to make some improvements over the types of data collected beyond what’s easy currently.