Block Arrivals, Home Stakers & Bumping the blob count

Block Arrivals, Home Stakers & Bumping the blob count

Thanks to all the community members that are selflessly sharing their data that allowed this analysis to be possible, to MigaLabs for their validator entity data and to ProbeLab for an early version of their Mainnet bandwidth report.

Special thanks to Parithosh, Toni, Mikel, Andrew, and Matty for their feedback and help throughout this analysis.

TL;DR

Summary

  • Timing games have historically made it hard to predict a safe blob count bump
  • ethPandaOps recently started receiving block arrival timing data from the community
  • Analysis of the most bandwidth sensitive scenario shows a healthy block & blobs arrival time

Outcomes

  • When naively extrapolating this data and combining with EIP7623, this analysis supports increasing the blob count in EIP7691 to either 4/8 or 6/9.
    • This is only one data point, but it leads to a more optimistic outlook on a potential blob count bump than previous analysis.

Intro

The ethPandaOps team has recently started receiving data from members of the community. Home stakers are one of the Ethereum network’s most valuable assets, and this scheme is starting to shine a light on how they see the network. A sidecar binary is run alongside a beacon node, and records the events happening on the node via the beacon API event stream. These events are sent to the ethPandaOps team, who then publishes the data (with a small delay & some data scrubbing).

For more info:

The team has been collecting community data for around 6 weeks, and we now have enough data to make some interesting observations that were previously not possible.

Background

With the arrival of EIP4844, a block is only considered valid once the block & the blobs referenced in the block are received by a node. This block/blob bundle has until 4s in the slot to be received by the majority of the network otherwise it runs the risk of being re-orged.

Sophisticated operators (and now MEV-Relays) play block proposal timing games. These operators submit their blocks as late as possible to maximise their profit while minimizing their risk of being re-orged. These timing games have historically obfuscated block arrival timing data.

Unreleased, upcoming research from ProbeLab indicates that the 50th percentile of non-cloud nodes on the network have 20Mbps of upload bandwidth.

Blob usage on Mainnet continues to grow.

Problem statement

Ethereum’s decentralization is paramount to it’s success. EIP7691 aims to increase the blob count, and runs the risk of unintentionally excluding some node operators from it’s set if the parameters are too high.

We need to:

  • Ensure that a blob count increase is safe for home stakers, as this group of actors is the “worst case” for a blob count increase as they have the lowest available bandwidth.
  • Ensure that the network has enough data throughput to support Layer 2 growth.

Given the existing landscape, we can make some assumptions with regards to a potential blob count increase:

Least at risk:
Counterintuitively, if you looked at block arrival data, operators playing timing games would appear to be the most at risk of being impacted by a blob count increase. Being reorged out for proposing late is bad for business, and we can assume they’ll adjust their block proposal timings accordingly. A blob count increase is unlikely to be problematic here.

Most at risk:
A solo staker building a block locally (no mev-relay) and being attested to by other home stakers. In this scenario:

  • The proposer:
    • needs to publish their block and all blobs to the network. This node needs to publish the block (~100KB), and then all of the blobs (128KB each) to all of its mesh peers as fast as possible.
      • when building blocks locally, the proposer does not have the help of the MEV Relay gossiping the block/blobs bundle to its own peers.
  • The attesters:
    • need to download the block/blobs bundle from the p2p network before 4s in to the slot.

This analysis will ask the following questions:

  • Question 1: How is 3/6 performing on Mainnet?
  • Question 2: Does arrival time scale with block/blob bundle size?
  • Question 3: How much more can we support on Mainnet today?

We’ll answer these questions from the perspective of a home staker as this is our most at-risk group of operators.

Analysis

Start: 2024-10-04T22:00:00Z
End: 2024-11-25T02:00:00Z
Blocks: 366,384
Arrival events: 75,945,392
Countries: 9 (Australia, Bulgaria, Czechia, Germany, Italy, Spain, The Netherlands, United Kingdom, United States)

Check out the juypter notebook here.

The timing data was captured by Xatu Sentry, a sidecar binary that runs alongside a beacon node and records many events via the beacon API event stream. These events can be considered worst case for block arrival times as there is additional processing time and network overhead for the event to be recorded. From analysis, this overhead ranges between 50-300ms but we have kept the timing data as-is in the interest of safety.

Each beacon node implementation emits block, head, and blob_sidecar events in different orders. To address this, we define a block/blob bundle as “arrived” only after all associated events for the slot have been recorded from each beacon node. This is once again a worst case scenario.

Question 1

How is 3/6 performing on Mainnet?

TL;DR: Pretty well!

This chart shows block/blob bundle seen arrival times against the combined size of the bundle. When looking at locally built blocks proposed by solo stakers and seen by home users, a lot of block/blob bundles are seen before 4s.

We should also look at blocks provided by MEV Relay as the reality is that a lot of blocks are proposed via this route. We can see an increase in the min, as there are additional round trips involved in this process, but things still look healthy!

Outcome: Block/blob bundles are arriving well within the 4s deadline for our home users.

Question 2

Does arrival time scale with block/blob bundle size?

TL;DR: Yes

The trend lines show the 99th, 95th, and 50th percentiles of arrival times - meaning what percentage of blocks arrive were seen faster than that line.

These percentile trend lines answer our question: as bundle sizes increase, arrival times also increase. This suggests bandwidth is the primary bottleneck for these actors.

Again looking at blocks provided via MEV Relay, we see a similar story.

Outcome: Yes, arrival times scale with block/blob bundles size

Question 3:

How much more can we support on Mainnet today?

TLDR: 4/8 and 6/9 are both achievable

To answer this question we need to check how big the block is. The 99th percentile of compressed beacon block size through our time period is 101KB. Our blobs are fixed at a size of 128KB.

Using these parameters, we can overlay the block/blob count:

We can very naively plot a trend line to see when would cross the 4s attestation deadline.

This trend line assumes a linear relationship between blob count and arrival time. Under this assumption, we can support up to 14 blobs per block while maintaining 95% of block/blob bundles arriving within the deadline. The 95% target provides margin for the 50-300ms processing overhead in our measurements, while also accounting for outliers.

When looking at MEV Relay blocks specifically, the data shows an even more optimistic picture - the 95th percentile trend line indicates support for up to 20 blobs per block. This improved performance can be attributed to MEV Relays being high-bandwidth nodes that help distribute blocks and blobs across the network in parallel with the proposer.

EIP7623

EIP7623 improves the worst case compressed block size to ~720KB - about 7x larger than our average historical block. Let’s analyze if we can still support more blobs with this increased block size.

Even with an absolute worst case block size with EIP7623 we still support a blob increase. Note that the current maximum compressed block size on Mainnet is 1.79MB (and we’re seemingly going ok!), so take this data point with a grain of salt.


The trend for MEV Relay blocks again supports a much higher blob count compared to locally built blocks.

Outcome: The data supports a blob count increase, especially if EIP7623 is included at the same time. 4/8 or 6/9 are both safe to apply. There is potential for a higher blob count, but we’ll need to see how the network performs with an initial bump first.

Conclusion

Ethereum’s decentralization is fundamental, and home stakers play a crucial role in this picture. The network is a delicate and intricate system that demands thoughtful and deliberate consideration.

Our analysis indicates that block arrival performance surpasses initial expectations for nodes with limited bandwidth. The community-contributed data offers valuable real-world insights into the network’s capabilities, and we would like to once again thank those who are contributing their data.

While we naively assumed a linear relationship between blob count and arrival time, this is a simplified view of a highly complex distributed system. Additionally, there are ongoing work streams that could either improve or worsen bandwidth requirements over time. Our analysis is focused on the data available to us now, based on observations from the past six weeks of network performance.

Based on block/blob arrival metrics alone, increasing the blob count from (target 3/max 6) to (target 4/max 8) or (target 6/max 9) appears to be viable. However, this is just one of many factors to evaluate when deciding on blob count adjustments.

19 Likes

A deadline of 4s gives zero time for validation. Nodes can in principle optimize and precompute head optimistically and then discard the node if it’s not available but that’s not done in practice so at the very least any forkchoice accounting should be included (and presumably database access on some clients). Another end is the VC BN interaction and the signing by a potentially remote signer. I expect accounting for these should not change much your thesis, but anyway felt it should be included

5 Likes

Thanks for noting ‘arrival’ as when it’s been processed by the node and the events distributed, helps understand the graphs.

This looks like more than 1% of proposals don’t meet the 4 second benchmark, meaning they’d be missed when looked at for attestations, which isn’t great…

If you follow the trail to 9 it kind of looks like that number is approaching 5%… admittedly 5% of a small percentage of overall blocks being late but still definitely relevant.

This was a good set of data to look into, really appreciate all the work it must have taken to gather!

4 Likes

Thanks for these very insightful results!

ProbeLab’s study that @samcm referred to is this one that came out yesterday: Bandwidth Availability in Ethereum: Regional Differences and Network Impacts.

I think a relevant point to this study is the number of blobs per block carried over the network currently: Bandwidth Availability in Ethereum: Regional Differences and Network Impacts. We see that in 42% of the cases (slots), there are 5-6 blobs anyway.

Although it’s not straightforward, or accurate to make a direct correlation, the network is currently carrying what the future target would be (6) in 40+% of slots and there doesn’t seem to be any disruption at all.

1 Like