Wen fast payload broadcast? Segment, code, push, pull, and everything in between
Note: this post contains my own views, shaped by discussions with many I have worked with on these topics in the past years. The Vac/nim-libp2p work on large messages in gossipsub (staggering and fragmentation, PREAMBLE and IMRECEIVING), RLNC block propagation from @potuz, erasure-coded broadcast in ethp2p from @raulk, and works from @nashatyrev, @kamilsa and @kaseyk all describe various aspects of this collaborative effort to improve Ethereum networking.
TL;DR
-
Whole-payload gossip is viable on mainnet today because datacenter builders and nodes carry it — a fact of the deployment, not of the protocol. At roadmap payload sizes or shorter slot times, it may no longer work.
-
The fix is small: split the payload into fixed-size segments, commit to the segmentation in one extra field of the builder’s bid, and add three diffusion rules. Segments pipeline through the network instead of store-and-forward.
-
In our least favourable setting — a 1 MiB payload from a home builder with no datacenter nodes — the segmented design reduces median receiver completion time from 4.9 s to 0.73 s and cuts the payload bytes each node receives threefold, from 4.4 copies to 1.4. The dependence on datacenter infrastructure is removed, not mitigated.
-
Every segment is independently verifiable before forwarding: a Merkle proof against the bid’s commitment, anchored through the block the receiver already validated.
-
Add the commitment field at Gloas or Hegota; activate the networking-only rules at Hegota.
Broadcasting large execution payloads over plain gossipsub becomes a consensus problem at the sizes the gas-limit roadmap points toward. It also obstructs plans to shorten slot times. This post measures the segmented design summarized above, then compares it with more elaborate transports.
The problem
Gossipsub was not designed for large messages, so whole-payload gossip is slow.
Payload size has always mattered: before Gloas the execution payload rides inside the beacon block, so a large one makes the block late and costs attestations. Gloas turns that delay into a separate failure. The proposer publishes a block carrying a SignedExecutionPayloadBid, and the winning builder later publishes a SignedExecutionPayloadEnvelope on its own gossip topic. The payload therefore gets its own message, deadline, committee vote, and responsible party. It can be late even when the block was timely. Throughout, block means the consensus block — the beacon block the proposer signs — and never the execution block; payload means what gets segmented, with one container nuance in Part 1.
Two properties make diffusion slow. Every validator needs all of it: data-availability sampling does not apply here yet, because that would need ZK execution proofs. And a single gossip message is store-and-forward at every hop: a node receives and validates the whole object before forwarding it, so serialization delay repeats instead of pipelining.
How large is it? Mainnet execution payloads today are around 100 KB, nearer 200 KB at the current 60M gas limit. This study tests 1 MiB — roughly 5× that, chosen because it is where the gas-limit roadmap points rather than where mainnet sits. Figure 1 returns to payload size directly.
The three-second budget
The payload is due at half the slot — PAYLOAD_DUE_BPS, 6 seconds on today’s 12-second slots. The payload-timeliness committee (PTC) votes at three quarters, 9 seconds, but its payload_present bit records whether a valid envelope arrived before the 6-second mark. Arrival in between earns a negative vote, not partial credit. Both deadlines are fractions of the slot, so they shrink with it.
The spec fixes the ordering, block first, but not the reveal time. This study assumes a reveal near 3 seconds and therefore a 3-second reveal-relative budget. That is an operating assumption, not a Gloas constant. One scope note: we measure network lateness against this budget; mapping shares of timely receivers to actual payload_present outcomes would require the PTC’s sampling and quorum rules and is out of scope.
Why whole-payload gossip works today
Whole-message diffusion of a payload this size is timely on today’s mainnet because datacenter builders and nodes carry it — a property of the deployment, not the protocol. We simulated both sides of that dependence with real Prysm and go-libp2p-pubsub code.
The model has 500 nodes, connectivity degree 70, gossipsub mesh degree 8, synthetic geographic latency across five regions with one-way mean ≈56 ms, 50/100 Mbps (up/down) home links, and a 1 MiB payload compressing to ~746 KB. Its virtual clock models network delay and bandwidth but not computation; we measured that asymmetry separately and found it negligible here.
With a home publisher and no datacenter nodes, whole-message diffusion misses badly: median completion is 4.89 s, and the median share of timely receivers is 5%, never above 11%. Completion is per node — the interval from reveal until that node holds the full payload — and a run’s median is the p50 across its 499 receivers.
Table 1 sweeps the builder’s link against the fraction of nodes that are datacenter-hosted. The all-home cell with a home builder is the case a protocol independent of centralization must pass; it is the same configuration as the headline above. The 20%/home cell estimates today’s realistic worst case: a proposer building locally in a network where a conservative fifth of nodes have datacenter bandwidth to spare.
| datacenter nodes | home builder | external builder |
|---|---|---|
| 0% (all home) | 4.89 s / 1–11% timely | 3.08 s / 34–70% |
| 20% | 2.90 s / 0–99% | 1.80 s / 98–100% |
| 50% | 1.97 s / 78–100% | 1.11 s / 100% |
Table 1. Each cell gives the mean of per-seed p50s and the min–max share of 499 receivers timely by the budget. The timely share is the tail metric because the deadline is fixed and the committee counts heads: what matters is the percentile at the deadline, not the latency at a percentile. This is illustrative, not predictive: wide ranges, and a randomly placed 1 Gbps link class standing in for “datacenter”.
The direction matters more than a threshold. Node bandwidth moves the result more than builder bandwidth because every forwarding node serializes a complete 746 KB copy — several in parallel — on its own uplink, which builder provisioning cannot improve.
Thus this payload is timely when a datacenter builder publishes it and datacenter nodes occupy its paths, as on current mainnet under MEV-boost. A proposer building locally on a home link has no such path. At payloads the roadmap is heading toward, the mechanism is not sound; the deployment happens to be favourable.
What has to be solved, then, is three problems in one. The first is a networking problem: one source, one large message, every node needs all of it. The second is a trust problem, cryptographic and about information flow: a receiver must be able to verify a piece against something it already trusts, before it holds the rest. The third is their product. The piece’s anchor arrives in a different message from a different party — the block from the proposer, the segments from the builder — so payload diffusion becomes a special case of large-message diffusion: two sources, and a gate between them. Part 1 takes the trust problem, Part 2 the networking one, and “What the coupling costs” measures their product.
The proposed solution: segmented diffusion
Split the payload into K segments (e.g. 32 × 32 KiB for 1 MiB) and diffuse them as independent gossip messages on one shared topic. Each hop forwards a 32 KiB unit rather than a 1 MiB object, so paths pipeline and no single node’s uplink gates the whole payload.
On the same base, with a home builder and no datacenter nodes, the recommended configuration (segments, batch publishing, phase forwarding, disciplined pulls — all measured one by one below) gives median 0.73 s, every receiver timely in every seed, 1.4 received payload copies per node against 4.4 for whole message. The price is a consensus-container field and networking rules that every client must implement. Add the field at Gloas; activate the networking-only rules at Hegota.
Figure 1. Mean per-seed median completion time for whole-message diffusion and the recommended segmented configuration, swept by payload size with a home builder and no datacenter nodes.
The proposal has two parts. A bid commitment lets each segment prove membership in the committed segmentation. A networking change carries those segments without violating the trust chain. The prototype implements the networking half in full; for the bid commitment it uses an interim admission rule — first group offered per slot, the right cardinality but no authenticity — with the same map-lookup cost and no effect on transport results.
Part 1 — the bid commitment
A node must not forward what it cannot verify. The builder is already deterred: payment is unconditional, and the reassembled payload is checked against the bid’s commitments, so a builder that reveals garbage pays and delivers nothing. Per-segment validation instead protects the forwarding node, which no payment or slashing reaches. Without it, one malicious node can inject a corrupt segment that the mesh propagates until every downstream node discards the reassembled payload. Whole-message gossip validates the complete object before relay; a segment is meaningless alone. Each segment therefore needs a commitment and a proof that a receiver can check without the rest of the payload.
The first requirement is a vector commitment to the segment sequence, allowing each segment to prove membership. A Merkle tree provides a 32-byte root and log K hashes per proof, is cheap, and composes with erasure coding if decoders enforce codeword consistency. The proof can travel with the segment, as in DAS; as an alternative, a protocol could send the tree or its top layers separately.
The second requirement is a chain of trust from that commitment to something the receiver already trusts:
installed block → its bid → the committed descriptor → Merkle proof → this segment
Each link authenticates the next, anchored by the block carrying the bid after the receiver has validated and applied it. Installed does not mean attested or final; the anchor is exactly as strong as the receiver’s own block validation. The chain runs through the block, so a segment cannot be authenticated before its block arrives. That is the cost of a consensus anchor; at the end of this post we briefly discuss alternative authority routes.
Figure 2. First-segment arrival and block installation are the two relevant wavefronts; raw block arrival is shown for reference. The shaded interval marks nodes holding a segment they cannot yet verify. Curves are schematic; installation adds an assumed 200 ms.
Concretely, add execution_payload_segment_group_id to ExecutionPayloadBid — the bid message the builder signs, not its signed wrapper. It commits the canonical descriptor: version, hash id, segment count, segment size, total length and Merkle root. These framing fields make wire bounds sanity checks rather than load-bearing constants once the block is installed.
The obvious object to segment is the serialized envelope, but it contains beacon_block_root. Committing that descriptor in a bid later carried by the block would be circular. Instead, segment an SSZ container of what the builder holds at bid time:
ExecutionPayloadSegmentBody:
payload
execution_requests
beacon_block_root, parent_beacon_block_root and builder_index then come from the block when the ordinary envelope is reconstructed.
Part 2 — the networking change
The wire must identify, both in each segment and in its message ID, the block that would authorize it.
A segment carries its index, Merkle proof, slot + beacon_block_root, and group id, but no descriptor or signature. The group id is a structural claim: after the named block installs, it must match the group that block committed to. Overhead is ~210 bytes per 32 KiB segment (~0.6%), but the important property is that the segment asserts no authority of its own. The proof catches forged content.
Message ids must also be self-describing, as in FullDAS. Today’s content hashes do not reveal which block would authorize an announcement, so a receiver cannot decline what it cannot verify. A hybrid id combines a structural (slot, block root, group, index) prefix with a content digest; a purely structural id would let a bad first arrival poison gossipsub deduplication. The group stays in the id because, before installation, a receiver cannot resolve block → group. That untrusted namespace is what the buffering bounds cover. The id function is defined by the Ethereum spec and installed by the application, so this requires no libp2p wire change. It remains normative and incompatible and therefore activates at a fork boundary.
Diffusion rules. These three form the complete normative set. Disciplined pulls and phase forwarding, measured below, remain local performance policies:
-
Do not relay a segment whose authorizing block is not installed. Here installed means validated and applied, not merely received. Buffer it in a bounded queue with per-peer reservations so one peer cannot exclude another’s copy of the same segment.
-
A receiver may decline to request what it cannot authenticate. Because an announcement is not re-offered automatically, a bounded ledger of declined ids and a block-installed event must resubmit them, with
ExecutionPayloadEnvelopesByRootas fallback after the announcer’s cache expires. -
A sender may push eagerly only with evidence the peer holds the block: it sent the block to that peer, received it from them, or saw an IHAVE or IDONTWANT for it. The receiver can reconstruct whether the push was entitled, making a violation grounds for descoring or disconnection. The prototype does not yet score it. The rule presupposes something the other two do not: a sender has evidence only for peers it exchanges blocks with, and with independent meshes for the block topic and the segment topic (eight of seventy connections each) those overlap in about one peer per node, so eager pushes would go almost nowhere and phase forwarding would degrade toward pull-only. It needs a shared mesh for blocks and segments — co-routing them over one topology — which also makes the block tend to precede the segments along every path. That helps the gate without being authority: validation runs asynchronously, so wire order does not imply installed-before-validated. Our measurements use independent meshes and unconditional pushes; the rule’s cost on them is unmeasured.
What the coupling costs
The block-install delay, not network arrival, determines how many nodes hold segments they cannot yet verify. At an assumed 200 ms block-install delay, 439–489 of 499 nodes are affected; non-relay reduces that to 8–11, each buffering a few hundred kilobytes for roughly the install delay. Isolated, non-relay affects 7–16 nodes and request gating 51–68. Every receiver met the 3 s threshold in every tested configuration.
The 200 ms is a number we picked, not a measurement, and it is the dominant variable here; the real value is a profiling question on a loaded node, not a simulation one.
What each mechanism buys
Segmentation buys most of the latency improvement; publishing and forwarding policy buy the rest, while disciplined pulls primarily reduce bytes. The mechanisms also buy different kinds of loss and omission tolerance. Table 2 adds one mechanism at a time on the same base and repeats the experiment on the two datacenter mixes from Table 1, whose whole-message row it shares. Cells are means of per-seed median completion, and payload copies received per node (bytes over one compressed payload):
| configuration | home builder, all home | home builder, 20% datacenter | datacenter builder, 20% datacenter |
|---|---|---|---|
| whole message | 4.89 s / 4.4× | 2.90 s / 4.5× | 1.80 s / 4.1× |
| + segmentation alone — ordered, full-mesh push | 1.76 s / 6.4× | 1.45 s / 6.2× | 0.94 s / 5.7× |
| + batch publishing — first copies before repeats | 1.01 s / 6.3× | 0.81 s / 6.2× | 0.70 s / 5.7× |
| + phase forwarding — push two, announce the rest | 0.76 s / 3.1× | 0.72 s / 3.2× | 0.62 s / 2.9× |
| + disciplined pulls — one request per id, offer table, move-on and ban | 0.73 s / 1.4× | 0.67 s / 1.4× | 0.62 s / 1.5× |
Pipelining does most of the work. Cutting the payload into 32 KiB units that nodes forward as they arrive is the largest single step, 2.8× on its own: no uplink serializes the whole object. It costs bytes at first — full-mesh push of 32 pieces carries more copies than one push of the whole — and the last two rows are what takes them back. Batch publishing sends one copy of everything before a second copy of anything, so no segment waits at the source and each takes a different first path. Phase forwarding trims the remaining latency.
Disciplined pulls buy the bytes. Requesting each announced id from one peer rather than every announcer halves the phase arm’s received bytes at equal latency, 3.1 to 1.4 copies in Table 2’s last two rows, and takes pull-only to essentially one copy. This local fix is already in flight upstream as #625. A single request can be captured by a peer that announces but does not serve, so the requester records alternative announcers in a bounded offer table, tries the next after a short window, and briefly bans a peer whose claim lapses. That memory is free when nothing goes wrong, costs a fraction of a copy in eager re-asks, and keeps honest receivers complete even when most announcers withhold.
Phase forwarding offers a push-pull compromise. A few eager first copies plus announcements for the rest cost about 0.4 of a copy over pull-only. A lost copy is repaired by the announce-and-pull path; a lost request instead idles that segment for a full retry window. Under loss, the phase arm degrades mildly while pull-only’s tail stretches, making the byte-cheapest configuration the most brittle.
Erasure coding buys loss tolerance and cuts the delay distribution tail. A withheld segment can come from another peer; a segment the source never sent cannot. Omitting 4 of 32 plain segments strands every receiver, whereas a 32-of-64 coded group completes everyone because any 32 shards reconstruct. The price is complexity (not much of it), and code consistency to get right.
The queue bounds under attack
Before a receiver installs the slot’s commitments, a fabricated group is indistinguishable from an honest early arrival and earns only bounded buffering. After installation, it is refused for the cost of a parse. Block-install delay sets normal deferral; expiry and the per-sender slot bound cover the case where no block installs.
The exposure is local because deferred or refused junk is never relayed. The slot bound must be per sender: a global bound would let an attacker fill the table first and exclude later honest groups.
Open specification work
The envelope signature. Reassembly yields the envelope’s contents, but Gloas requires the builder’s signature over the assembled envelope, which contains the block root, so its bytes exist only two steps after the bid is signed and the bid cannot commit to them. Either it travels as a small message of its own, which then needs the same admission and recovery treatment the segments got, or segmented reveals drop it: the signed bid already authenticates content and builder, and the signature’s only non-redundant role is authorizing the reveal act, so dropping it changes what “revealed” means for envelope validity and the PTC’s payload_present bit. That is a consensus question, not a networking one.
The wire type and its bounds. Topic name, SSZ type, maximum message size, and the count, size and length bounds are unspecified. They must be tight rather than sanity constants: a loose total-length bound admits a 256 MiB claim against a 10 MiB gossip limit, and a 1 MiB per-segment cap admits an unsegmented segment. Committing count and size in the bid makes them exact once the block is installed; before that, the message-size cap and the admission bounds stay load-bearing.
Design alternatives and transport choice
The measurements chiefly compare transports. Consensus-side alternatives need separate treatment:
-
Commitment scheme. An SSZ generalized-index multiproof authenticates tree nodes rather than byte ranges and forecloses erasure coding; Pedersen-style homomorphic commitments (as proposed for RLNC block propagation) fit if network coding enters. A Merkle tree composes with coding and is the choice here.
-
Where the commitment lives. A builder-signed descriptor avoids changing a consensus container, but its one-candidate bound is policy — with first-wins and reorg semantics, a signing domain, and one BLS verification per group. The bid field gives a cryptographic bound: one group per authorized block root, checked by hash comparison.
-
Where authority comes from. Wait-for-the-block admits exactly one group per authority object, cryptographically. A proposer-header proof or gossiped bid relies on policy because a proposer may sign many, while a detached builder signature is unbounded within its window. This ranking yields Part 2’s rule: authentication is a membership test, so a segment asserts no authority.
-
The header proof alone provides independence from block arrival. It costs ~1.9% of payload bytes per segment, or 7.6% at 8 KiB segments, and exchanges a cryptographic bound for a policy one to avoid a coupling measured as small. Unbuilt; no cost comparison has been run.
-
Slashing. A segmentation-specific offence would punish a builder already deterred by unconditional payment and would not reach forwarding nodes. We see no need for one.
Transport variants. We evaluated four ways to move segments:
-
A — one shared topic. Every segment is an ordinary gossip message on one topic. This is the proposal and the source of every headline number here.
-
B — partial messages. One logical message carries all segments, with per-peer bitmaps recording inventories; that requires a gossipsub extension. Tuned like A, it is timely at 1.8–2.2 s and at byte parity with A, assuming per-segment compression, but remains a second slower and needs its own memory to recover claims that lapse unserved under withholding.
-
C — one topic per segment. Separate meshes buy path diversity, producing 667 ms median and 1.20 copies at 64 topics on the clean 500-node base. It degrades first under withholding — 89–93% timely and p90 2.75–3.05 s — so its clean tail depends on honest announcers.
-
D — custody subnets with erasure coding. Each node custodies a subset of a coded group, the FullDAS shape, capping byte cost near the subscription ratio: ~1.33 copies versus 2.23 for coding on the shared topic. One shard beyond the code margin stalled only the 8 nodes whose custody covered every omitted shard; 491 of 499 completed.
Figure 3. Completion time versus payload equivalents received per node on the 1,000-node base. Points are per-seed medians, bars span seeds, both axes are logarithmic, and lower left is better.
Variant A uses the least machinery. It meets the worst-base deadline with wide margin — 0.73 s against 3 s — so C’s extra quarter second buys nothing the budget needs and costs a mesh per segment plus weaker withholding behaviour. B pays for inventories the deadline does not need. D’s coding earns its place only if source omission enters the threat model. A uses stock gossipsub messages end to end; everything beyond the fork-gated changes is local policy. However, note that everything above depends on implementation details, so do not take it as a strick ordering. Take it as A is good enough and simple.
The recommendation is the one-shared-topic transport (variant A) with phase forwarding and disciplined pulls backed by an offer table, a bid-field Merkle commitment, per-segment Merkle proofs, and block-anchored non-relay — a measured direction, not a specification.
The skeptic’s questions
A speedup this large invites questions about the harness. We tested what we could; burstiness and real install delay remain less comfortable gaps.
“Isn’t this really just some gossipsub bug you fixed?” The headline pair differs in segmentation and the request path: the disciplined pulls in Table 2’s last row are a substrate fix that applies to whole-message gossip too. Applying only that fix moves the worst corner from 4.89 to 4.70 s at 2–12% timely, still a bad miss in the design’s target case. But there are many implementation details, which admittedly can make comparing variants a comparison of implementation choices (and bugs) rather than of fundamental differences.
“Your clock hides CPU, and segmentation does 32× the work.” True; the blind spot favours the proposal. The measured total is ~2.5 ms against a 3.5 s effect — 0.07%, which cannot threaten the result. These are service times on an idle machine; contention on a loaded node is unmeasured.
“Cold connections flatter segmentation.” Warming every link moves whole-message latency by −7.8% across three seeds, small against a 5.4× gap. Segments at ~23 KB already fit the initial congestion window, 40 KiB in modern QUIC implementations; warming actually slowed that arm by 34%. Sweeping the initial window from 12.5 to 160 KiB moves it by at most 11%, with the stock window at the optimum. Neither result is an artifact of transport state.
“Does it hold on a loaded network? On a thin link?” Background gossip up to 32× a mainnet slot’s average non-payload traffic moved nothing. Above that, flat results at loads the uplink could not carry suggest the stack sheds the flooded topic before the wire; that is inferred, not directly measured. The 3-second budget holds down to about 15 Mbps of residual uplink for a 1 MiB payload, then degrades through lateness rather than failure. Coding’s extra copies stop being free first, from around 20 Mbps. Slot-boundary burstiness from blob-scale co-arrivals remains the largest untested realism gap.
“Why not erasure code?” It provides tail performance and robustness against network errors and adversaries with almost no CPU cost. It remains a compatible follow-on, but is not necessary for the first version recommended here.
“Does the design require a fixed segment size?” No. It can fix a count or size, derive the size from payload length, or allow a bounded declared choice; this post does not choose among them.
“What if your link model is wrong?” It is necessarily one operating point: networks vary across places and time. It is sufficient for directional conclusions, not prediction.
“Is this a problem mainnet has today?” Yes and no — it depends on who publishes a block, with what timing, and what state the network is in.
OK, so when should all this happen?
These are just my views on how I would do it, definitely not the only reasonable option on the table. The two changes have different deployment costs and constraints.
The bid commitment is a consensus change — one field and one container — and consensus changes travel only by fork. Gloas is already defining ExecutionPayloadBid as it detaches the payload. Adding the field while that container is being designed costs a spec review; changing it after shipment costs a fork of its own. Even if we are far ahead with the timeline, I think we can add it in Gloas. If not, then I would add it in Hegota.
The networking change is relatively easy to introduce, but needs loads of testing: the prototype and measurements exercise it, and it needs no libp2p change. However, this is only a prototype, and multiple production implementations are needed. Segmented diffusion therefore needs time, but I think Hegota is an easily reachable target with the shape proposed here. Adding erasure coding is also possible, and provides clear benefits, if there is appetite, but it is not necessary in the first phase.
