FullDAS: towards massive scalability with 32MB blocks and beyond

[cskiraly]

@cskiraly

We can speed up delivery by “rarest first” like techniques. Rarest-fist scheduling is used in P2P dissemination to cut the tail of the latency distribution, equalizing the diffusion of different pieces. Rarest-first requires precise “have” information, which is typically not available, but it can be approximated. For example, a technique we have already implemented in our simulators is to make sure a copy of every single segment is sent out by the block producer before sending the second copy to another neighbor. We start by sending only one copy of each segment, to one of the block producer’s column/row neighbors (randomized), seeding segments in random places in the network. Only then we send second copies to other peers. Later, one can use information collected through e.g. IHAVE bitmaps to estimate which segments are diffusing well, and which seem to be hindered, prioritizing further seeding based on this information.

I’ve posted more details on this technique in a follow-up, under the name “Batch Publishing”. Here it is:

Improving DAS performance with GossipSub Batch Publishing

Author: @cskiraly This research was conducted by the Codex Research Team: @cskiraly, @leobago and @dryajov TL;DR We can achieve significantly lower latencies in DAS by applying chunk/peer scheduling techniques compared to using “standard” GossipSub. If the main bandwidth constraint is the publisher’s uplink, individual segments will get diffused in the system shortly after sending out their first copy. Hence, first copies should be prioritized. Batch publishing is a way to introduce this prio…