Doubling the blob count with Gossipsub v2.0

[nisdas]

@djrtwo

Funny to see that 0 is strictly worse (under these non-adversarial conditions). This is because duplicates are just wasting local/global bandwidth? Assuming we were hitting peak load locally or globally, you’d expect 0 to be strictly better. Maybe it’s worth running an experiment with all 1Gbps (or even 10Gbps) nodes to show that when bandwidth is not saturated, that push is faster. This would help us validate the hypothesis of why pull is faster for more bandwidth bound networks.

We re-ran simulations with much lower message sizes. The first 3 charts show the message arrival times with a size of 1kb. As it can be seen here D_announce=0 has significantly lower latency vs D_announce=8. It takes roughly a fourth of the time to reach the whole network compared to D_announce=8 . It does show in the absence of congestion, only pushing is strictly better.

If you look at the next 3 charts, we ran the simulation with progressively larger message sizes from 1kb to 128 kb with a single message broadcasted each time. For D_announce=0 , you can see that message arrival times cluster together until 64kb and 128kb, There is a non-trivial jump when sending out messages with 64 kb and finally a much bigger jump at 128kb indicating congestion. For D_announce=7 and D_announce=8 , at a message size of 128kb the latency is equivalent to that of D_announce=0 . Inferring from these results, at a size of 128kb the network is congested.

@CPerezz

I was wondering if we can take profit of these and already perform some sort of TLS-Handshake or similar. Such that we can establish a connection with Err Correcting Codes or similar properties. This is an idea that appears in other protocols such as PQ-DNSSEC

For gossipsub, we have long-lived streams with all the peers we are connected to. So there is no need to perform any handshake when sending IANNOUNCE/INEED messages.

@CPerezz

This should work as follows:

• We always start with our most restrictive (hence demanding) timeout for every peer.
• For any peer that goes over the timeout:
  • If we receive the data but later, we adjust the timer to the next tier without a penalization to the peerscore.
  • If we don’t ever receive the data, we ask again and adjust the peerscore to the next tier.
• If a peer is at the less-demanding timeout, and still misses, then we demote the peerscore accordingly; This simply means that the peer has bad connection towards us or by itself, or it’s actually misbehaving.

Yeah, this is one way we could handle timeouts and progressively make it larger. However waiting for larger timeouts isn’t helpful when the first/smaller one already failed. It would be better to penalize for repeat failures in responding to INEED messages in time as the timeout bounds the worst case latency for a single node in receiving a particular message.

@AgeManning

The main complication of episub, was the choice of which peers to lazy send to vs which to eagerly send to. I built a system based on measuring statistics of duplicates and message latency to make the decision. Ultimately, I didn’t push for it too hard because the decision on which peers to do it for was complex. Specifically, there were a few constants around the statistics that users would need to decide, and ultimately I think an automatic version would have been better. I also tested a version of this, where all but 2 mesh peers propagated lazily, and the results showed a worse result than if we just forwarded the message directly, which is the opposite result of what is shown here. (i.e I’d expect the latency to be small when you propagate everything without 2 round trips for each message). The caveat here being it was only for 50 nodes, as it used testground which couldn’t scale at the time. I see in these simulations that we are hitting upload bandwidth thresholds, which we weren’t testing in our simulations. On this point it might be worth running simulations where the upload bandwidth isn’t saturated.

Yeah, we wanted to primarily keep the peer selection logic simple rather than do bandwidth estimation of peers, etc. So each message you just randomly choose D_announce peers to send the announcements to. On the episub simulations having different outcomes, the charts I linked above should explain why. In the absence of any network congestion, you would always have lower latency with an announcement degree of 0. However our main goal was to optimize for larger messages and when the network is congested which is the case now in Ethereum mainnet. There should still some benefit to have a low announcement degree for smaller messages if it is tolerable to have slightly higher latency with a smaller amount of duplicates. We can try with D_announce = 1 , 2 and see how the results are for smaller messages.

@AgeManning

One of the main things lacking in gossipsub today, I think is what I’ve seen referred to as “Peer Selection/Sampling”. Fundamentally, it’s just some algorithm, that decides which peers we eagerly push to (mesh), which we lazily push to (IANNOUCNE in this proposal, or IWANT in episub) and which just get occasional IWANTs (for security from censorship).

In this proposal, the Peer Sampling algorithm would have a static mesh size, randomize peers in that mesh between eager and lazy and all others exist outside.

There could be many forms of this “Peer Selection/Sampling” algorithm. The statistics based approach (which tries to minimize duplicates/latency) would encompass the episub thing we attempted a while ago.

Yeah, this would be very helpful for the testing of current and future designs if we can agree on how we want peer selection to be abstracted. For now to keep it straightforward we just have the mesh, since involving non-mesh peers might interfere with IHAVE/IWANT.

@AgeManning

In episub, we would send IHAVE’s directly instead of forwarding the messages. Is it possible that this proposal can be implemented using the current control messages? Is it vital that we have IANNOUNCE/INEED for this and not just use the IHAVE/IWANT methods. I think doing this would allow us to experiment with a wide range of “Peer Sampling/Selection” algorithms without having to modify the spec each time (also scoring methods).

The main reason we had it as a separate control message is to delineate the difference of purpose between the two. The former is for non-mesh peers who will send back any requested IWANTs periodically but not on the critical or fast path. For announcements, these need to be responded back to immediately and within the defined timeout. You could have it as IHAVE/IWANT messgaes but then gossip routers will need to modify how how their current IHAVE/IWANT pipelines to handle this new usecase.