Execution & Consensus Client Bootnodes

Having looked in depth at the dependency of bootnodes, I would like to express my concerns about the centralisation and reliance on third-party cloud services that current bootnodes exhibit.

Overview Execution Clients

Go-Ethereum

Nethermind

Erigon

Besu

On the CL side (i.e. Beacon node), I couldn’t figure out any information on the hosting location.

I was wondering what other people think about this, especially since Hetzner, for example, doesn’t really seem to be crypto-friendly. Personally, I would strongly advocate for more bare metal in Ethereum! In summary, I would like to see more transparency from the EL and CL side on hosting information.

Please see my replies below about the source links of the different client bootnodes.
Update: I was now able to put all relevant links into this description. I deleted the further replies accordingly.

1 Like

Will second and echo the sentiment shared here. The problem I think is that it seems like operating bare metal reasonably well requires some degree of sysadmin expertise and I don’t think these types of people are easy to come by? Could be wrong.

To that end, if there’s any docs or community resources, esp from client teams, would be good to curate and consolidate in one place to make everyone’s life easier. I am not aware of what’s out there given I haven’t had a chance to look but happy to pitch in and help on this if there’s some degree of consensus that this is somewhat of an issue. Even something as simple as a vendor sheet with basics around pricing, jurisdiction, geographic diversity of DCs, etc would be helpful. Ideally we don’t need to rely on DCs but I don’t see why bare metal can’t be a compelling option (alongside independently operated infra / self-hosting).

1 Like

This is concerning, certainly, but the failure mode is a liveness failure for new participants which isn’t the worst situation, especially since you can change your bootnodes via configuration.

It would be great to see more boot nodes that are hosted independently (e.g., in someone’s basement), or at least spread across many different datacenters in many different jurisdictions. This doesn’t mean just different AWS regions, because all of AWS (except China, sort of) are under the jurisdiction of the US. Ideally we would find some smaller locally owned/operated DCs in various jurisdictions including places that have a bit of a history with censorship resistance and privacy like Iceland, Sweden, Switzerland, as well as some jurisdictions that are just non-functional like many “developing nations”.

2 Likes

especially since you can change your bootnodes via configuration

yes, but many don’t even know this! So the issue here is also about creating awareness & create a game plan for the extreme case that the bootnodes get censored. One very important feature that needs to be implemented by client teams is a warning when bootnodes are becoming unavailable. This would be an early warning for the community and we could react by opening PRs, publishing Tweets etc. about alternative enodes or Ethereum Node Records as defined in EIP-778.

It would be great to see more boot nodes that are hosted independently

Exactly, this is pivotal to mitigate any liveness failure. The liveness failure is not only for new participants but can also happen to existing ones that need to restart.

2 Likes

I thought clients kept track of peers they discovered across restarts?

Better yet, have the DCs located in your geographically diversified basements :blush:

1 Like

I thought clients kept track of peers they discovered across restarts?

Yes, but in an extreme censoring event they might lose the peers during restart and need the bootnodes.

I think to @MicahZoltu’s overall point though (while I do appreciate your caution here), realistically speaking, seems like it would be quite difficult to fully censor the peering process at least for any significant period of time, no? And it doesn’t seem tremendously difficult to respond to this type of censorship attack, though it is good that you are calling attention to something that not many people pay attention to (and the vast majority are simply unaware).

Counterpoint to the point I made above on “this being too hard to censor for long”:

  • I guess a state actor could go through the local ISPs (and in any given country, there are typically only a few given they are basically always oligopolies and occasionally state-owned/operated and/or somehow state-sponsored public-private actors in some regions such as Asia) and do the damage that way…hm

At the core of Ethereum should be a censorship-resistant design. I think we all agree on this. Thus, it’s also important to think about the unimaginable (even though we’ve globally seen already such censoring events in certain countries or similar ones like GitHub with Tornado Cash). It will be very difficult to execute such an extreme event, but it remains a possibility (coordination across a couple of countries is not unusual…). That’s why I think it’s important to think about a game plan and document it properly in case such extreme (or even less severe) events take place so operators/participants know how to act accordingly. Where could you potentially find new uncensored bootnodes? How can I configure it? etc.

That’s why I’m convinced that it’s so important to have globally diversified DCs for the bootnodes to circumvent supranational but still local censoring attacks. Peers can always be censored via ISPs so it’s important to have globally distributed bootnodes available to preserve the censorship-resistance core value and resilience of Ethereum as a whole.

1 Like

Sounds like we need better docs and better visibility on the hosting options available. What else do you think we need to do here?

Ideally we don’t need a DC but for boot nodes I understand why putting these on someone’s laptop in a basement is probably not the most robust long-term solution (well unless a lot of people do that but that’s not reasonable to assume as likely)

The most important part IMHO is to create awareness about the issue. We can’t fully resolve this challenge but we have to make the network participants conscious of this.

The problem with cloud solutions is the following: most of them a running under US law, so in case we even diversify on the cloud providers for the bootnodes, a single point of failure exists: the US enforcement possibility. One option is that the EF is setting up in at least 20 countries around the globe independent bootnodes running each on country-specific (i.e. local) DCs. Another additional option could be that the EF maintains an official bootnode list (including the hosting details) from which each of the clients pulls the information and you can be also added as an individual there after being carefully vetted - i.e. trying to include the broader community. We would need to think about an incentive scheme (& slashing possibility) there of course. Maybe others have other ideas…

2 Likes

Obviously it would be nice to have support, but in the absence of that I don’t see why we can’t do some of the stuff ourselves? “Ourselves” being community members, sysadmin-types, devops people, Core devs who have time to comment, and enlist folks from the various home staker & solo staker communities — would involve some degree of cat herding yes

fully agreed: how can we best create more awareness about this important discussion @timbeiko, @MicahZoltu? Cc: @vbuterin

2 Likes

Hey Guys,

@randomishwalk sent this thread to me over twitter and I am jumping here as I can potentially help with distributed global infrastructure

latitude.sh, bare metal company I operate, run in 15 locations (9 of them being our of the US) - Global regions to deploy dedicated servers and custom projects - Latitude.sh

Happy to chat more