Eth2.0-specs: adverse effect of sharing peers between testnets/mainnet

[jrhea]

I am concerned that a shared routing table between testnets and mainnet could significantly slow down the search for peers within specific attestation subnets.

I spoke @AgeManning about this and he said it should be easy to add a config option that only allows peers to only be inserted into the routing table if the fork-id matches the local enr.

Thoughts?

cc: @djrtwo @fjl @zilm13 @mkalinin others

[mkalinin]

IMO, the speed does mostly depend on the total number of peers in the DHT and how the number of advertising peers is related to it. I think this relationship is hard to predict for Phase 0 but at least we could try to simulate out the threshold when it becomes infeasible to keep your discovery up-to-date with the network state and quickly connect to a subnet on demand.

The more peers in the DHT we have, the better network resists eclipse and sybil attacks exploiting the DHT. If we lock in with only mainnet participants we will obviously reduce that number, hence, resistance. But it also depends on the total number of Phase 0 mainnet nodes that we expect.

Mixing testnets with mainnet into one discovery churn could also be a slight mitigation to the privacy problem.

These are the concerns that I can observe. But we really don't know how impactful they are, do we? And we don't know how big is the impact of having shared DHT between testnets and mainnet on the speed of capability discovery. Perhaps @zilm13's simulation could shed a light on the latter?

[zilm13]

@mkalinin

IMO, the speed does mostly depend on the total number of peers in the DHT

Mixing two networks in one Kademlia table means you have, say, 220 peers instead of 200 (because it's exponential) in table of long running node, but if they splits equally in networks, it means you have 110 from one and 110 from another instead of 200 from one. So if you have 1/64 probability of being in one subnet plus say not all nodes are validator, let's say 1/2, it means you have 1/128 of peers in certain subnet. So before this when all nodes from one peer's Kademlia table you got 1.56 peers from one subnet, now you are getting 0.86. So you could count how many additional queries do you need to gather the same number of nodes from one subnet. Yeah, we could parry that in subnetwork context you will use not only kademlia, but anyway in standard query for nodes you get back 16 nodes and 8 of them (if networks have equal split) are not valid for you

The more peers in the DHT we have, the better network resists eclipse and sybil attacks exploiting the DHT

How will it work, when you have 24 (16*3 /2) nodes in radius of 1 distance now over you instead of 48 (16*3)?

[mkalinin]

How will it work, when you have 24 (163 /2) nodes in radius of 1 distance now over you instead of 48 (163)?

You're right. It won't save us. And it won't increase BFT of the gossip mesh as peers from other networks won't participate in it either.

[mkalinin]

Recap after Networking call 5 discussion:

• mixing peers from different network instances into a single discovery churn _does not_ improve the security of the network unless peers share the responsibilities of this network
• it results in a negative outcome instead, i.e. more round-trips when searching for peers

Potential DHT subnet denominators:

• _fork digest_ -- intends to uniquely identify the network instance but changing through time; peers will have to track several fork digests when doing discovery
• _genesis root_ -- is constant but does not identify the network uniquely after contentious fork event

[fjl]

I'm trying to understand the issue here. Bit confused how you arrived at this conclusion:

mixing peers from different network instances into a single discovery churn does not improve the security of the network

The security of the discovery is very much affected by the size of the DHT. It's the main security factor actually. If you pre-filter the node table to only include nodes of your chain, the network will become very insecure.

We can't quantify the 'adverse effect' without knowing what the actual numbers are. Mixing networks in the same discovery DHT will always lead to some degradation in peer finding performance, and it's a good tradeoff to make because the benefit of mixing the networks is better DHT security for all of them.

The question here is whether performance is good enough. How many useful nodes are you finding per minute? You can easily track this metric in your TCP dial pipeline because the ENR includes all the information to decide if the node is useful or not.

How many nodes do we need to find per minute? We can compute this number by deciding on a goal peer count. For eth2, you need to fill your connection slots with peers related to your epoch within 6 minutes. If we assume goal peer count of 20, we need to successfully connect 3 peers per minute to reach this goal. Assuming 80% dial failure rate (random number), you need to find and try ~17 peers per minute to make it to the goal.

[fjl]

Note that I just made up some numbers there. For testnets, maybe a goal peer count of 20 is unrealistic, and the dial failure rate might actually be lower. My point is that we need to focus on the things we can measure. The discovery performance we really need to make it work may be way lower than you'd think.

[djrtwo]

agreed that we need to quantify this a bit more and that carving out testnets from the dht reduces their security.

I'll leave this open until we can get some data on how quickly we can search for peers.
@AgeManning had some optimizations in mind to speed up this process

[mkalinin]

Thanks a lot for the input @fjl!

Bit confused how you arrived at this conclusion:

There was an assumption that the Eth2 mainnet has enough nodes to maintain the security of discovery service on near the same level as if its DHT contained nodes from other networks, like Eth1, testnets, etc.

The security of the _discovery_ is very much affected by the size of the DHT. It's the main security factor actually. If you pre-filter the node table to only include nodes of your chain, the network will become very insecure.

Could you please elaborate more on that or point out to an article or a paper where an impact of the size of DHT on its security is analyzed? What kind of fault tolerance is affected by the size? Is this resilience to attacks or to single node crashes?

[djrtwo]

I'm closing this. Will re-open if the launch of mainnet degrades discovery for testnets