Tendermint: e2e: control the level of connectedness in networks

[cmwaters]

Problem Definition

Currently, generated e2e tests are capped with the size of the network (at most 3 seed nodes, 10 validators and 4 full nodes) predominantly because larger networks would be an increased strain on the cpu. This means that the network usually and quite quickly becomes fully connected. In comparison, main nets will have hundreds of validators and full nodes that are not fully connected with one another.

Proposal

To properly test the "gossip" characteristic of tm networks, I would propose that the e2e allows configuration of MaxNumInboundPeers and MaxNumOutboundPeers. This is by default set to 40 and 10 respectively, but I believe if we were to lower it below the total size of the network that we could emulate the kind of partial connectedness that large nets experience.

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[erikgrinaker]

Note that setting these will cause non-deterministic connectivity, such that rerunning the same network locally will cause a different connection topology. It might be better to explicitly construct the network topology, which overlaps somewhat with the need to set up specific kinds of topologies that are often seen in the wild (e.g. sentry nodes and NATed nodes).

[cmwaters]

Hmm you're right. We could also set these values to 0 and use persistent peers only which I believe would construct a static, and thus deterministic, network topology. I do believe that it would also be valuable to test the dynamism/robustness of the network and the pex by having partially connected networks subject to a series of perturbations by the nodes even if this does come at the cost of determinism.

[erikgrinaker]

I think it'd be better to construct the actual network topology via Docker and/or iptables, rather than relying on Tendermint behavior that may or may not be working as intended. This also allows for testing of additional failure modes, e.g. how a node handles lots of gossiped addresses that it's unable to reach, and that private network addressed should not be advertised to peers connected via public networks.

I agree that we'd still want to run perturbations in these networks, to make sure they are robust to additional failures.