weave network of 100 machines

[rachit1arora]

Hello

We have a requirement of having a weave networking involving 100+ machines .
We are currently using weave version 1.0.1 . In this we see that we have a default limit of 30 peers .

I want to get your opinion on overriding this value of 30 peers to 100 .

It will be great if any one can answer our following questions

1. Is it possible to override the default value of 30 peers to 100 ? If yes how ?
2. What are the implications of this in terms of performance and availability of weave network ?
3. Is there any other way to bring 100 machines under weave network so that we can seamlessly spwan containers on these 100 machines and they can talk to each other

Thanks in Advance .

[rade]

default limit of 30 peers

The limit is actually on the number of _connections_ a peer has to other peers.

Is it possible to override the default value of 30 peers to 100 ?

weave launch --connlimit=100

What are the implications of this in terms of performance and availability of weave network ?

Connections take up memory, CPU and network resources. So there are limits. But 100 should be fine.

Is there any other way to bring 100 machines under weave network

Yes, peers do not need to be all directly connected to each other in order for a full network to form, since weave performs multi-hop routing. In order to take advantage of that you'd have to carefully construct a suitable topology where each peer has fewer than 30 connections and can reach any other peer indirectly. That will likely require disabling weave's built-in peer discovery with --no-discovery.

[justinsb]

Sorry to bump an old issue, but this is the number of control plane connections, and it doesn't impact the ability of nodes to communicate with each other directly?

More concretely: we can still have a cluster of (say) 1000 nodes with CONN_LIMIT=30, and each node will only gossip directly with 30 others, but the 1000 nodes will each connect to each other directly for application traffic. Is that right?

[bboreham]

Hi @justinsb right now the control plane and data plane must match, so it does impact the ability to communicate directly. However we do multi-hop routing so the packets would still get there through an indirect connection, just a bit slower. Issue #374 is about allowing them to be different.

With a full mesh (i.e. 1000 nodes each with 1000 connections) gossip will go to 10 random nodes each time (it's Log2()). The issue with 1000x1000 connections is the shortest-path routing computation will get expensive.

[bboreham]

Note the default connection limit was raised from 30 to 100 in #3234, released in version 2.2.1

[avarf]

Hi, I like to know is there any table or reports that give me the limitations of Weave Net regarding the maximum number of machines, the resource usage per adding 10 machine, etc? At the moment we are using Weave Net as CNI for our small in house K8s cluster and we run our platform on it but we like to know the limits or requirements in case if we want to add more machines to the cluster.

[murali-reddy]

@avarf anectodally (from GitHub issues and slack conversations) many kops users running on AWS were able to run upto 150 nodes with default resource settings. There are performance issues (e.g. https://github.com/weaveworks/weave/issues/3593, https://github.com/weaveworks/weave/issues/3595) beyond 150 nodes which are getting addressed in 2.6 release.

[avarf]

Thank you @murali-reddy for your answer.