Metallb: [Question] How to avoid service disruptions in L2 mode

[liuyuan10]

Currently in L2 mode, master is selected by sorting the hashing of node+service name so that each speaker can do this by themselves. One downside I can see is that when a node is added, it may change master of some services (even all is possible) causing service disruptions. What's worse is when a node is restarted, it causes two service disruptions in a row for those with masters on the node.

So I'm wondering if we can have sticky master selection in some way. One straightforward way is to do master selection at a centralized place (like the controller) which can stick a service to a node. The reason not doing so must have been clear but some quick answers are appreciated.

[rata]

Just curious, how much time does this disruption take and using which OS? From https://metallb.universe.tf/concepts/layer2/:

All modern versions of major OSes (Windows, Mac, Linux) implement layer 2 failover correctly, so the only situation where issues may happen is with older or less common OSes.

Just to understand the impact of the current behavior. Of course, improving is always good and thanks for noticing this use case that can be improved :-)

[liuyuan10]

Here I'm talking about existing TCP connections got disrupted because connection state is missing on a new master. Gateway doesn't play a role here. When a new node shows up, some of services may be migrated to this node but existing TCP connection states are not (kept in conntrack in most cases). If we can have sticky master election, a new node will not causing such migration

[champtar]

Grafana team implemented a small KV store on top of Memberlist: https://grafana.com/blog/2020/03/25/how-were-using-gossip-to-improve-cortex-and-loki-availability/
That might be a way to exchange info and have "sticky" nodes

[guoh1988]

@liuyuan10
Layer2
“Local” traffic policy

With the Local traffic policy, kube-proxy on the node that received the traffic sends it only to the service’s pod(s) that are on the same node. There is no “horizontal” traffic flow between nodes.

Because kube-proxy doesn’t need to send traffic between cluster nodes, your pods can see the real source IP address of incoming connections.

The downside of this policy is that incoming traffic only goes to some pods in the service. Pods that aren’t on the current leader node receive no traffic, they are just there as replicas in case a failover is needed.
The program pods of the udp server run on 192.168.1.105 and 192.168.1.106 respectivelyk8s service is service ExternalTrafficPolicy = Local mode，192.168.1.105 is leader node，When the pod program on 192.168.1.105 exits or restarts，192.168.1.106 will become the leader node，But wait for the pod of 192.168.1.105 to start ok，leader node will become 192.168.1.105，Its switching will cause a service interruption

[russellb]

It looks like this discussion has been quiet for a good while now, so I'm going to close this out. #metallb on kubernetes slack is another good place to discuss further. Thanks!