Metallb: Load balancing questions

[runningman84]

Based on your great podcast I have a few questions:
If you have multiple services in layer 2 mode is the whole traffic routed through one instance? It would be cool if metallb would distribute the traffic to multiple instances. This would allow users to define the same service multiple times like foo-a foo-b and balance them via dns.
Does metallb prefer instances where a local pod is available? Imagine you have one Mailserver pod it would make sense that the corresponding metallb would be on the same instance...
How does the traffic flow look like with networking solutions like kube-router? I would think in a DSR mode only the inbound traffic would go through metallb but the outbound traffic would flow directly from the instances?

[danderson]

Breaking out your questions a bit...

Which nodes receive traffic in L2 mode?

In L2 mode, each service independently picks one "leader" node that owns the IP. The selection is pseudorandom, based on a hash of a bunch of information in the service. So, if you have 2 machines and 100 services, _in general_ each machine will end up owning ~50 services. It's not a 100% guarantee, because to avoid lock contention and SPOFs these decisions are made using a stateless algorithm, so we're relying on probabilities instead of certainty.

Distributing traffic to multiple instances in L2 mode

This is currently a bit difficult, because as I said above the "spreading" of different services across nodes is pseudorandom, so there's no way to say specifically "I want these 2 services to be on different nodes". Recently on Slack I had a discussion with someone (can't find their github ID :( ) about extending MetalLB so you could say, for one service, "please give me 4 IPs for this service, on different nodes". It would be _relatively_ simple to implement this, and it would give you exactly what you wanted for DNS balancing. So, this might happen at some point soon, I'm waiting for that person to create a bug with our design discussion so we can figure out the details.

Does MetalLB prefer instances where a local pod is available?

Currently in L2 mode, _only_ nodes with a local pod are eligible to become the announcer for a service. So, the announcer will always be on the same node as at least 1 target pod for the service. Depending on the setting of externalTrafficPolicy on the service, when the node receives the traffic it might forward it only to the local pods (Local mode), or redistribute it to all pods in the service (Cluster mode).

I'm actually going to change this a bit in the next release, because this logic (introduced in 0.7) broke some use cases of MetalLB where you have dedicated "announcer" nodes that forward to pods on different nodes. In a future release, when the service is in Cluster mode, MetalLB will _prefer_ nodes with local pods if possible, but if not possible it'll still allow using any other node.

How does the traffic flow look like with kube-router?

I actually have no idea. For most network addons, it's actually kube-proxy on the node that handles LoadBalancer traffic, and in those cases there's strict DNAT and connection tracking, so if the inbound traffic does world -> node A -> node B -> pod, the return traffic will go pod -> node B -> node A -> world.

However, kube-router is unusual in the network addon world because it replaces the functionality of kube-proxy. So, it's possible that it's doing "correct" DSR, where the outbound path is just pod -> node B -> world. But, doing "correct" DSR can be challenging, because it can cause all kinds of network issues both on the nodes (if they enable strict reverse path filtering, like Debian does) or on upstream routers (if they're doing conntrack and see traffic flowing in a way they don't expect). We would have to ask the kube-router folks for details on how they handle LoadBalancer traffic.

[runningman84]

Thanks for the quick and detailed response... you might want to add these answers to some faqs in your docs...

[MoJo2600]

I have another question about the externalTraficPolicy setting. I have a pod running and would like to route the clients IP address to the pod. As i understand it, the "Local" traffic policy would be the correct setting.
What i don't understand is the documentation. It states: "The downside of this policy is that incoming traffic only goes to some pods in the service. Pods that aren’t on the current leader node receive no traffic, they are just there as replicas in case a failover is needed."

I understand, the pod has to be run on the same host (k8s node) as the leader. But can i somehow enfoce the pod to be run on the leader node? Is there a way to make sure the pod that uses the service is run on the same node as the leader?

Best regards and thanks for this awesome piece of software!

[runningman84]

I guess the leader can only be a machine which has running at least one corresponding pod.

[danderson]

There's no need to ensure anything, this happens automatically :). Leader election happens per-service, and only nodes that have at least one ready pod (passing all configured healthchecks) are eligible to become leader for the service. So, leadership will automatically move around as your pods do, and will always stick with your pods.

What the documentation is trying to say (and please send a PR if you can think of a way of clarifying it!) is that L2 mode cannot do load _balancing_ in the "Local" traffic policy mode, only failover. If your cluster has 10 nodes, and your service pods are on 5 of them, the leader will be one of those 5 nodes... But the other 4 will receive no traffic at all, they'll just sit idle. If the one node receiving traffic goes away or becomes unviable, one of the other 4 will take over.

This is in contrast to BGP mode, where we can do true load _balancing_ across all nodes with viable pods, even in the Local traffic policy mode.

[MoJo2600]

Thank you for the clarification! I tested it and it is working as expected. I had an issue in my service beforehand.