Metallb: Layer 2: impossible to use MetalLB on a subset of Kubernetes nodes.

[adidenko]

Is this a bug report or a feature request?:
Feature request (possibly a regression since v0.6)

What happened:
If you run a service with externalTrafficPolicy: Cluster MetalLB Speaker will run announcer only on nodes with at least one ready endpoint. Which makes perfect sense for externalTrafficPolicy: Local, but not required for services with externalTrafficPolicy: Cluster.

Due to this it's impossible to use MetalLB with layer2 protocol on a subset of Kubernetes workers. It will work only if Speaker is running on all the nodes.

What you expected to happen:
It's possible to run MetalLB on a subset of k8s nodes and it runs Layer2 announcer on a leader node for services with externalTrafficPolicy: Cluster.

How to reproduce it (as minimally and precisely as possible):

1. On a 2+ node cluster run MetalLB speaker with nodeSelector set to a part of nodes.
2. Run the Nginx deployment and service from https://raw.githubusercontent.com/google/metallb/v0.7.2/manifests/tutorial-2.yaml with nodeSelector set to the other part of nodes. I.e. make sure Nginx pod is not running on the same node as MetalLB speaker.
3. Speaker is not responding to ARP who-has requests for external IP.

Anything else we need to know?:
The typical use case is border LB/ingress nodes. Which are supposed to terminate external traffic and load balance / ingress into the k8s cluster. Such nodes usually do not run any workloads.

Environment:

• MetalLB version: v0.7.3
• Kubernetes version: 1.11.2
• BGP router type/version: layer2
• OS (e.g. from /etc/os-release): CentOS Linux release 7.5
• Kernel (e.g. uname -a): 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[adidenko]

It looks like there's no easy fix for this with the current (0.7.x) design. In order to implement this feature properly Speaker needs to either have leader election (per service or global like in 0.6.x) or some other external control mechanism (from Controller?).

So basically, it's not possible to use MetalLB with layer2 protocol on a subset of Kubernetes workers. It will work only if Speaker is running on all the nodes.

[kfox1111]

bummer. I was hoping to do this too.

[kfox1111]

I actually don't need externalTrafficPolicy: Local. would it be simple to patch (maybe with a command line flag) to change it to work with Cluster only by running it on all nodes always. Just until a better fix can go in?

[danderson]

As you can tell, I haven't had time to dig into this more lately. Yes, the way 0.7 refactored things to enable the local traffic policy makes it hard to support running MetalLB on only part of the cluster. To be fair, I don't think that's a configuration I ever explicitly said I supported. All of MetalLB assumes that any node in the cluster could serve traffic, under the right conditions.

That said, partial node coverage is a reasonable configuration, and we should support it. For now the quickest thing I can suggest to make this work is to use MetalLB 0.6. Really the only change in 0.7 is this changed handling of layer2 mode, so if you just revert to 0.6 your configuration will work while I figure out how to support partial node coverage for 0.8.

Right now, it's looking like I will have to do some kind of active probing between speakers anyway to get nicer failover properties in Local mode. So, given that, partial node coverage would just show up as fewer nodes eligible to carry traffic, and the current deterministic selection logic would take it from there.

[danderson]

This won't be in 0.8, because I'm releasing 0.8 with "whatever is in master right now", and this isn't fixed in there. Sorry. Still want to fix it, but unknown timeline.

[piyushns]

This feature is indeed useful as we too will only expose a portion of the nodes to internet. Will try to use the 0.6.2 version till this is incorporated.

[cjreyn]

+1 for this feature