Metallb: Install routes learned via BGP

[mhristache]

Is this a bug report or a feature request?:
feature request

What happened:
I am trying to get MetalLB to work in a setup where I have different interfaces for OAM and traffic on the K8s nodes.

K8s traffic is supposed to use the traffic interface but the default route is via OAM interface.

In order to get return traffic to work, I have to add static routes for all possible clients of the K8s Services.

However, MetalLB is receiving all those routes via BGP from the uplink router but it does not install them on the K8s node.

What you expected to happen:
I would like MetalLB to install and remove BGP routes received from the BGP peers, on the K8s nodes so that manual static routes are not needed.

I guess default routes received from the BGP peers should be ignored to avoid breaking the OAM routing.

How to reproduce it (as minimally and precisely as possible):
-

Anything else we need to know?:
-

Environment:

• MetalLB version: 0.6.0
• Kubernetes version: 1.10
• BGP router type/version: not relevant
• OS (e.g. from /etc/os-release): not relevant
• Kernel (e.g. uname -a): not relevant

[danderson]

Currently, this is "working as intended." MetalLB is deliberately not a full BGP router, because once you start down that road, it never ends with more and more elaborate support for more and more elaborate BGP features.

MetalLB's BGP support is designed to be a "BGP injector", that pushes a small amount of state into a BGP backbone without participating in it.

Unfortunately, given the development and support resources available (i.e. me), this is not going to get fixed. My recommendation is that, if you need the nodes to fully participate in a BGP network, you should use a BGP-aware Kubernetes network addon, and integrate MetalLB with that. Romana is currently the best network addon for that, in the sense that we have fully documented how to run Romana's BGP functionality and MetalLB alongside it.

Alternatively, you could run BIRD yourself as a Kubernetes DaemonSet, configured to your liking, and peer MetalLB with it (the Romana instructions are the right thing to do in that case as well). This gives you total control over the BGP router configuration and policy, which it sounds like you need in order to make correct routing decisions in your network topology.

[lynchs61]

Quick question about the answer you provide. You state that an alternative to Romana would be to run BIRD in a daemonset and peer MetalLB with that. This makes sense to me because as you said, MetalLB isn't a full BGP router.

But then I think about the original question.. "I would like MetalLB to install and remove BGP routes received from the BGP peers, on the K8s nodes..."

Since MetalLB is just a BGP injector it obviously can't do that. But I don't understand how BIRD can do it either. I get how BIRD can learn the routes from the BGP peers. But how does BIRD install the routes on the K8s nodes?

This has taken me to the limits of my networking knowledge, especially in a K8s cluster, so i'm probably missing something important. But some clarification would be appreciated.

[mhristache]

@lynchs61 you can run the BIRD daemonset with hostNetwork: true so it can change the routing table on the host.

[mhristache]

You can use this Kubernetes deployment file as example.

[raiviskrumins]

or you can use kube-router with metallb: https://github.com/google/metallb/issues/160#issuecomment-384739716