Metallb: Service type LoadBalancer sets an ip but deployment is unreachable

[RdL87]

Is this a bug report or a feature request?:

bug

What happened:

wget http://90.147.161.128:2368/
--2018-08-08 14:47:03-- http://90.147.161.128:2368/
Connecting to 90.147.161.128:2368... failed: Connection timed out.

What you expected to happen:

Connecting to http://90.147.161.128:2368/

How to reproduce it (as minimally and precisely as possible):

kubectl run ghost --image=ghost --port=2368
kubectl expose deployment ghost --port=2368 --target-port=2368 --type=LoadBalancer

Anything else we need to know?:

This is the state of the ghost deply:

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/ghost 1 1 1 1 29m

NAME DESIRED CURRENT READY AGE
rs/ghost-7b7d68d64c 1 1 1 29m

NAME READY STATUS RESTARTS AGE
po/ghost-7b7d68d64c-gn59c 1/1 Running 0 29m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ghost LoadBalancer 10.152.183.238 90.147.161.28 2368:30161/TCP 30m

This is the state of the metallab deployment:

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
ds/speaker 4 4 4 4 4 3h

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/controller 1 1 1 1 3h

NAME DESIRED CURRENT READY AGE
rs/controller-67cb74b4b5 1 1 1 3h

NAME READY STATUS RESTARTS AGE
po/controller-67cb74b4b5-gxqdm 1/1 Running 0 3h
po/speaker-6gtwk 1/1 Running 0 3h
po/speaker-hxrgv 1/1 Running 0 3h
po/speaker-tmvl5 1/1 Running 0 3h
po/speaker-xvm49 1/1 Running 0 3h

Last logs lines from the controller:

Environment:

• MetalLB version:
  Image: metallb/controller:v0.7.2
• Kubernetes version:
  GitVersion:"v1.11.1"
• BGP router type/version:
  Not using BGP

Config FILE:

apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: public-ip-space
protocol: layer2
addresses:
- 90.147.161.28/32

• OS (e.g. from /etc/os-release):
  NAME="Ubuntu"
  VERSION="16.04.4 LTS (Xenial Xerus)"

• Kernel (e.g. uname -a):
  4.4.0-64-generic

[RdL87]

I tried also this one https://github.com/google/metallb/blob/master/manifests/tutorial-2.yaml
but i get the same behaviour. Thanks in advance for helping me

[RdL87]

Some steps ahead: i can access using the public address from the worker nodes. Not from the master. Note that i've deployed my k8s cluster using Juju and the bundle of canonical (https://jujucharms.com/canonical-kubernetes/). I say this because in this bundle the master node is different from the worker node:

• https://github.com/kubernetes/kubernetes/blob/release-1.11/cluster/juju/layers/kubernetes-master/reactive/kubernetes_master.py#L243

• kubernetes-workers have kubelet and kube-proxy installed:
  https://github.com/kubernetes/kubernetes/blob/release-1.11/cluster/juju/layers/kubernetes-worker/reactive/kubernetes_worker.py#L153

I think this explains why i cannot reach the deployment through the service from the master and from outside the cluster.

[RdL87]

I could reach the deployment from outside using the public address by adding a public address to the worker node that host the deployment. But i'm not sure this one could be a nice solution. Any suggestion?
Thanks

[danderson]

Thanks for the report.

First off, I suggest not bothering to check connectivity on the nodes themselves. That data path is completely different, MetalLB is not involved at all. If that path doesn't work, then it's a good indicator that something not-MetalLB is broken, but the fact that it works doesn't tell you anything about MetalLB. I need to document that fact better :).

I'm not sure I understand your network setup. Could you describe it a little more? You've configured MetalLB with a public IP address, and in the past when I've seen that it's part of some hosting provider's infrastructure, where some provider-specific configuration is required - or some provider-specific network security stuff gets in the way of MetalLB. So, I need to better understand the layout your have. In particular, can you describe the traffic path from the internet to the worker node, assuming MetalLB is working correctly?

Can you also provide the output of kubectl logs -nmetallb-system -lcomponent=speaker ? That will show us if MetalLB is responding to layer2 requests at all (which would indicate some other problem in the cluster).

Are you running any kind of non-kubernetes firewall, such as firewalld or ufw? In general those interfere with Kubernetes's networking, so that might be a reason for problems.

Based on the data you provided, your cluster seems mostly fine, so we're looking at either some unclear documentation that led you astray, or a bug.

I'm going to file a sibling bug to finally do what a friend suggested to me months ago, write troubleshooting documentation and a diagnostic tool, so that it's easier for folks like you to understand what MetalLB is doing and why it might be failing :). But let's continue debugging together as well.

[RdL87]

Hi Dave,
thank you for your support. I try to give you a clearer picture:

• we have a datacenter on top of which we deployed a k8s cluster
• we are in control of the physical infrastructure including the edge routers
• there is no firewall blocking the traffic both on the physical machines and on the routers
• we used juju as a deployment tool (canonical bundle) that by design does not install kubelet on the master node
• we are testing MetalLB as a LB to give users access from outside to deployments
  as it is, MetalLB does not receive connections. As a workaround (given the lack of kubelet on the master node) we added a public interface on the hosting worker node.
• following you can see the logs as suggested
  speaker_log.txt

I wonder how/if the workaround has an impact on the security side.

Thank you in advance for any further suggestion

[danderson]

Okay, looking at the logs, MetalLB is receiving ARP requests for your service IPs, and is sending responses. So, from MetalLB's perspective at least... Everything is working correctly, we're telling the network where to find your service.

So, at this point, all I can offer is some generic suggestions for debugging networking issues :/

One big thing I see on your machines is many interfaces, and you say that you added a public interface. That triggers two immediate thoughts in my mind: split routing, and reverse path filtering.

Split routing would happen if you are receiving traffic on your "public" interface, but the default route on the machine points to the "private" interface. If your private network cannot route the responses back to the client, the service will just time out, because TCP connections cannot establish. An easy way to see if split routing is happening is to tcpdump on the worker node, something like tcpdump -v -e -n -i any host <service ip>, and try to connect to the service IP from a client. If the SYN/ACK packet (looks like [S.] in tcpdump) is being sent to a different MAC address than the original SYN came from (looks like [S] in tcpdump), then you have a split routing situation.

You can resolve that kind of problem a couple of ways, but I suggest talking to the network engineers in your company to help you figure out the best solution. You can either make the asymetric return path work correctly (by tweaking your network setup), or you can do policy routing on the k8s worker node so that connections that initiated on the public interface use an alternative routing table that transmits responses on the public interface. Or, of course, just add a specific route just for your clients, if all your clients are coming from a well-known IP range.

Unfortunately all of this is out of scope for MetalLB, so I can only point you in the direction of the problem.

The other potential issue is reverse path filtering, which is a security feature that essentially blocks any traffic that would be split-routed. It enforces that you can only receive a packet, if the response to that packet would go out the same interface. Split routing obviously breaks this :). If that's your problem, you can set RPF to loose mode on linux, which will accept the packet if the machine can respond using _any_ interface. This still protects you against IP spoofing, and allows split routing to happen (which is desirable in complex network setups).

[RdL87]

Hi Dave,
sorry for the delay, I just came back from holidays.
Thank you for your answer, unfortunately i think i've been a little bit unclear in my previous msg and i caused confusion.
I opened the issue cause metallb didn't work on our cluster. After that i got the problem and i added a public interface on each worker node as workaround to make it working. And actually it works fine. I was wondering if this workaround, in your opinion, could have an impact on the security side.
Sorry for the misunderstanding

[danderson]

Sorry, but I no longer have the time to debug MetalLB deployments for people. I want to keep this tracker specifically for bugs and features, so I'm going to close this. At some point I may have time to write general troubleshooting documentation, but as a first step towards that I need to clear up the issue tracker.

You could try asking for help on the #metallb channel on the Kubernetes slack, where there are people other than me who can assist.

[conexia-it]

la conf de metallb debe ser por bgp al enrutador.
conf l2 solo es accesible desde el nodo.