Metallb: Layer 2 mode on RPi

Created on 28 Jul 2018 · 4Comments · Source: metallb/metallb

Is this a bug report or a feature request?:
bug

What happened:
I'm trying to use layer 2 routing with RPi setup. However, I can't reach exposed service from outside of cluster.

BGP works as expected but it's a home network and I don't want to overcomplicate things.

More details:

from pod to pod, using service IP, everything works
from node to pod, using service IP, everything works
from outside of the cluster I'm unable to access exposed service

arping + tcpdump:

from outside -- no luck. Details below*.
from router -- no luck. Router is definitely not blocking ARP traffic and I'm able to arping physical RPi.
from pod to service -- works
from node to service -- works only through networks' plugin interface, "weave" in my case

I've checked that IPVS is not enabled, tried flannel and WeaveNet, tried 1.9.3 and 1.11 k8s versions with the same result.

Speakers are correctly responding when requests are actually coming. During the startup, all interfaces are used for ARP responders, so thins seems to be correct. No errors in logs either.

iptables are updated

-A KUBE-SERVICES -d 192.168.0.110/32 -p tcp -m comment --comment "kube-system/traefik:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-X3WUOHPTYIG7AA3Q
-A KUBE-SERVICES -d 192.168.0.110/32 -p tcp -m comment --comment "kube-system/traefik:https loadbalancer IP" -m tcp --dport 443 -j KUBE-FW-IKNZCF5XJQBTG3KZ

*More details about outside arping:

From speaker logs:

{"caller":"announcer.go:87","event":"createARPResponder","interface":"wlan0","msg":"created ARP responder for interface","ts":"2018-07-28T00:20:15.560229854Z"}

From physical RPi:

pi@k8s-agent-1:~ $ sudo tcpdump -i wlan0 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:53:01.746441 ARP, Request who-has 192.168.0.110 tell vlad-mp, length 44
00:53:01.750945 ARP, Request who-has k8s-agent-1 tell DD-WRT, length 28
00:53:01.750993 ARP, Reply k8s-agent-1 is-at b8:27:eb:01:a6:73 (oui Unknown), length 28
00:53:02.772621 ARP, Request who-has 192.168.0.110 tell vlad-mp, length 44
00:53:03.794473 ARP, Request who-has 192.168.0.110 tell vlad-mp, length 44
00:53:04.003397 ARP, Request who-has tv-vizio tell vlad-mp, length 28
00:53:04.815499 ARP, Request who-has 192.168.0.110 tell vlad-mp, length 44

ip addr:

3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:01:a6:73 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.101/24 brd 192.168.0.255 scope global wlan0
       valid_lft forever preferred_lft forever

Nothing shows in Speakers logs during these attempts.

Environment:

MetalLB version: 0.7.1
Kubernetes version: 1.93 & 1.11
BGP router type/version: layer2
OS (e.g. from /etc/os-release): Raspbian Stretch
Kernel (e.g. uname -a): armv7l GNU/Linux

Source

vkorn

Most helpful comment

Ok, seems that it's a duplicate of #253.

I was under impression that M (multicast) stands for promiscuous as well, which is not the case, my bad. After setting it explicitly

sudo ifconfig wlan0 promisc

Everything works as expected. Turning off results in immediate stop of announcing.
Would be great if this could be added to documentation :)

vkorn on 28 Jul 2018

👍4

All 4 comments

Ok, seems that it's a duplicate of #253.

I was under impression that M (multicast) stands for promiscuous as well, which is not the case, my bad. After setting it explicitly

sudo ifconfig wlan0 promisc

Everything works as expected. Turning off results in immediate stop of announcing.
Would be great if this could be added to documentation :)

vkorn on 28 Jul 2018

👍4

Just encountered this myself, needed to enable promisc mode on wlan0 in order for ARP requests to make it from: PC -> Router -> RPI wlan0 -> arp.go response

Would be awesome if this can be documented as I spent a while messing around until I stumbled across this! :)

michael-robbins on 24 Dec 2018

Seems to be related to this issue https://github.com/raspberrypi/linux/issues/2677

pulpo on 21 Jun 2019

I'm seeing this exact issue, but through eth0 and not wlan0.
I'm running k3s v0.10.2 on raspberry pi 3 and 4.
Setting the eth0 to promisc mode does not seem to help. I am still unable to reach the external IP from machines outside the cluster. I could really use some guidance here.