Weave: Weave Net breaks when host OS uses iptables 1.8

Created on 1 Dec 2018 · 20Comments · Source: weaveworks/weave

What happened?

I installed Weave Net on a test cluster where the host OS is Debian Buster (aka Debian testing, rolling distro with the ~latest version of everything). After installing Weave, pod<>pod and pod<>internet communication is completely broken, no traffic flows at all, even between pods on the same machine.

I root-caused it to an incompatibility in iptables versions between the weave pods and the host OS. Debian Buster now ships iptables 1.8. It has a major change, which is that the iptables command is now a translating facade on top of nftables, i.e. creating rules with iptables or iptables-restore actually programs nf_tables in the kernel.

OTOH, the weave pod contains iptables 1.6, the previous stable release which programs the "classic" iptables netfilter stack. So, docker on the host OS ends up programming nf_tables rules (because it uses the host iptables 1.8), and weave ends up programming legacy iptables rules (because it uses iptables 1.6). For some reason I don't fully understand, having both programmed causes packets to get dropped instead of forwarded on the host, before the packets get transmitted to the target container.

How to reproduce it?

I filed extensive reproduction steps in a sibling bug with the Calico folks, please refer to https://github.com/projectcalico/calico/issues/2322#issuecomment-443402871 . The only changes for weave are to use Weave's pod-network-cidr in kubeadm, obviously install weave instead of calico, and then some of the output changes slightly like different interface names. Everything else, including the core failure mode and the fix, plays out the same.

Additionally, the hacky steps to verify that iptables 1.8 is the problem are at https://github.com/projectcalico/calico/issues/2322#issuecomment-443405705 - basically hackily overwrite the iptables binaries with the ones from debian stable (which still uses 1.6), reboot the machine, and Weave starts working perfectly again.

Anything else we need to know?

There's also a bug tracking similar problems in core k8s, at https://github.com/kubernetes/kubernetes/issues/71305 . In core k8s this mismatch breaks kube-proxy, but it's the exact same root cause, mismatched iptables versions.

Versions:

$ weave version

Whichever the latest one is - my test harness is running calico atm, can't look it up.

$ docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:43 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:06 2018
  OS/Arch:          linux/amd64
  Experimental:     false

$ uname -a

Linux cluster1-controller 4.18.0-2-amd64 #1 SMP Debian 4.18.10-2 (2018-11-02) x86_64 GNU/Linux

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

[componenkube] bug

Source

danderson

❤3 👍1

Most helpful comment

https://github.com/kubernetes/kubernetes/issues/71305 describes the same issue with kube-proxy. The comments there also include some workarounds, e.g. setting the iptables tool to legacy mode (update-alternatives --set iptables /usr/sbin/iptables-legacy).

praseodym on 1 Sep 2019

👍3

All 20 comments

thanks for reporting this issue @danderson

So, the root cause definitely seems to be mixing iptables 1.6 and iptables 1.8 against the same kernel. If you use all iptables 1.6, everything is fine. I'm guessing if you use only iptables 1.8 (which translates into nftables but faithfully emulates the userspace interfaces), everything would also work fine. But with the host OS using iptables 1.8 (which programs nftables) and containers like calico-node using iptables 1.6 (which programs legacy iptables), packet forwarding seems to break.

Latest image of Alpine that weave uses still has iptables 1.6.1 so updating the host iptable binaries should be a way to workaround this issue.

murali-reddy on 3 Dec 2018

👍1

Just had another user hit this.
I see that Calico added an explicit setting to request NFT mode.

bboreham on 29 Aug 2019

Just updated to buster and have the same problem with buster's docker.io. States working fine but pinging is impossible.

Because buster is stable now it would be necessary to have a weave version minimum build against alpine 3.10.

cy8aer on 1 Sep 2019

Built it on my own against alpine 3.10 with iptables 1.8 - does not work. We need non legacy iptables rule sets for buster.

cy8aer on 1 Sep 2019

praseodym on 1 Sep 2019

👍3

Yes I now also run the machines with iptables-legacy via update-alternatives and this works for me. This is a problem on other systems than docker/weave too.

Probably it would make sense to write a warning and description of update-alternatives in the installation documentation.

cy8aer on 1 Sep 2019

Yes I now also run the machines with iptables-legacy via update-alternatives and this works for me. This is a problem on other systems than docker/weave too.

Probably it would make sense to write a warning and description of update-alternatives in the installation documentation.

Definitive, waste half a day on this issue
https://www.weave.works/docs/net/latest/kubernetes/kube-addon/#-things-to-watch-out-for

HaveFun83 on 6 Oct 2019

@HaveFun83 would you like to make a PR which presents the information in a way that would have worked better for you?

bboreham on 7 Oct 2019

Fixed in kube-proxy to auto-detect the mode and invoke update-alternatives accordingly

https://github.com/kubernetes/kubernetes/pull/82966

murali-reddy on 9 Dec 2019

@murali-reddy was this fixed by #3747, or should we expect further changes?

drigz on 8 May 2020

It's believed to be fixed in release 2.6.1; I don't know why this issue didn't auto-close.

bboreham on 8 May 2020

I've installed several hosts mixing Alpine and Debian.
All hosts using iptables legacy work; all using nf_tables don't.
I'm using weavenet 2.6.2 downloaded today.