K3s: Networking breaks in --docker mode on Debian 10 Buster

[pmahoney]

Version:

0.8.0 (but same issue in 1.0.0)

(running in a docker container)

/bin/k3s server --https-listen-port 6443 --no-deploy traefik --disable-agent

(running on the host)

k3s agent  --server https://172.17.0.2:6443 --docker

Describe the bug

Inter-pod networking is broken.

(DNS queries fail, helm CRD cannot fetch charts because of DNS failure, etc.)

To Reproduce

On Debian 10, run k3s server in docker as above, exposing port 6443. Run k3s agent on the host as above. The coredns pod starts, but other pods cannot use DNS or other pod-to-pod networking (via service IP or pod IP).

Expected behavior

Networking works.

Actual behavior

On Debian 10, iptables points to iptables-nft (and another iptables-legacy is present). Install docker from docker.com's debian packages, and observe that it sets the policy of the FORWARD chain to DROP.

$ iptables -t filter -L FORWARD
Chain FORWARD (policy DROP)
...

Note iptables-legacy still has ACCEPT policy

$ iptables-legacy -t filter -L FORWARD
Chain FORWARD (policy ACCEPT)

When flannel starts in the k3s agent, it adds two rules to the FORWARD chain: one each to accept traffic with a source or destination of the pod CIDR (10.42.0.0/16). But these only appear in the output of iptables-legacy -t filter -L FORWARD and do not appear in the output of iptables-nft.

I believe the kube-proxy rules also appear in iptables-legacy and not iptables-nft

Additional context

One fix is to run the following (per https://github.com/docker/libnetwork/issues/2331) and reboot:

$ update-alternatives --set iptables /usr/sbin/iptables-legacy

Then docker also uses iptables-legacy and networking works again.

Not sure if this is a bug in flannel, kube-proxy, both, or what a proper fix is. Wanted to report it here in case someone else runs into the same issue.

[erikwilson]

It is a known kubernetes issue that iptables should be in legacy mode (or pre 1.8.0).

[outofcoffee]

FWIW I'm _not_ running k3s in docker mode, but I had the same issue (Debian 10, broken pod networking; can't hit any IPs outside the cluster etc.)

The workaround of switching iptables to legacy mode worked for me too. The flavour of Debian I'm using is Raspbian Buster.

Might be worthwhile updating the title of this issue to remove the 'in --docker mode' bit.