K3s: Fedora 32 - no network access from within pods running on k3s

[christian-korneck]

After upgrading from Fedora 31 (with k3s 1.17.4 installed) to Fedora 32 (GA release) I noticed that my pods couldn't reach destinations on the network/Internet anymore.

As a quick fix, allowing NAT in the firewall helped:

firewall-cmd --zone=FedoraWorkstation --add-masquerade --permanent
firewall-cmd --reload

Version:
k3s v1.17.4+k3s1 (3eee8ac3)

K3s arguments:
defaults via systemd service unit (usr/local/bin/k3s server)

Describe the bug
With a default/vanilla k3s installation on Fedora 32 pods running on k3s don't have ext network / Internet access.

To Reproduce
install k3s on Fedora 32 (or upgrade a Fedora 31 system with k3s installed to Fedora 32), start and attach to a pod and try to ping/curl/telnet any destination on the ext network or Internet.

Expected behavior
Pods have network access

Actual behavior
Pods don't have network access

Additional context / logs
see above for workaround

[lukasmrtvy]

• afaik k3s does not handle nf_tables in Fedora32 ( Fedora31 had legacy iptables )
• also selinux and k3s policy is not compatible for Fedora31/32
• cgroupv2 is also problem for k3s

Just go with supported os...

[christian-korneck]

hi @lukasmrtvy, thanks for your comment!

Just go with supported os...

I understand that Fedora isn't the optimum OS for k3s but I've opened this issue because the k3s system requirements say _"K3s should run on just about any flavor of Linux."_, so I thought this might be a valid problem report? (also k3s worked fine for me on Fedora 31 with cgroups v2 disabled and with the optional k3s selinux policy rpm installed).

cgroupv2 is also problem for k3s

cgroups v2 can get disabled via boot cmdline param (grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" && reboot). That's a hard requirement if you use the embedded containerd (default) or the optional docker backend. (Maybe it would be helpful if this would be added to the k3s docs?)

also selinux and k3s policy is not compatible for Fedora31/32

k3s provides rpm packages for a selinux policy. Is it only intended for EL7? Are there known issues with it on Fedora?

(Also, in case that there's a critical selinux issue it should still be possible to run k3s unconfined and with the k3s flag --disable-selinux?)

afaik k3s does not handle nf_tables in Fedora32 ( Fedora31 had legacy iptables )

is it a firewalld issue? Would changing the firewalld backend from nftables to iptables resolve it? (/etc/firewalld/firewalld.conf -> FirewallBackend=iptables)

[lukasmrtvy]

• dont disable selinux! ( and sepolicy file https://github.com/rancher/k3s-selinux is incompatible with f31/32 )
• latest k3s is running 1.3.3 containerd, which is supporting cgroups v2 afaik, but k8s will support cgroups v2 in 1.19
• for iptables.., you need older than v1.8.0 or in legacy mode

still, there is no official fedora support, you are on your own..

[derba]

I play with several nodes on my laptop, so I use docker-compose based k3s. It worked fine on Fedora 31 with old cgroups. After upgrading to Fedora 32 I followed this guide and I can access external networks.
But the agents cannot communicate with the server:
level=error msg="failed to get CA certs at https://127.0.0.1:36683/cacerts: Get https://127.0.0.1:36683/cacerts: read tcp 127.0.0.1:39962->127.0.0.1:36683: read: connection reset by peer"
In Wireshark I saw "Communication administratively filtered". I found this post. But that is a temporary fix, and I wonder how that affects VPN connection.

[rytis]

@lukasmrtvy I too have problems on Fedora 31/32... See #1719. I'm not too bothered about Fedora not being supported, but how about RHEL/CentOS 8? I'm going to give it a go and see if it's any better.

[erikwilson]

I wonder if iptables could be a problem

[derba]

Sadly I do not know how firewalls work. But it is clear that the traffic on Docker's interface (not docker0) is blocked somehow. And I found a way (guide) to overcome that:

$ ifconfig | grep br
br-140b51f1acd2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.22.0.1  netmask 255.255.0.0  broadcast 172.22.255.255
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet 192.168.1.112  netmask 255.255.255.0  broadcast 192.168.1.255
$ sudo firewall-cmd --zone=trusted --change-interface=br-140b51f1acd2

Or as a 1-liner:
sudo firewall-cmd --zone=trusted --change-interface=$(ifconfig | grep br- | cut -d ':' -f1)

[vbohinc]

* afaik k3s does not handle nf_tables in Fedora32 ( Fedora31 had legacy iptables )

* also selinux and k3s policy is not compatible for Fedora31/32

* cgroupv2 is also problem for k3s

Just go with supported os...

Most of this can be avoided with the proper boot parameters, which must be edited with grubby.
grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1"

As for nf-tables:
cat < /etc/sysctl.d/90-k3s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

And iptables has to be switched to legacy mode:
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy

Kubernetes and/or k3s does not support nftables (which is the default in Fedora 32)!

[derba]

Set FirewallBackend=iptables in /etc/firewalld/firewalld.conf and restart firewalld.
https://github.com/kubernetes-sigs/kind/issues/1547#issuecomment-623756313

[vbohinc]

Tested this with Fedora 32 VM's, can confirm that inter-node communication breaks, though in my tests internet connectivity is lost only on the worker nodes...

[vbohinc]

Try setting AllowZoneDrifting=yes in firewalld.conf. In my testing this fixes the issue.

[BenTheElder]

Kubernetes and/or k3s does not support nftables (which is the default in Fedora 32)!

Kubernetes (unclear on k3s) supports iptables_nft since 1.18 and the fix is likely to be backported as far back as 1.16.