Microk8s: Stoping or removing microk8s does not clear iptables

[jareks]

Installing and starting microk8s adds rules to host's iptables but stopping / removing snap does not remove them (without reboot).

---

Version: microk8s v1.16.0 2019-09-25 (920).
Tested on ubuntu/xenial64 with Vagrant.

# no kube rules in iptables at start
$ sudo iptables -L -v | grep -i kube | wc -l
0

# install microk8s and notice 15 new iptables rules
$ sudo snap install microk8s --classic
$ sleep 15 # wait for microk8s to start
$ sudo iptables -L -v | grep -i kube | wc -l
15

# stop microk8s and notice that rules are still up
$ sudo iptables -L -v | grep -i kube | wc -l
15

# remove snap and notice that rules are still up
$ sudo snap remove --purge microk8s
$ sudo iptables -L -v | grep -i kube | wc -l
15

If you reboot the machine the rules are finally removed.

(Possibly the problem is not with iptables rules itself, but with k8s network interface(s) not being removed)

[rzr]

Should a wrapper script handle this ?

[hernil]

I'm dropping this here since I did not find anyone mentioning this on the internet so far.

-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP

This iptables rule created by microk8s might interfere with WireGuard using wg-quick. I'm guessing it's because wg-quick adds a fwmark to the packets and this rule catches that but this is where my networking fu stops :smile:

The rule can be reverted by running

sudo iptables -D KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP

However microk8s does a great job reinserting the rule as long as it is running, you therefore need to stop the service for this to help.

Hopefully this will save someone some head scratching later on!

IMO all iptable rules created by the microk8s service should be reverted upon stopping it.

[bingh0]

I recently came across this issue and spent a week tearing my hair apart before finding this issue. I can confirm this issue with Ubuntu 20.04 and Fedora 32. Has anybody figured out a workaround for this? I'm surprised so few people have reported this - I would imagine there are people running microk8s and wireguard.