Microk8s: Pods don't have internet access

[mogwai]

I have a single node cluster running on a Ubuntu 20.04 machine. None of the pods are able to ping ips or resolve dns requests. The coredns kube-system pod is able to receive requests but isn't able to resolve them:

[INFO] 10.1.72.109:58333 - 12496 "AAAA IN google.com. udp 28 false 512" NOERROR - 0 2.000540708s
[ERROR] plugin/errors: 2 google.com. AAAA: read udp 10.1.72.108:38219->8.8.4.4:53: i/o timeout

inspection-report-20200813_094313.tar.gz

cat /proc/sys/net/ipv4/ip_forward is 1

iptables -P FORWARD ACCEPT has no effect

ufw status is inactive

[balchua]

Do you have proxy?

[mogwai]

No I haven't setup a proxy on the machine. I'm able to perform dig, curl and pings on the node without being in a container.

[mogwai]

Here are the iptables on the node:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-FIREWALL  all  --  anywhere             anywhere            
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
ACCEPT     all  --  10.1.0.0/16          anywhere             /* generated for MicroK8s pods */
ACCEPT     all  --  anywhere             10.1.0.0/16          /* generated for MicroK8s pods */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-FIREWALL  all  --  anywhere             anywhere            
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             anywhere             /* container-registry/registry:registry has no endpoints */ ADDRTYPE match dst-type LOCAL tcp dpt:32000 reject-with icmp-port-unreachable

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP       all  -- !localhost/8          localhost/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             10.152.183.193       /* container-registry/registry:registry has no endpoints */ tcp dpt:5000 reject-with icmp-port-unreachable

I'm able to ping the nodes ip address on the network it is on.

[ktsakalozos]

Here is what might be happening.

When you microk8s enable dns you let kubernetes (the kubelet service) know of the existence of the dns service. When pods are created kubelet will inject the coredns endpoint in the containers so that coredns is used when resolving dns requests.

In the attached logs I see the coredns pod created 13 hours ago and all the pods before that. So, is it possible you enabled dns after you created the all pods? To test this theory you could delete one pod and let kubernetes reschedule it. The new pod will be created with resolv.conf pointing to the coredns service.

[mogwai]

I disabled and enabled the dns, in an attempt to resolve the issues I was having. The dig requests from the pods are making it to the coredns service. The problem is that no request from the k8s cluster is able to reach any IP outside of the node. The core dns included it seems.

[mogwai]

Reinstalling microk8s seems to have resolved the issues. I'll close for now but I didn't discover why there was an issues with connectivity within the pods

[UnasZole]

I have exactly the same issue.
All my pods are deployed and communicate with each other just fine (using internal DNS resolution, working just fine as well), but no external IP can be reached.
Ubuntu 20.04, same iptables config as shown above (except I have some additional "DOCKER*" chains defined, as I installed docker.io on the same machine)

From inside a container running in MicroK8s, if I ping :

• 127.0.0.1 : no problem
• 10.1.57.1 (my cni0 interface address) : no problem
• 192.168.1.37 (my host IP on local network, on wlo1 interface) : no problem
• 192.168.1.1 (or any other machine on my local network) : 100% packet loss
• 8.8.8.8 (or any other public IP) : 100% packet loss

All these failing addresses are of course reachable if I run ping from the host.
And they are also reachable if I run ping from an identical container running inside docker.io.
So there definitely is something wrong specific to microk8s here.

@mogwai How did you install/reinstall microk8s ? Just snap remove / snap install, or did you do anything different ?

[mogwai]

I snap removed, rebooted, snap installed.

[UnasZole]

Thanks a lot ! I just did the same and it worked.
I still have no idea what was causing the issue though.

Looking at https://github.com/ubuntu/microk8s/issues/75#issuecomment-407357469 , I was wondering if it could be an issue related to my frequent changing of IP on the host machine (I regularly switch from Ethernet to Wifi), but apparently not.

• Before the reinstall, I had tried disabling and re-enabling the snap and it didn't fix the issue,
• After the reinstall, I've tried switching IP again, and everything still seems to be working fine.

And the installed version is still the same (v1.19.0).
That's weird.

[mutech]

I have the same problem, however, the Windows 3.11 solution (uninstall/reboot/install) did not help in my case.

Since I had a fresh Ubuntu install, I went a bit further and also reinstalled Ubuntu, (server & desktop versions, both 20.04) and then tried installing microk8s, uninstall, reboot and install. At no point could any pod get internet connections.

I think reopening this issue would be justified.

[ktsakalozos]

@mutech please open a new issue attaching the microk8s inspect tarball.

[abonny]

Hey everyone I know this is closed but wanted to alert you. This is not _microk8s_ specific! I have a kubeadm initialized cluster and am seeing the exact same behavior. If anyone wants me to run commands or show logs etc... I'll try to pay attention to this thread.