Microk8s: DNS is crashlooping

Created on 13 Jul 2018  Â·  7Comments  Â·  Source: ubuntu/microk8s

$ microk8s.kubectl get all --all-namespaces 
NAMESPACE     NAME                                                  READY     STATUS             RESTARTS   AGE
kube-system   pod/heapster-v1.5.2-84f5c8795f-m466m                  4/4       Running            0          23m
kube-system   pod/kube-dns-864b8bdc77-6mst4                         2/3       CrashLoopBackOff   15         23m
kube-system   pod/kubernetes-dashboard-6948bdb78-262gm              0/1       CrashLoopBackOff   8          23m
kube-system   pod/monitoring-influxdb-grafana-v4-7ffdc569b8-dbmvg   2/2       Running            0          23m

NAMESPACE     NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
default       service/kubernetes             ClusterIP   10.152.183.1     <none>        443/TCP             23m
kube-system   service/heapster               ClusterIP   10.152.183.109   <none>        80/TCP              23m
kube-system   service/kube-dns               ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP       23m
kube-system   service/kubernetes-dashboard   ClusterIP   10.152.183.178   <none>        443/TCP             23m
kube-system   service/monitoring-grafana     ClusterIP   10.152.183.68    <none>        80/TCP              23m
kube-system   service/monitoring-influxdb    ClusterIP   10.152.183.252   <none>        8083/TCP,8086/TCP   23m

NAMESPACE     NAME                                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/heapster-v1.5.2                  1         1         1            1           23m
kube-system   deployment.apps/kube-dns                         1         1         1            0           23m
kube-system   deployment.apps/kubernetes-dashboard             1         1         1            0           23m
kube-system   deployment.apps/monitoring-influxdb-grafana-v4   1         1         1            1           23m

NAMESPACE     NAME                                                        DESIRED   CURRENT   READY     AGE
kube-system   replicaset.apps/heapster-v1.5.2-84f5c8795f                  1         1         1         23m
kube-system   replicaset.apps/kube-dns-864b8bdc77                         1         1         0         23m
kube-system   replicaset.apps/kubernetes-dashboard-6948bdb78              1         1         0         23m
kube-system   replicaset.apps/monitoring-influxdb-grafana-v4-7ffdc569b8   1         1         1         23m

Most helpful comment

@tvansteenburgh OMG! I was hitting my head to wall for a week for this, works like a charm! Thanks a ton!

All 7 comments

Anything interesting in the logs for kube-dns?

@dustinkirkland can you also make sure there is not firewall blocking dns as reported in this issue: https://github.com/juju-solutions/microk8s/issues/66

Indeed, I do have ufw enabled. Disabling it entirely is not possible.
I'll need to punch the right holes.

@DustinKirkland

On Fri, Jul 13, 2018 at 3:10 AM, Konstantinos Tsakalozos <
[email protected]> wrote:

@dustinkirkland https://github.com/dustinkirkland can you also make
sure there is not firewall blocking dns as reported in this issue: #66
https://github.com/juju-solutions/microk8s/issues/66

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/juju-solutions/microk8s/issues/67#issuecomment-404760239,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAzMhiXr4U1gu0U94foNLkvuQZKXncDHks5uGFX-gaJpZM4VOIZH
.

Inspecting the ufw log showed that all the denials were happening on the cbr0 interface.

ubuntu@ip:~$ ifconfig cbr0
cbr0      Link encap:Ethernet  HWaddr 0a:58:0a:01:01:01  
          inet addr:10.1.1.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::e0d0:96ff:fee2:633e/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:5577 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4904 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:989161 (989.1 KB)  TX bytes:2862109 (2.8 MB)

The 10.1.1.0/32 subnet corresponds to the pod IP addresses:

ubuntu@ip:~$ microk8s.kubectl get po -n kube-system -o wide
NAME                                              READY     STATUS    RESTARTS   AGE       IP         NODE
heapster-v1.5.2-577898ddbf-8mz8j                  4/4       Running   0          9m        10.1.1.7   ip-172-31-19-85
kube-dns-864b8bdc77-4n5s9                         3/3       Running   6          15m       10.1.1.2   ip-172-31-19-85
kubernetes-dashboard-6948bdb78-62n4r              1/1       Running   5          15m       10.1.1.4   ip-172-31-19-85
monitoring-influxdb-grafana-v4-7ffdc569b8-2t2b4   2/2       Running   0          15m       10.1.1.3   ip-172-31-19-85

So the fix was:

sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0

This worked like a champ for me! Thanks so much!

@DustinKirkland

On Fri, Jul 13, 2018 at 1:53 PM, Tim Van Steenburgh <
[email protected]> wrote:

Inspecting the ufw log showed that all the denials were happening on the
cbr0 interface.

ubuntu@ip:~$ ifconfig cbr0
cbr0 Link encap:Ethernet HWaddr 0a:58:0a:01:01:01
inet addr:10.1.1.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::e0d0:96ff:fee2:633e/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:5577 errors:0 dropped:0 overruns:0 frame:0
TX packets:4904 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:989161 (989.1 KB) TX bytes:2862109 (2.8 MB)

The 10.1.1.0/32 subnet corresponds to the pod IP addresses:

ubuntu@ip:~$ microk8s.kubectl get po -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
heapster-v1.5.2-577898ddbf-8mz8j 4/4 Running 0 9m 10.1.1.7 ip-172-31-19-85
kube-dns-864b8bdc77-4n5s9 3/3 Running 6 15m 10.1.1.2 ip-172-31-19-85
kubernetes-dashboard-6948bdb78-62n4r 1/1 Running 5 15m 10.1.1.4 ip-172-31-19-85
monitoring-influxdb-grafana-v4-7ffdc569b8-2t2b4 2/2 Running 0 15m 10.1.1.3 ip-172-31-19-85

So the fix was:

sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/juju-solutions/microk8s/issues/67#issuecomment-404921973,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAzMhoE5PxRXqxssvnQVOK0Pr0Ay3MQxks5uGOyZgaJpZM4VOIZH
.

@tvansteenburgh OMG! I was hitting my head to wall for a week for this, works like a charm! Thanks a ton!

sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0

Don't blindly copy this, ufw does now complain if there is no cbr0.
Use:

sudo brctl show

Then find out which bridge it is actually (mine was cni0).

Was this page helpful?
0 / 5 - 0 ratings