Microk8s: No public DNS resolution inside pods

Created on 23 Jul 2018 · 14Comments · Source: ubuntu/microk8s

I cannot resolve public DNS inside running pods (with and without dns addon), althought internal k8s DNS works ok with the dns addon enabled. ufw is disabled. Running on DigitalOcean.

$ snap version
snap    2.33.1ubuntu2
snapd   2.33.1ubuntu2
series  16
ubuntu  16.04
kernel  4.4.0-130-generic

$ snap list
Name      Version    Rev   Tracking  Developer  Notes
core      16-2.33.1  4917  stable    canonical  core
microk8s  v1.11.0    104   beta      canonical  classic

any-pod$ curl google.com
curl: (6) Could not resolve host: google.com

Q&A

Source

adrianchifor

Most helpful comment

There are a couple of things that may be causing this behavior.

Your machine switched its IP. A workaround is to restart microk8s:

sudo snap disable microk8s
sudo snap enable microk8s

Your IP can change if for example you are on a laptop moving from place to place or you suspend/resume. The API server advertises and listens on your external IP. Services such ad DNS and the dashboard that need to contact the API server will stop working properly if its IP changes. If you do a microk8s.kubectl get ep kubernetes you should get the endpoint of the API server if this endpoint is not an IP in your system then you know you have this issue.

This was also discussed in https://github.com/ubuntu/microk8s/issues/72 as well. We know we need to address this in a more elegant way.

Traffic forward is blocked. Do this for a quick check:

sudo iptables -P FORWARD ACCEPT

And see if your pods can now access the internet.

What happens here is that your system is not aware it is functioning as a router so it drops any packets it is not aware of and that includes any traffic from the k8s cluster to the outside world. You would better be more precise on the traffic you want to allow. For example you may want to sudo iptables -A FORWARD -i wlan2 -j ACCEPT and even filter the k8s related traffic.

This is more difficult to address from within microk8s since network interfaces and routing may change without any notice.

Finally, make sure you have your firewall correctly setup so it allows traffic from and to cbr0 as described in the troubleshooting part of the README.

@adrianchifor @toxsick please let me know if you are affected by any of those two issues.

ktsakalozos on 24 Jul 2018

👍25 🎉4

All 14 comments

Thank you for reporting this @adrianchifor.

The dns we ship with microk8s uses 8.8.8.8 and 8.8.4.4 as upstream nameservers (https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/dns.yaml#L55) We are looking into why this approach in resolving external IPs does not work.

ktsakalozos on 23 Jul 2018

Hi,

I'm having the same issue. I am a bloody Kubernetes beginner, but as far as I can see it I can not access any external network from inside the pod. In my case i want to access a host in out internal network. Is that possible?

regards
Hannes

toxsick on 23 Jul 2018

👍3

@toxsick if the host can access it, the pod should be able to as well if you don't have any egress rules.

@ktsakalozos Thanks for looking into it, my gut feel is that there's something weird going on with the docker network config.

adrianchifor on 23 Jul 2018

There are a couple of things that may be causing this behavior.

Your machine switched its IP. A workaround is to restart microk8s:

sudo snap disable microk8s
sudo snap enable microk8s

This was also discussed in https://github.com/ubuntu/microk8s/issues/72 as well. We know we need to address this in a more elegant way.

Traffic forward is blocked. Do this for a quick check:

sudo iptables -P FORWARD ACCEPT

And see if your pods can now access the internet.

This is more difficult to address from within microk8s since network interfaces and routing may change without any notice.

Finally, make sure you have your firewall correctly setup so it allows traffic from and to cbr0 as described in the troubleshooting part of the README.

@adrianchifor @toxsick please let me know if you are affected by any of those two issues.

ktsakalozos on 24 Jul 2018

👍25 🎉4

Hi,

thanks for the detailed clarification! Here is what I found:

I already found the iptables problem. I thought since my firewall service is disabled this could not be the problem, but my pods immediately had internet access after I ran iptables -P FORWARD ACCEPT on the node.
As you pointed out the kube-dns uses the google dns servers, so I was not able to reach any servers in my local network. After I changed the upstreamNameservers in Config and storage ->
Config Maps -> kube-dns (namespace kube-system) to my local dns server everything was working fine!

Thank you for the hints!

regards
Hannes

toxsick on 24 Jul 2018

👍7

@adrianchifor Does running iptables -P FORWARD ACCEPT fix your problem?

tvansteenburgh on 24 Jul 2018

@tvansteenburgh will check when I get home, but I assume that should do as my machine hasn't changed IP so that rules out point 1 from @ktsakalozos's message

adrianchifor on 24 Jul 2018

@tvansteenburgh @ktsakalozos sudo iptables -P FORWARD ACCEPT did the trick! thanks a lot guys. Would be good to document this in the readme.

adrianchifor on 24 Jul 2018

👍4

Is there a way to make microk8s honor the /etc/hosts of the host machine?

gortiz on 28 Nov 2018

👍7

Is there a way to make microk8s honor the /etc/hosts of the host machine?

This is what I am looking for.
OR
Can I tell microk8s to use dns on host machine to resolve urls

imaimai86 on 30 Oct 2019

Can I tell microk8s to use dns on host machine

You should be able to set any forward DNS by editing the respective config map: microk8s.kubectl edit -n kube-system cm/coredns

ktsakalozos on 30 Oct 2019

👍1

Can I tell microk8s to use dns on host machine

You should be able to set any forward DNS by editing the respective config map: microk8s.kubectl edit -n kube-system cm/coredns

@ktsakalozos can you give an example how to make microk8s respect the /etc/hosts of the host machine?

RAbraham on 25 Jan 2020

I'm also facing the same issue. sudo iptables -P FORWARD ACCEPT didn't help and ufw is inactive.

This is my configmap:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        log . {
          class error
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n    log . {\n      class error\n    }\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . 8.8.8.8 8.8.4.4\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2020-02-05T11:45:03Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
  resourceVersion: "706"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 589c9bb9-31d8-4d2d-8e8e-a7283714c3a4

I can ping any external IP address, but as soon as I try ping google.com it does not work. Pinging internal host names like kubernetes.default works just fine

I also get these weird log errors, but I can't seem to find the cause of the problem:

[INFO] 10.16.238.16:37744 - 3 "AAAA IN google.com. udp 28 false 512" NOERROR - 0 2.000165416s
[ERROR] plugin/errors: 2 google.com. AAAA: read udp 10.16.183.38:41109->8.8.4.4:53: i/o timeout
[INFO] 10.16.238.16:53253 - 7 "A IN google.com. udp 28 false 512" NOERROR - 0 2.000320021s
[ERROR] plugin/errors: 2 google.com. A: read udp 10.16.183.38:57285->8.8.8.8:53: i/o timeout
[INFO] 10.16.238.16:46368 - 3 "AAAA IN google.com. udp 28 false 512" NOERROR - 0 2.000350667s
[ERROR] plugin/errors: 2 google.com. AAAA: read udp 10.16.183.38:53685->8.8.8.8:53: i/o timeout
[INFO] 10.16.238.16:49205 - 7 "A IN google.com. udp 28 false 512" NOERROR - 0 2.000165133s
[ERROR] plugin/errors: 2 google.com. A: read udp 10.16.183.38:55002->8.8.8.8:53: i/o timeout
[INFO] 10.16.129.74:43313 - 7801 "AAAA IN grafana.com. udp 29 false 512" NOERROR - 0 2.000595431s
[ERROR] plugin/errors: 2 grafana.com. AAAA: read udp 10.16.183.38:48023->8.8.8.8:53: i/o timeout
[INFO] 10.16.129.74:58679 - 43484 "A IN grafana.com. udp 29 false 512" NOERROR - 0 2.000555707s
[ERROR] plugin/errors: 2 grafana.com. A: read udp 10.16.183.38:38047->8.8.8.8:53: i/o timeout
[INFO] 10.16.129.74:46171 - 57548 "AAAA IN grafana.com. udp 29 false 512" NOERROR - 0 2.000614742s
[INFO] 10.16.129.74:34684 - 32154 "A IN grafana.com. udp 29 false 512" NOERROR - 0 2.000546798s
[ERROR] plugin/errors: 2 grafana.com. AAAA: read udp 10.16.183.38:47430->8.8.4.4:53: i/o timeout
[ERROR] plugin/errors: 2 grafana.com. A: read udp 10.16.183.38:40870->8.8.8.8:53: i/o timeout

LeKristapino on 15 Jul 2020

@ktsakalozos I'm facing almost the same issue as @LeKristapino ping -c 1 www.google.com works fine, but wget "http://httpbin.org/get" gives me a 403 forbidden.