K3d: [BUG] K3d not able resolve dns

[rjshrjndrn]

What did you do?
k3d create

Logs

docker logs c -f

E1112 01:13:16.354329 7 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "helm-install-traefik-pp2cl_kube-system(af91145b-441d-41ba-9ef4-7eab9950754a)" failed: rpc error: code = Unknown desc = failed to get sandbox image "k8s.gcr.io/pause:3.1": failed to pull image "k8s.gcr.io/pause:3.1": failed to pull and unpack image "k8s.gcr.io/pause:3.1": failed to resolve reference "k8s.gcr.io/pause:3.1": failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.1: dial tcp: lookup k8s.gcr.io on 127.0.0.11:53: read udp 127.0.0.1:37126->127.0.0.11:53: i/o timeout
E1112 01:13:16.354347 7 kuberuntime_manager.go:710] createPodSandbox for pod "helm-install-traefik-pp2cl_kube-system(af91145b-441d-41ba-9ef4-7eab9950754a)" failed: rpc error: code = Unknown desc = failed to get sandbox image "k8s.gcr.io/pause:3.1": failed to pull image "k8s.gcr.io/pause:3.1": failed to pull and unpack image "k8s.gcr.io/pause:3.1": failed to resolve reference "k8s.gcr.io/pause:3.1": failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.1: dial tcp: lookup k8s.gcr.io on 127.0.0.11:53: read udp 127.0.0.1:37126->127.0.0.11:53: i/o timeout
E1112 01:13:16.354411 7 pod_workers.go:191] Error syncing pod af91145b-441d-41ba-9ef4-7eab9950754a ("helm-install-traefik-pp2cl_kube-system(af91145b-441d-41ba-9ef4-7eab9950754a)"), skipping: failed to "CreatePodSandbox" for "helm-install-traefik-pp2cl_kube-system(af91145b-441d-41ba-9ef4-7eab9950754a)" with CreatePodSandboxError: "CreatePodSandbox for pod \"helm-install-traefik-pp2cl_kube-system(af91145b-441d-41ba-9ef4-7eab9950754a)\" failed: rpc error: code = Unknown desc = failed to get sandbox image \"k8s.gcr.io/pause:3.1\": failed to pull image \"k8s.gcr.io/pause:3.1\": failed to pull and unpack image \"k8s.gcr.io/pause:3.1\": failed to resolve reference \"k8s.gcr.io/pause:3.1\": failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.1: dial tcp: lookup k8s.gcr.io on 127.0.0.11:53: read udp 127.0.0.1:37126->127.0.0.11:53: i/o timeout"

What did you expect to happen?

get a k3d single node cluster

Which OS & Architecture?

↑6% ❯ uname -a
Linux work 5.3.10-arch1-1 #1 SMP PREEMPT Sun, 10 Nov 2019 11:29:38 +0000 x86_64 GNU/Linux

Which version of k3d?

↑8% ❯ k3d --version
k3d version v1.3.4

Which version of docker?

↑8% ❯ docker --version
Docker version 19.03.4-ce, build 9013bf583a

[iwilltry42]

Hi @rjshrjndrn , thanks for opening this issue. Can you please exec into the k3d node container (docker exec -it shand see if you can pull the image manually viactr image pull k8s.gcr.io/pause:3.1. Then please check if you can connect to it / lookup the domain name and how name lookups work for you in there (e.g.cat /etc/resolv.confornslookup k8s.gcr.io`).

I cannot reproduce this issue with k3d version v1.3.4, Docker version 19.03.3, build a872fc2f86 on Linux trv3181 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux, so I guess it's something with your specific environment. You're on Arch Linux, right?

[rjshrjndrn]

I got the issue. I had one entry in /etc/systemd/network/en0.networking
Deleted that file, and everything is fine.
Thank you for your prompt response.

[mac9416]

I'm running into the same problem, but I don't have any files in /etc/systemd/network/.

When I exec into k3d-k3s-default-server, /etc/resolv.conf has the line nameserver 127.0.0.11. That line isn't in my host's resolv.conf, so I'm guessing it's written there by k3d.

If I change that line to nameserver 1.1.1.1, images are successfully pulled from docker.io.

But I'm not sure if I'm messing anything up by changing that line. I haven't found any documentation on DNS in k3d. Can anyone help me understand how DNS is configured and why my configuration isn't working?

[rjshrjndrn]

@mac9416 127.0.0.11 is docker's resolver.

[iwilltry42]

@mac9416

When I exec into k3d-k3s-default-server, /etc/resolv.conf has the line nameserver 127.0.0.11. That line isn't in my host's resolv.conf, so I'm guessing it's written there by k3d.

Nope, k3d does not manage any networking apart from creating a separate docker network for each cluster that you create (Or attaches to existing networks). But that also does not affect networking _inside_ the cluster (that's k3s' part).

As @rjshrjndrn said correctly, 127.0.0.11 is docker's internal resolver which will try to resolve to docker service VIPs and forwards request to the default DNS if it can't. See https://success.docker.com/article/ucp-service-discovery-swarm#servicediscoverywithdns

If I change that line to nameserver 1.1.1.1, images are successfully pulled from docker.io.
This can lead to other failures, since now the containers (k3d nodes) cannot resolve other docker service names. E.g. nodes may loose connection to each other.

But I'm not sure if I'm messing anything up by changing that line. I haven't found any documentation on DNS in k3d. Can anyone help me understand how DNS is configured and why my configuration isn't working?

There are 3 levels of DNS here:

1.inside k3s (Kubernetes Level): CoreDNS (configurable deployment, where you can also specify upstream nameservers)

1. inside k3d nodes (Container Level): 127.0.0.11 (Docker Resolver) tries to resolve docker service names (in Docker Engine DNS) and forwards requests if it can't
2. your host (whatever you specify in your resolv.conf): this may be different per host system

-> I don't know if there's another layer on Docker for Desktop (the VM)

But nothing of this is influenced by k3d itself.