K3d: [BUG] DNS not resolving

[luisdavim]

What did you do?

• How was the cluster created?

 k3d create -n mycluster

• What did you do afterwards?

Start a pod and try a DNS query:

$ export KUBECONFIG="$(k3d get-kubeconfig --name='mycluster')"
$ kubectl run --restart=Never --rm -i --tty tmp --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # nslookup www.gmail.com
Server:         10.43.0.10
Address:        10.43.0.10:53

;; connection timed out; no servers could be reached

/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
/ # exit

Exec into the k3d container and do the same DNS query:

docker exec -it k3d-endpoint-server sh
/ # nslookup www.gmail.com
Server:         127.0.0.11
Address:        127.0.0.11:53

Non-authoritative answer:
www.gmail.com   canonical name = mail.google.com
mail.google.com canonical name = googlemail.l.google.com
Name:   googlemail.l.google.com
Address: 172.217.164.101

Non-authoritative answer:
www.gmail.com   canonical name = mail.google.com
mail.google.com canonical name = googlemail.l.google.com
Name:   googlemail.l.google.com
Address: 2607:f8b0:4005:80b::2005

/ # cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0
/ # exit

What did you expect to happen?
I would expect the pods in the k3d cluster to be able to resolve DNS names

Which OS & Architecture?
MacOS 10.15.3

Which version of k3d?

• output of k3d --version

k3d version v1.7.0

Which version of docker?

• output of docker version

docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[iwilltry42]

Hi there, thanks for opening this issue.
As mentioned in the related issue (https://github.com/rancher/k3d/issues/101#issuecomment-603668428), CoreDNS is doing the name resolution inside the cluster.
I couldn't reproduce this on Linux (with the same docker and k3d versions), so I guess that the difference in DNS settings is caused by Docker for Desktop.

[luisdavim]

To workaround this issue I'm patching the coredns configmap.

[iwilltry42]

@luisdavim what's the patch that you apply?

[consideRatio]

@luisdavim I'm also interested in what patch you applied

[irizzant]

I'm pretty sure that adapting the forward . /etc/resolv.conf part is enough.

This let k8s use the machine DNS when the name cannot be resolved internally.

I think this should be the default behaviour.

[irizzant]

After more investigation I found that this could be related to the way k3d creates the docker network.

Indeed, k3d creates a custom docker network for each cluster and when this happens resolving is done through the docker daemon. The requests are actually forwarded to the DNS servers configured in your host's resolv.conf. But through a single DNS server (the embedded one of docker).

This means that if your daemon.json is, like mine, not configured to provide extra DNS servers it defaults to 8.8.8.8 which does not resolve any company address for example.

It would be useful to have a custom options to provide to k3d when it starts the cluster and specify the DNS servers there, as proposed here https://github.com/rancher/k3d/issues/165

[iwilltry42]

Thanks for your additional input @irizzant , how would you add additional DNS servers here on k3d's side?
The network opts of docker don't seem to have such an option (after having a quick glance over what's available there) :thinking:

[irizzant]

Personally I fixed this by injecting a custom ConfigMap for CoreDns, by changing

forward . /etc/resolv.conf

to:

forward . /etc/resolv.conf xxx.xxx.xxx.xxx

replacing the x with the IP of your DNS servers.

I had a quick look to the docker options and I confirm that I don't see an option to configure the custom DNS servers on the docker network.

Maybe a feasible option would be to add a custom flag to k3d command which adds the custom DNS servers to the CoreDns ConfigMap directly.

[iwilltry42]

Currently, k3d doesn't interact with any Kubernetes resources inside the cluster (i.e. in k3s) and I tend to avoid this because of the huge dependencies on Kubernetes libraries it could draw in. Upon cluster creation this could work however by modifying the Chart that's being auto-deployed by k3s. Not sure if this could go into k3s itself instead :thinking:

[irizzant]

Maybe interacting with k8s itself it's not needed.
k3s deploys whatever is under /var/lib/rancher/k3s/server/manifests, so you could add a valid CoreDns configuration just customizing the DNS part according to the command line flag.

[dminca]

This only happens to me if I deploy something in the default namespace, in other namespaces it worked just fine, idk why

LE: just noticed it doesn't matter which namespace you deploy stuff to, it's about the network you're in. So if I'm in the office (company LAN) I get this issue, but when I'm trying it from home, it just simply works. And I cannot say what network restrictions they applied in the company 😄

Also, @Athosone 's solution works for me now

[yamlCase]

I seem to be running into this issues about every couple of weeks. This is the only workaround that seems to "just work" so I can get back to the job I'm paid to do:

sudo iptables -t nat -A PREROUTING -p udp -d 8.8.8.8  --dport 53 -j DNAT --to <your DNS server IP>

[eigood]

This seems to be broken, because the coredns pod does not have an /etc/resolv.conf in it, while the ConfigMap is configured to forward to that. All the docs have pointed told me that coredns will use the $HOST resolv.conf, but when I used k3d, which uses k3s, the coredns "pod" doesn't run as a container, or as a process on the $HOST. It runs as a process of containerd, and therefore it doesn't get any of the correct settings.

[Athosone]

For those who have the problem a simple fix is to mount your /etc/resolve.conf onto the cluster:

k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf

[yamlCase]

What does that --volume command do exactly? I've used it to take advantage of other things registries.yaml, etc) but haven't taken the time to dig into what all it gets mapped to

[Athosone]

From what I understand there is nothing special. It just mounts the volume to the container running the k3s server.
Thus I guess you could mount anything and even maybe do some airgapped setup

[iwilltry42]

Hey folks, sorry for the radio silence, just getting back to k3d now...
Let me reply to some of the messages here.

---

@irizzant

Maybe interacting with k8s itself it's not needed.
k3s deploys whatever is under /var/lib/rancher/k3s/server/manifests, so you could add a valid CoreDns configuration just customizing the DNS part according to the command line flag.

That's a good starting point. Unfortunately, this would require us to write the file to disk and bind mount it into the container, as exec'ing into it afterwards to update the ConfigMap manifest, wouldn't update the actual thing inside the cluster (IIRC, there is no loop to do so). It's definitely doable, but we'd need to keep state somewhere and react to changes k3s does in the auto-deploy manifests.

---

@dminca

This only happens to me if I deploy something in the default namespace, in other namespaces it worked just fine, idk why

This is for real the weirdest thing on this thread that you're experiencing :thinking: No clue, what's going on there..

---

@yamlCase

I seem to be running into this issues about every couple of weeks. This is the only workaround that seems to "just work" so I can get back to the job I'm paid to do:

sudo iptables -t nat -A PREROUTING -p udp -d 8.8.8.8  --dport 53 -j DNAT --to <your DNS server IP>

Are you executing this on your local host (I assume so because of the sudo) to just route all the Google-DNS traffic (default Docker DNS) to your own DNS server?

---

@eigood & @Athosone

This seems to be broken, because the coredns pod does _not_ have an /etc/resolv.conf in it, while the ConfigMap is configured to forward to that. All the docs have pointed told me that coredns will use the $HOST resolv.conf, but when I used k3d, which uses k3s, the coredns "pod" doesn't run as a container, or as a process on the $HOST. It runs as a process of containerd, and therefore it doesn't get any of the correct settings.

What do you mean by "it doesn't run as a container"? It surely is running in a container :thinking:

For those who have the problem a simple fix is to mount your /etc/resolve.conf onto the cluster:

k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf

Also, those two statements seem to conflict, right?

---

@yamlCase

What does that --volume command do exactly? I've used it to take advantage of other things registries.yaml, etc) but haven't taken the time to dig into what all it gets mapped to

It's doing basically the same as docker's --volume flag: bind-mounting a file or a directory into one or more containers, overlaying anything that might already be there (you can specify the access mode).

---

Is anyone experiencing this on a Linux machine or in WSL2, i.e. in Docker versions which do not run inside a VM?
I could imagine, that we can modify the CoreDNS ConfigMap in the running cluster like we might be doing it to inject the host.k3d.internal entry in #360 .
However, just mounting in your resolv.conf might be the easiest solution :thinking:

[Athosone]

Well maybe it conflicts but for me it solves the problem.
For information I am running in wsl2.
I dont know why it does not managed to use the right resolv.conf if I dint mount it.

[iwilltry42]

@Athosone , I meant, that CoreDNS does indeed pick up the resolv.conf, as your solution works :+1:
I guess the problem is more that the "default" resolv.conf in the k3s container (node) does not work for it (and that one you're effectively replacing with the volume mount) :+1:

[irizzant]

Just as a clarification, if you modify the docker daemon configuration (daemon.json) to add the company DNS and then you launch the k3d cluster you can "docker exec" into the containers where k8s running and you'll see that nslookup finds DNS entries served by company DNS.
Consequently the docker container in which k8s is running picks up the right resolv.conf configuration.

The problem is that the default resolv.conf in k8s just reports:

forward . /etc/resolv.conf

which forces CoreDNS to use its own resolv.conf bypassing the docker one.

[eigood]

When I run k3d cluster create, I can certainly volume mount $HOST files into the docker containers(agent+server). However, containerd is then used to start the CoreDNS pod, running inside k8s. It is this internal container that needs to have an /etc/resolv.conf. Nothing I do to the k3d command will allow me to adjust the internal containerd/pod that is created.

I did a bit of research, trying to figure out that this was the case, by figuring out where containerd stores it's filesystems.

[eshepelyuk]

I am running k3d in a Docker that runs in VM in Virtual Box (actually I'm using Docker Toolbox for Windows product that does all the setup).

And I'm experiencing the same problem

• inside k3d docker container rancher/k3s:v1.18.8-k3s1 resolutions of external and custom (private VPN) domains is happening
• inside pods - fails.

I can see that my VM in Virtual Box and all containers running inside it are using proper DNS configurations, but pods - not.

[avodaqstephan]

Mounting the resolv.conf works but this can't be the best solution. If you missed that oppertunity at the beginning you need to re-create a k3d cluster just to mount that volume.

forward . /etc/resolv.conf xxx.xxx.xxx.xxx

This one did not work for me.

Edit: Forward is working but I need to place my DNS server in front.

forward . xxx.xxx.xxx.xxx /etc/resolv.conf

[szapps]

For those who have the problem a simple fix is to mount your /etc/resolve.conf onto the cluster:

k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf

VERY HELPFULL HINT

[iwilltry42]

Can someone give those binaries a try and use the experimental --enable-dns-magic flag to try to solve this problem similar to how kind does it?
https://drive.google.com/drive/folders/1Hu4Z0fAYr4ktw7r07naBUxOJY9EYr0Bq?usp=sharing
They are built from feature/dns

[eshepelyuk]

@iwilltry42

• Windows 10 20H2
• Docker Toolbox ( ie VirtualBox VM with Linux + Docker)
• Msys2 based bash as a shell

$ ./k3d-windows-amd64.exe --verbose cluster create k3s --enable-dns-magic --wait --no-lb --api-port 192.168.99.103:6550
DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Network with name 'k3d-k3s' already exists with ID '856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a'
INFO[0000] Created volume 'k3d-k3s-images'
INFO[0001] Creating node 'k3d-k3s-server-0'
DEBU[0001] Created container k3d-k3s-server-0 (ID: 4711af3bcb9c3efb41d4e5bee47ce846e96afda76d9c69fc703d3ea72c656612)
DEBU[0002] Created node 'k3d-k3s-server-0'
DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0066] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0066] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
DEBU[0066] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
DEBU[0067] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
ERRO[0067] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0067] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0067] Enabling DNS Magic...
DEBU[0067] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
DEBU[0069] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
ERRO[0069] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0069] Failed to enable DNS Magic: %!w(*errors.errorString=&{Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response})
INFO[0069] Cluster 'k3s' created successfully!
DEBU[0069] Updating default kubeconfig with a new context for cluster k3s
DEBU[0069] Setting new current-context 'k3d-k3s'
DEBU[0070] Wrote kubeconfig to 'C:\Users\eshep/.kube/config'
INFO[0070] You can now use it like this:
kubectl cluster-info

Apparently DNS resolution not working from pods since magic not applied.
And DNS resolution still works from the k3s docker container itself.

[iwilltry42]

@eshepelyuk that seems to be quite a special case. Can you please try again with --trace enabled (You can trim the log output after the DNS magic.
I have no experience with Docker Toolbox for Windows. Do you know, how it refers to the Host system?
We're trying docker-machine ip and the host.docker.internal on Windows.

[eshepelyuk]

Hello @iwilltry42

1. Adding requested logs.

Please note that I ran docker-machine ip before k3d and it returned proper address of VirtualBox VM with Docker inside,
the same address I've passed via --api-port.

1. Another question, is there a way to pass that host.docker.internal to k3d command ?

Here are the logs, click to expand

$ docker-machine.exe ip
192.168.99.103

$ ./k3d-windows-amd64.exe --trace --verbose cluster create k3s --enable-dns-magic --wait --no-lb --api-port $(docker-machine ip):6550
DEBU[0000] Selected runtime is 'docker.Docker'
TRAC[0000] PortFilterMap: map[]
TRAC[0000] LabelFilterMap: map[]
TRAC[0000] EnvFilterMap: map[]
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Network with name 'k3d-k3s' already exists with ID '856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a'
INFO[0000] Created volume 'k3d-k3s-images'
INFO[0001] Creating node 'k3d-k3s-server-0'
TRAC[0001] Creating node from spec
&{Name:k3d-k3s-server-0 Role:server Image:docker.io/rancher/k3s:v1.19.3-k3s3 Volumes:[k3d-k3s-images:/k3d/images] Env:[K3S_TOKEN=jWiJFvqWengrHwISsBKh] Cmd:[] Args:[] Ports:[192.168.99.103:6550:6443/tcp] Restart:false Labels:map[k3d.cluster:k3s k3d.cluster.imageVolume:k3d-k3s-images k3d.cluster.network:856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a k3d.cluster.network.external:true k3d.cluster.token:jWiJFvqWengrHwISsBKh k3d.cluster.url:https://k3d-k3s-server-0:6443] Network:856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:192.168.99.103 HostIP:192.168.99.103 Port:6550}} AgentOpts:{} GPURequest: State:{Running:false Status:}}

TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443/tcp:{}] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=jWiJFvqWengrHwISsBKh K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 192.168.99.103] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.19.3-k3s3 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s k3d.cluster.imageVolume:k3d-k3s-images k3d.cluster.network:856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a k3d.cluster.network.external:true k3d.cluster.token:jWiJFvqWengrHwISsBKh k3d.cluster.url:https://k3d-k3s-server-0:6443 k3d.role:server k3d.server.api.host:192.168.99.103 k3d.server.api.hostIP:192.168.99.103 k3d.server.api.port:6550] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[6443/tcp:[{HostIP:192.168.99.103 HostPort:6550}]] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc0002cc06a} NetworkingConfig:{EndpointsConfig:map[856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a:0xc000194840]}}
DEBU[0001] Created container k3d-k3s-server-0 (ID: c8d4f2fd7253433d4fb94be5015743882750e31ebdf6a4f1bad599da67efe720)
DEBU[0002] Created node 'k3d-k3s-server-0'
DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0059] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0059] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0059] Runtime GOOS: windows
DEBU[0059] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0059] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0060] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0060] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0060] Scanning Log Line 'Server:           127.0.0.11'
TRAC[0060] Scanning Log Line 'Address:  127.0.0.11:53'
TRAC[0060] Scanning Log Line ''
TRAC[0060] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0060] Scanning Log Line ''
TRAC[0060] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0060] Scanning Log Line ''
ERRO[0060] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0060] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0060] Enabling DNS Magic...
TRAC[0060] Runtime GOOS: windows
DEBU[0060] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0060] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0061] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0061] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0061] Scanning Log Line 'Server:           127.0.0.11'
TRAC[0061] Scanning Log Line 'Address:  127.0.0.11:53'
TRAC[0061] Scanning Log Line ''
TRAC[0061] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0061] Scanning Log Line ''
TRAC[0061] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0061] Scanning Log Line ''
ERRO[0061] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0061] Failed to enable DNS Magic: %!w(*errors.errorString=&{Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response})
INFO[0061] Cluster 'k3s' created successfully!

..........

[iwilltry42]

@eshepelyuk , thank you!
I just realized, that for the DNS magic (and all other related parts), we're not trying docker-machine ip. I added this now and uploaded a new build to the google drive folder (or you build it yourself).
Thanks for testing!

[eshepelyuk]

Hello @iwilltry42

Just downloaded a new build from the link provided above.

The error and behaviour are the same.

$ ./k3d-windows-amd64.exe --version
k3d version v3.3.0-2-g296bd5c
k3s version v1.19.3-k3s3 (default)

$ ./k3d-windows-amd64.exe --trace --verbose cluster create k3s --enable-dns-magic --wait --no-lb --api-port $(docker-machine ip):6550

.....

DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0062] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0062] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0062] Runtime GOOS: windows
DEBU[0062] Docker Machine found: default
DEBU[0062] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0063] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0064] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0064] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0064] Scanning Log Line 'Server:           127.0.0.11'
TRAC[0064] Scanning Log Line 'Address:  127.0.0.11:53'
TRAC[0064] Scanning Log Line ''
TRAC[0064] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0064] Scanning Log Line ''
TRAC[0064] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0064] Scanning Log Line ''
ERRO[0064] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0064] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0064] Enabling DNS Magic...
TRAC[0064] Runtime GOOS: windows
DEBU[0064] Docker Machine found: default
DEBU[0064] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0064] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0065] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0065] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0065] Scanning Log Line 'Server:           127.0.0.11'
TRAC[0065] Scanning Log Line 'Address:  127.0.0.11:53'
TRAC[0065] Scanning Log Line ''
TRAC[0065] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0065] Scanning Log Line ''
TRAC[0065] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0065] Scanning Log Line ''
ERRO[0065] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0065] Failed to enable DNS Magic: Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0065] Cluster 'k3s' created successfully!

....

[iwilltry42]

@eshepelyuk thanks again for testing! Apparently my workday before this was too long and stupid me forgot a return there :roll_eyes:
I uploaded a new build of feature/dns now which should actually use the docker-machine IP after looking it up (which worked according to the logs) :+1:

[eshepelyuk]

Hello @iwilltry42

Unfortunately, there's no progress. No errors in logs, though.

DEBU[0001] Created container k3d-k3s-server-0 (ID: f34492b4b1bf149d86f0957a5b20aa82d56ef3213cfb4f65fc08bb32bf1104d8)
DEBU[0002] Created node 'k3d-k3s-server-0'
DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0080] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0080] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0080] Runtime GOOS: windows
DEBU[0080] Docker Machine found: default
DEBU[0081] Adding extra host entry '192.168.99.103 host.k3d.internal'...
DEBU[0081] Executing command '[sh -c echo '192.168.99.103 host.k3d.internal' >> /etc/hosts]' in node 'k3d-k3s-server-0'
TRAC[0081] Exec process '[sh -c echo '192.168.99.103 host.k3d.internal' >> /etc/hosts]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0082] Exec process in node 'k3d-k3s-server-0' exited with '0'
DEBU[0082] Executing command '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' in node 'k3d-k3s-server-0'
TRAC[0082] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0083] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0084] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0085] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0086] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0087] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0088] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0089] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.99.103 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0090] Exec process in node 'k3d-k3s-server-0' exited with '0'
INFO[0090] Successfully added host record to /etc/hosts in 1/1 nodes and to the CoreDNS ConfigMap
INFO[0090] Enabling DNS Magic...
TRAC[0090] Runtime GOOS: windows
DEBU[0090] Docker Machine found: default
DEBU[0091] Executing command '[sh -c iptables-save | sed -e 's/--to-source :53/--to-source 192.168.99.103:53/g' -e 's/-A OUTPUT \(.*\) -j DOCKER_OUTPUT/\0\n-A PREROUTING \1 -j DOCKER_OUTPUT/g' -e 's/-d 127.0.0.11/-d 192.168.99.103/g' | iptables-restore && cp /etc/resolv.conf /etc/resolv.conf.original && sed -e 's/127.0.0.11/192.168.99.103/g' /etc/resolv.conf.original >/etc/resolv.conf]' in node 'k3d-k3s-server-0'
TRAC[0091] Exec process '[sh -c iptables-save | sed -e 's/--to-source :53/--to-source 192.168.99.103:53/g' -e 's/-A OUTPUT \(.*\) -j DOCKER_OUTPUT/\0\n-A PREROUTING \1 -j DOCKER_OUTPUT/g' -e 's/-d 127.0.0.11/-d 192.168.99.103/g' | iptables-restore && cp /etc/resolv.conf /etc/resolv.conf.original && sed -e 's/127.0.0.11/192.168.99.103/g' /etc/resolv.conf.original >/etc/resolv.conf]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0092] Exec process in node 'k3d-k3s-server-0' exited with '0'
INFO[0092] Successfully applied DNS Magic to 1/1 nodes
INFO[0092] Cluster 'k3s' created successfully!

[iwilltry42]

Yeah, so it seems like it at least achieved what should've been achieved.
The change is similar to what kind does to "fix" DNS resolution on some systems: Set the host IP as a nameserver in /etc/resolv.conf and configure the iptables rules accordingly.
Seems like this doesn't give us anything though :thinking:
I'll research a bit more on this. Any input is highly appreciated :)

The problem behind this is, that in custom docker networks, docker uses the embedded DNS server (which also forwards DNS requests). This one is reachable from within the containers via special routes (which is why names can properly be resolved inside the k3s containers), but not from within the pods running inside those containers (which is why the forward . /etc/resolv.conf in CoreDNS doesn't give us much here).
When you run a container in docker's default bridge network however, the containers will receive a copy of the host's /etc/resolv.conf, making the name resolution fairly straight forward.
We could try to mirror this behavior now when creating the custom docker networks, however, I don't know, how this translates to Docker for Desktop and docker-machine/docker-toolbox :thinking:
Any ideas?

[therealmikz]

I am having the exact same issue @eshepelyuk has - inside k3s container all custom domains are seen without any issue but inside any pod they're not resolved. Linux here, newest k3d (installed today).
My host is using custom networkmanager connection with few domains set up to use VPN dns.
Forward trick worked but it's annoying to do this everytime when I create cluster.

[avodaqstephan]

Workaround: Edit the coredns configmap

forward . xxx.xxx.xxx.xxx /etc/resolv.conf

Problem: This is not persistent if you restart k3d. That is actually the real issue! Which component is responsible for re-writing / resetting changes in the configmap of coredns?

[iwilltry42]

@avodaqstephan

Workaround: Edit the coredns configmap

forward . xxx.xxx.xxx.xxx /etc/resolv.conf

Problem: This is not persistent if you restart k3d. That is actually the real issue! Which component is responsible for re-writing / resetting changes in the configmap of coredns?

That's k3s itself, which deploys some manifests automatically upon startup: https://rancher.com/docs/k3s/latest/en/advanced/#auto-deploying-manifests