K3d: [BUG] OSX - k3s:latest (1.19.3-k3s1) creates cluster but Host IP fails.

[stephenpope]

What did you do

• How was the cluster created?

  • k3d cluster create

• What did you do afterwards?

  • Nothing

What did you expect to happen

Create without any errors.

Screenshots or terminal output

DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] API-Port Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for API: '61308'
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Created network 'k3d-k3s-default'
INFO[0000] Created volume 'k3d-k3s-default-images'
INFO[0001] Creating node 'k3d-k3s-default-server-0'
DEBU[0001] Created container k3d-k3s-default-server-0 (ID: a64a55917263cbbd72bbb04869e27c3f039f57a5f9b8028a88728cb96d0d2e1d)
DEBU[0001] Created node 'k3d-k3s-default-server-0'
DEBU[0001] Waiting for server node 'k3d-k3s-default-server-0' to get ready
DEBU[0007] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-default-server-0'
INFO[0007] Creating LoadBalancer 'k3d-k3s-default-serverlb'
DEBU[0007] Created container k3d-k3s-default-serverlb (ID: e637b4825160de9f616e4d9879ed25408cfd73c4d58b02a333e8e47aa22c3cbb)
DEBU[0008] Starting to wait for loadbalancer node 'k3d-k3s-default-serverlb'
DEBU[0008] Finished waiting for log message 'start worker processes' from node 'k3d-k3s-default-serverlb'
INFO[0008] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
DEBU[0008] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-default-server-0'
DEBU[0009] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
ERRO[0009] Logs from failed access process:
Server:     127.0.0.11
Address:    127.0.0.11:53

Non-authoritative answer:
Name:   host.docker.internal
Address: 192.168.65.2

** server can't find host.docker.internal: NXDOMAIN

WARN[0009] Failed to get HostIP: %+v Failed to read address for 'host.docker.internal' from nslookup response
INFO[0009] Cluster 'k3s-default' created successfully!
DEBU[0009] Updating default kubeconfig with a new context for cluster k3s-default
DEBU[0009] Setting new current-context 'k3d-k3s-default'
DEBU[0009] Wrote kubeconfig to '/Users/stp/.kube/config'
INFO[0009] You can now use it like this:
kubectl cluster-info

Which OS & Architecture

• MacOS (10.15.7)

Which version of k3d

k3d version v3.1.3
k3s version latest (default)

Which version of docker

Client: Docker Engine - Community
 Cloud integration  0.1.18
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 16:58:31 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Notes

From --trace output ..

<snip>
INFO[0008] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0008] Runtime GOOS: darwin
DEBU[0008] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-default-server-0'
TRAC[0008] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-default-server-0'.. sleeping for 1 second...
DEBU[0009] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0009] -> Exec Process Error was: Logs from failed access process:
Server:     127.0.0.11
Address:    127.0.0.11:53
<snip>

[iwilltry42]

Hi @stephenpope , thanks for opening this issue!
I'm looking into this now, while I don't really get it yet.
Can you please paste the full --trace output here?
However, the injection into CoreDNS won't work afterwards, because xargs is missing from k3s:v1.19-x images..

[stephenpope]

Full trace ..

❯ k3d cluster create testing --trace
DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] API-Port Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for API: '57988'
TRAC[0000] PortFilterMap: map[]
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Created network 'k3d-testing'
INFO[0000] Created volume 'k3d-testing-images'
INFO[0001] Creating node 'k3d-testing-server-0'
TRAC[0001] Creating node from spec
&{Name:k3d-testing-server-0 Role:server Image:docker.io/rancher/k3s:latest Volumes:[k3d-testing-images:/k3d/images] Env:[K3S_TOKEN=dJICWEAODFTFNYBgGbCP] Cmd:[] Args:[] Ports:[] Restart:false Labels:map[k3d.cluster:testing k3d.cluster.imageVolume:k3d-testing-images k3d.cluster.network:8bf09c578ba43f09e6a5b1b45ffc73378498eb31be5bc82d90cad160fc339fda k3d.cluster.network.external:false k3d.cluster.token:dJICWEAODFTFNYBgGbCP k3d.cluster.url:https://k3d-testing-server-0:6443] Network:8bf09c578ba43f09e6a5b1b45ffc73378498eb31be5bc82d90cad160fc339fda ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:57988}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-testing-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=dJICWEAODFTFNYBgGbCP K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:latest Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:testing k3d.cluster.imageVolume:k3d-testing-images k3d.cluster.network:8bf09c578ba43f09e6a5b1b45ffc73378498eb31be5bc82d90cad160fc339fda k3d.cluster.network.external:false k3d.cluster.token:dJICWEAODFTFNYBgGbCP k3d.cluster.url:https://k3d-testing-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:57988] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-testing-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00046fdaf} NetworkingConfig:{EndpointsConfig:map[8bf09c578ba43f09e6a5b1b45ffc73378498eb31be5bc82d90cad160fc339fda:0xc000236840]}}
DEBU[0001] Created container k3d-testing-server-0 (ID: 6e23dba59338ca24a7bcd7d6c045c4dcac1fa2d419c0dd02b172e5b47c6237fe)
DEBU[0001] Created node 'k3d-testing-server-0'
DEBU[0001] Waiting for server node 'k3d-testing-server-0' to get ready
DEBU[0007] Finished waiting for log message 'k3s is up and running' from node 'k3d-testing-server-0'
INFO[0007] Creating LoadBalancer 'k3d-testing-serverlb'
TRAC[0007] Creating node from spec
&{Name:k3d-testing-serverlb Role:loadbalancer Image:docker.io/rancher/k3d-proxy:v3.1.3 Volumes:[] Env:[SERVERS=k3d-testing-server-0 PORTS=6443 WORKER_PROCESSES=1] Cmd:[] Args:[] Ports:[0.0.0.0:57988:6443/tcp] Restart:false Labels:map[app:k3d k3d.cluster:testing] Network:8bf09c578ba43f09e6a5b1b45ffc73378498eb31be5bc82d90cad160fc339fda ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host: HostIP: Port:}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0007] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-testing-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443/tcp:{}] Tty:false OpenStdin:false StdinOnce:false Env:[SERVERS=k3d-testing-server-0 PORTS=6443 WORKER_PROCESSES=1 K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-proxy:v3.1.3 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:testing k3d.role:loadbalancer] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[6443/tcp:[{HostIP:0.0.0.0 HostPort:57988}]] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc001130c0f} NetworkingConfig:{EndpointsConfig:map[8bf09c578ba43f09e6a5b1b45ffc73378498eb31be5bc82d90cad160fc339fda:0xc001264a80]}}
DEBU[0007] Created container k3d-testing-serverlb (ID: 120ae44925946cdbf9476f52c558a872671a96958a766ecd91a909579710b639)
DEBU[0007] Starting to wait for loadbalancer node 'k3d-testing-serverlb'
DEBU[0008] Finished waiting for log message 'start worker processes' from node 'k3d-testing-serverlb'
INFO[0008] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0008] Runtime GOOS: darwin
DEBU[0008] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-testing-server-0'
TRAC[0008] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-testing-server-0'.. sleeping for 1 second...
DEBU[0009] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0009] -> Exec Process Error was: Logs from failed access process:
Server:     127.0.0.11
Address:    127.0.0.11:53

** server can't find host.docker.internal: NXDOMAIN

Non-authoritative answer:
Name:   host.docker.internal
Address: 192.168.65.2

ERRO[0009] Logs from failed access process:
Server:     127.0.0.11
Address:    127.0.0.11:53

** server can't find host.docker.internal: NXDOMAIN

Non-authoritative answer:
Name:   host.docker.internal
Address: 192.168.65.2

WARN[0009] Failed to get HostIP: %+v Failed to read address for 'host.docker.internal' from nslookup response
INFO[0009] Cluster 'testing' created successfully!
DEBU[0009] Updating default kubeconfig with a new context for cluster testing
TRAC[0009] Container Details: &{ID:6e23dba59338ca24a7bcd7d6c045c4dcac1fa2d419c0dd02b172e5b47c6237fe Names:[/k3d-testing-server-0] Image:docker.io/rancher/k3s:latest ImageID:sha256:8694fcae5cb9e343d2b33bac22158d53b561ebf9a9b97a1f74ff91b4f76aac2e Command:/bin/k3s server --tls-san 0.0.0.0 Created:1603104086 Ports:[] SizeRw:0 SizeRootFs:0 Labels:map[app:k3d k3d.cluster:testing k3d.cluster.imageVolume:k3d-testing-images k3d.cluster.network:8bf09c578ba43f09e6a5b1b45ffc73378498eb31be5bc82d90cad160fc339fda k3d.cluster.network.external:false k3d.cluster.token:dJICWEAODFTFNYBgGbCP k3d.cluster.url:https://k3d-testing-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:57988 org.label-schema.build-date:2020-10-16T00:20:33Z org.label-schema.schema-version:1.0 org.label-schema.vcs-ref:974ad30be7b7b15f695d88e46ffa96a4d43b2370 org.label-schema.vcs-url:https://github.com/rancher/k3s.git] State:running Status:Up 8 seconds HostConfig:{NetworkMode:default} NetworkSettings:0xc000824d00 Mounts:[{Type:volume Name:0d5ad80425e33ffa96fd1f9a732a501dbf785c5cd0f69f5c4f341b3845e8e878 Source: Destination:/var/lib/cni Driver:local Mode: RW:true Propagation:} {Type:volume Name:5c234f578bcc452edd6c665cd9f00907f9f09d1b971544ba162524190d2572cb Source: Destination:/var/lib/kubelet Driver:local Mode: RW:true Propagation:} {Type:volume Name:dbc64b56c9dedb2da1174f5f1c61a383ae99d5d431649aa574261254a6ccc0dd Source: Destination:/var/lib/rancher/k3s Driver:local Mode: RW:true Propagation:} {Type:volume Name:e593182f1193715cf7509f8bcba7e0a3aa7bd828f3f62ffc95c4cef4ad34d5c2 Source: Destination:/var/log Driver:local Mode: RW:true Propagation:} {Type:volume Name:k3d-testing-images Source:/var/lib/docker/volumes/k3d-testing-images/_data Destination:/k3d/images Driver:local Mode:z RW:true Propagation:}]}
TRAC[0009] Modified Kubeconfig: &{Kind: APIVersion: Preferences:{Colors:false Extensions:map[]} Clusters:map[k3d-testing:0xc000c91500] AuthInfos:map[admin@k3d-testing:0xc00118c000] Contexts:map[k3d-testing:0xc0000b2d20] CurrentContext:k3d-testing Extensions:map[]}
TRAC[0009] Merging new Kubeconfig:
&{Kind: APIVersion: Preferences:{Colors:false Extensions:map[]} Clusters:map[k3d-testing:0xc000c91500] AuthInfos:map[admin@k3d-testing:0xc00118c000] Contexts:map[k3d-testing:0xc0000b2d20] CurrentContext:k3d-testing Extensions:map[]}
>>> into existing Kubeconfig:
&{Kind: APIVersion: Preferences:{Colors:false Extensions:map[]} Clusters:map[aks-test-1:0xc0007a0480 coke-cluster:0xc0007a0540 k3d-cluedin:0xc0007a0780 minikube:0xc0007a0840] AuthInfos:map[admin@k3d-cluedin:0xc00118c0f0 clusterUser_aks-test-1_aks-test-1:0xc00118c1e0 clusterUser_rg-aks-customer-cocacola_coke-cluster:0xc00118c2d0 minikube:0xc00118c3c0] Contexts:map[aks-test-1:0xc0000b3590 coke-cluster:0xc0000b35e0 k3d-cluedin:0xc0000b3630 minikube:0xc0000b3680] CurrentContext:k3d-cluedin Extensions:map[]}
DEBU[0009] Setting new current-context 'k3d-testing'
TRAC[0009] Merged Kubeconfig:
<snip>
DEBU[0009] Wrote kubeconfig to '/Users/stp/.kube/config'
INFO[0009] You can now use it like this:
kubectl cluster-info

[iwilltry42]

This is fixed by 425b9b7 (and the k3s 1.19 issue by b69b874) :+1:
I'll create a new release now that will include both fixes :)

[stephenpope]

Awesome .. thank you @iwilltry42 for the super speedy resolution ! :)