K3d: [BUG] `k3d cluster create` does not work with Docker Machine

[zulhfreelancer]

What did you do

• How was the cluster created?

  • k3d cluster create

• What did you do afterwards?

  • kubectl cluster-info with error below:

  To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
  The connection to the server 0.0.0.0:54833 was refused - did you specify the right host or port?
  

  • kubectl cluster-info dump with error below:

  The connection to the server 0.0.0.0:54833 was refused - did you specify the right host or port?
  
  

What did you expect to happen

• I'm expecting a single-server cluster from the k3d cluster create command above but it didn't
• I tried k3d cluster create --no-hostip too but the same problem happened

Screenshots or terminal output

https://asciinema.org/a/368792

$ docker-machine status
Running

$ docker-machine ip
192.168.99.100

$ eval $(docker-machine env)

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ k3d cluster list
NAME   SERVERS   AGENTS   LOADBALANCER

$ k3d cluster create
INFO[0000] Network with name 'k3d-k3s-default' already exists with ID 'f0069ad1d20d1a5a67ea7e1c5d6c23cd7d980bd757ffdd663c2019a3801dc9e5'
INFO[0000] Created volume 'k3d-k3s-default-images'
INFO[0001] Creating node 'k3d-k3s-default-server-0'
INFO[0017] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0018] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
ERRO[0020] Exec process in node 'k3d-k3s-default-server-0' failed with exit code '1'
WARN[0020] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0020] Cluster 'k3s-default' created successfully!
INFO[0020] You can now use it like this:
kubectl cluster-info

$ kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 0.0.0.0:54833 was refused - did you specify the right host or port?

$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                             NAMES
687dd3d253c5        rancher/k3d-proxy:v3.1.5   "/bin/sh -c nginx-pr…"   13 seconds ago      Up 12 seconds       80/tcp, 0.0.0.0:54833->6443/tcp   k3d-k3s-default-serverlb
f55131a7599f        rancher/k3s:latest         "/bin/k3s server --t…"   30 seconds ago      Up 29 seconds                                         k3d-k3s-default-server-0

Which OS & Architecture

• Host: Mac OS 10.12.6
• Docker Engine/Daemon running on VirtualBox version 5.2.18

Which version of k3d

k3d version v3.1.5
k3s version latest (default)

Which version of docker

$ docker version
Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.15.3
 Git commit:        4484c46
 Built:             Thu Oct 29 12:59:29 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:35 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.130-boot2docker
 Operating System: Boot2Docker 19.03.12 (TCL 10.1)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 3.853GiB
 Name: default
 ID: ZU2Q:PTOJ:MFBT:VHVO:C5XC:77WC:OHHM:OAF3:7J4M:KNLN:JANC:SMMQ
 Docker Root Dir: /mnt/sda1/var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  provider=virtualbox
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[amimof]

I have the same problem. This issue appears on Mac OS. I can resolve the problem by replacing 0.0.0.0 with host.docker.internal in the kubeconfig.

[zulhfreelancer]

@amimof how did you download the kubeconfig file from the master node?

[amimof]

Try k3d kubeconfig merge <cluster_name> --switch-context

[iwilltry42]

Hi @zulhfreelancer , thanks for opening this issue!
It seems like the cluster creation works though.
It's just that the usual way of exposing the Kubernetes API does not work properly with docker machine.
@amimof already gave you the workaround for your problem, thanks for that!
You can also use --api-port "$(docker-machine ip):6443" (with whatever port you'd like) to get to this result upon cluster creation.

[iwilltry42]

Hey :wave:
I just ported the stuff from v1.x over to v3.x that allows us to use the Docker Machine IP (via docker-machine ip) or rather the internal docker IP (host.docker.internal) for exposing the Kubernetes API.
It will then be automatically added to the resulting Kubeconfig.

Please see PR #399 and give the new binaries a try: https://drive.google.com/drive/u/0/folders/15qbCUY5Uo-FoKFSUXaRqY_sh0j0cmbPm

[zulhfreelancer]

@iwilltry42 thank you. I've tried the new build from your PR #399. It's working fine now. I think you can go ahead and merge it.