K3d: [BUG] docker machine IP doesn't make sense and failed to start

Created on 12 Nov 2020 · 8Comments · Source: rancher/k3d

What did you do

How was the cluster created?
- k3d cluster create -x A -y B

k3d cluster create

What did you do afterwards?
- k3d commands?
- docker commands?
- OS operations (e.g. shutdown/reboot)?

Nothing.

What did you expect to happen

Working cluster as per older k3d versions

Screenshots or terminal output

k3d cluster create
INFO[0000] Using the docker machine IP 92.242.132.24 to connect to the Kubernetes API 
INFO[0000] Network with name 'k3d-k3s-default' already exists with ID 'ef48cd1c99c066794d5559527671dd116221ecc9d79c428e6355bf2cd111fdd0' 
INFO[0000] Created volume 'k3d-k3s-default-images'      
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0002] Pulling image 'docker.io/rancher/k3s:v1.18.9-k3s1' 
INFO[0068] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0069] Pulling image 'docker.io/rancher/k3d-proxy:v3.2.0' 
ERRO[0108] Failed to start container                    
ERRO[0108] Failed to create node 'k3d-k3s-default-serverlb' 
ERRO[0108] Failed to create loadbalancer                
ERRO[0108] Error response from daemon: Ports are not available: listen tcp 92.242.132.24:53615: bind: can't assign requested address 
ERRO[0108] Failed to create cluster >>> Rolling Back    
INFO[0108] Deleting cluster 'k3s-default'               
INFO[0108] Deleted k3d-k3s-default-server-0             
INFO[0108] Deleted k3d-k3s-default-serverlb             
INFO[0108] Deleting cluster network 'ef48cd1c99c066794d5559527671dd116221ecc9d79c428e6355bf2cd111fdd0' 
FATA[0108] Cluster creation FAILED, all changes have been rolled back!

Which OS & Architecture

Linux, Windows, MacOS / amd64, x86, ...?'

MacOS with Docker Desktop 20 running

Which version of `k3d`

output of k3d version

k3d version v3.2.0
k3s version v1.18.9-k3s1 (default)

Which version of docker

output of docker version and docker info

Client: Docker Engine - Community
 Cloud integration: 0.1.22
 Version:           20.10.0-beta1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        ac365d7
 Built:             Tue Oct 13 18:13:53 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0-beta1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       9c15e82
  Built:            Tue Oct 13 18:17:18 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.4.1
  GitCommit:        c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

bug

Source

alexellis

👍2

Most helpful comment

@nickgerace cleaned up the logs in a follow-up commit just now, thanks for testing :+1:
I'm also setting up a MacOS VM now for future testing on Mac :grimacing:

iwilltry42 on 13 Nov 2020

👍3

All 8 comments

Hey @alexellis thanks for opening this issue!
I just had a chat about this today with some other people.
Unfortunately I have no way of testing the changes and issues on a Mac, so I rely on community feedback.
According to #388 , we needed to use the docker-machine IP to expose the Kubernetes API on Mac that uses docker-machine.
While I think we could rather make this optional, I'd first like to understand, what's going on there. :thinking:
Can you help me understand/debug this?
Would it be enough to use the docker machine IP only for the kubeconfig and a TLS SAN and simple map the port to 0.0.0.0?
Why does it work for some (like in #388) but not others (like in your case)?
I guess I have to see that I get a MacOS VM set up or get hold of an old Mac :grimacing:

iwilltry42 on 12 Nov 2020

👍1

Nobody is using docker machine AFAIK on MacOS. Do you mean Docker Desktop?

I would just turn off the "great new feature" in a patch release and have whichever Mac user is desperate for this functionality test it properly and fix it themselves.

alexellis on 12 Nov 2020

It seems like the issue is that some users _are_ using docker-machine on macOS. Whether or not they should switch to Docker Desktop is one thing, but I'm wondering if it's possible to detect which method the user is using @iwilltry42?

nickgerace on 12 Nov 2020

Just tried to repro with docker-machine and Docker Desktop (3.1.5 and 3.2.0). The new feature appears to be necessary for networking for the VirtualBox VM. However, it's a breaking change for Docker Desktop users.

As stated in my above message, I think that the new ports change should be sandboxed. Perhaps, it should be toggled feature that is only allowed on Linux hosts (VM or bare metal). Thoughts?

nickgerace on 12 Nov 2020

Okay, honestly, I never used a Mac in my life and always relied on user/community testing for it.
And so I just learned a minute ago, that there's docker-machine and Docker for Desktop switched to HyperKit VM :thinking:

I'm pretty sure, that the thing that's breaking it for DfD users (non docker-machine) is the fallback method of using host.docker.internal instead of docker-machine ip, if the latter doesn't work.
I will revert this change for sure and push out a patch release now.

Sorry for the issues!

iwilltry42 on 12 Nov 2020

Please give https://github.com/rancher/k3d/releases/tag/v3.2.1 a try

iwilltry42 on 12 Nov 2020

The hotfix appears to have worked (macOS 10.15.7 with Docker Desktop 2.5.0.1). We may want to have cleaner output, but thanks for the quick hotfix!

[nickgerace at rancherbook in ~/local/bin] (0) 
% k3d version
k3d version v3.2.1
k3s version v1.19.3-k3s2 (default)
[nickgerace at rancherbook in ~/local/bin] (0) 
% k3d cluster create
INFO[0000] Error executing 'docker-machine ip'          
INFO[0000] Docker machine "" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one. 
WARN[0000] Using docker-machine, but failed to get it's IP: exit status 1 
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created volume 'k3d-k3s-default-images'      
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0009] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0010] Pulling image 'docker.io/rancher/k3d-proxy:v3.2.1' 
INFO[0013] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
INFO[0017] Successfully added host record to /etc/hosts in 2/2 nodes and to the CoreDNS ConfigMap 
INFO[0017] Cluster 'k3s-default' created successfully!  
INFO[0017] You can now use it like this:                
kubectl cluster-info
[nickgerace at rancherbook in ~/local/bin] (0) 
% kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS              RESTARTS   AGE
kube-system   local-path-provisioner-7ff9579c6-t55dx   0/1     ContainerCreating   0          3s
kube-system   helm-install-traefik-cm5jv               0/1     ContainerCreating   0          3s
kube-system   coredns-66c464876b-278v9                 0/1     ContainerCreating   0          3s
kube-system   metrics-server-7b4f8b595-4mv24           0/1     ContainerCreating   0          3s

nickgerace on 12 Nov 2020

@nickgerace cleaned up the logs in a follow-up commit just now, thanks for testing :+1:
I'm also setting up a MacOS VM now for future testing on Mac :grimacing:

iwilltry42 on 13 Nov 2020

👍3

Was this page helpful?

0 / 5 - 0 ratings