K3d: [BUG] Multiserver cluster creation fail (x509 cert not valid for the domain <clustername>)

[Xenon341]

What did you do

• How was the cluster created?
  
  
  
  • k3d cluster create cluster-mulriserver --servers 3 --no-rollback
  
  

What did you expect to happen

Cluster should be created

Screenshots or terminal output

INFO[0000] Created network 'k3d-cluster-mulriserver'
INFO[0000] Created volume 'k3d-cluster-mulriserver-images'
INFO[0000] Creating initializing server node
INFO[0000] Creating node 'k3d-cluster-mulriserver-server-0'
INFO[0005] Creating node 'k3d-cluster-mulriserver-server-1'
INFO[0007] Creating node 'k3d-cluster-mulriserver-server-2'
ERRO[0007] Failed waiting for log message 'Wrote kubeconfig' from node 'k3d-cluster-mulriserver-server-1'
ERRO[0007] Failed to start container
ERRO[0007] Failed to create node 'k3d-cluster-mulriserver-server-2'
ERRO[0007] Failed to create node
ERRO[0007] error during connect: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/1ef9cbbe4088c9fb328cb8ee1efcc2497db06cbea73451bf1595bc40f68537ac/start": context canceled
FATA[0007] Cluster creation FAILED, rollback deactivated.

Logs of container server-1 and server-2

time="2020-10-03T16:56:28.351711193Z" level=info msg="Starting k3s v1.19.2+k3s1 (d38505b1)"
time="2020-10-03T16:56:28.548194125Z" level=fatal msg="starting kubernetes: preparing server: server https://k3d-cluster-mulriserver-server-0:6443/cacerts is not trusted: Get \"https://k3d-cluster-mulriserver-server-0:6443/cacerts\": x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc.cluster.local, localhost, not k3d-cluster-mulriserver-server-0"

Which OS & Architecture

• macOS Catalina

Which version of k3d

k3d version v3.0.2
k3s version latest (default)

Which version of docker

Client: Docker Engine - Community
 Cloud integration  0.1.18
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 16:58:31 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker info

Client:
 Debug Mode: false

Server:
 Containers: 8
  Running: 3
  Paused: 0
  Stopped: 5
 Images: 21
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 1.944GiB
 Name: docker-desktop
 ID: PMEC:N4W4:PM6J:RIY3:L6ZT:GV5G:V3CG:AZNL:7RBA:7BX6:FWYY:FEPA
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 72
  Goroutines: 74
  System Time: 2020-10-03T17:00:45.176503025Z
  EventsListeners: 5
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[iwilltry42]

Hi @Xenon341 and thanks for opening this issue! :)
That exact error message I managed to fix pretty easily.. however I've run into subsequent issues that I have to check with the k3s maintainers :+1:

[iwilltry42]

As an update: it's currently not clear if this is going to be fixed in k3d or in k3s itself. Anyway it should be fixed soon :+1:
Option 1)

• fix automatic add of the TLS SAN in k3s when first request to /cacerts is being made
• retry joining etcd if there are already too many members learning (1)

Option 2)

• add tls SAN in k3d (one line to add)
• wait for preceding server to finish learning (watch log messages) and then start the next server (prolonging the time-to-cluster)

[iwilltry42]

Hey 👋
This should be fixed in the new v3.1.3 release, please give it a try 🙂

[Xenon341]

Seems to be good ;)

I got another ERROR message when creating a cluster but not sure if it's related

INFO[0061] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
ERRO[0062] Logs from failed access process:
Server:     127.0.0.11
Address:    127.0.0.11:53

** server can't find host.docker.internal: NXDOMAIN

Non-authoritative answer:
Name:   host.docker.internal
Address: 192.168.65.2

WARN[0062] Failed to get HostIP: %+v Failed to read address for 'host.docker.internal' from nslookup response
INFO[0062] Cluster 'cluster-mulriserver' created successfully!

Should I open a new issue ?

Thanks !

[iwilltry42]

Nice, thanks for the feedback :)
Nope, that new error is a known issue with that k3s version (see release notes https://github.com/rancher/k3d/releases/tag/v3.1.3) on MacOS. We're working on that :+1: