K3d: [BUG] Unable to connect to the server: x509: certificate signed by unknown authority

[jazzdan]

What did you do?

• How was the cluster created?
  
  
  
  • k3d --verbose create
  
  

What did you expect to happen?

I expected kubectl cluster-info to return a 0 exit code with some information about my cluster.

Screenshots or terminal output
Instead it returned this:

❯ export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"

❯ kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: x509: certificate signed by unknown authority

Which OS & Architecture?

• macOS 10.14.6 x86_64

Which version of k3d?

❯ k3d --version
k3d version v1.3.4

Which version of docker?

❯ docker version
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

kubeconfig that k3d generated:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJWakNCL3FBREFnRUNBZ0VBTUFvR0NDcUdTTTQ5QkFNQ01DTXhJVEFmQmdOVkJBTU1HR3N6Y3kxelpYSjIKWlhJdFkyRkFNVFUzTXpRNU9EQTVOREFlRncweE9URXhNVEV4T0RRNE1UUmFGdzB5T1RFeE1EZ3hPRFE0TVRSYQpNQ014SVRBZkJnTlZCQU1NR0dzemN5MXpaWEoyWlhJdFkyRkFNVFUzTXpRNU9EQTVOREJaTUJNR0J5cUdTTTQ5CkFnRUdDQ3FHU000OUF3RUhBMElBQkdHRWFWcVlOMU4xdjJFNU1KSVVpbXpiSkZmOXFXaHlvVGFPRlNsQk40U2UKT2FtRjhZTkZOanFVWmhtTlZUZWpLKzBKU0lQN2ZkdEwyRWJseWZjamFMdWpJekFoTUE0R0ExVWREd0VCL3dRRQpBd0lDcERBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUFvR0NDcUdTTTQ5QkFNQ0EwY0FNRVFDSUFhcUNPNTlQRWhmCitScW1kdUlTTll2cWtoS0lUUitFZEN6VzVqZjNzWHlDQWlBV0lieHZ5ckZwRHlUb3Rpd3AxZTRDZTcxOU9zMnMKUHF0QVMwR2hWQlBwaWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    password: d4f785f4ea1fb5a9ed7c7048c8380e6c
    username: admin

[iwilltry42]

Hey there, thanks for opening this issue.
I'm on my phone right now, so sorry if I overread some piece of information.

In your cluster creation command, you didn't specify an --api-port to expose the API to your local machine, did you?

And if you do, k3d should try to detect the docker-machine IP (Docker VM IP) and put it in the Kubeconfig.

[jazzdan]

Hi @iwilltry42,

In your cluster creation command, you didn't specify an --api-port to expose the API to your local machine, did you?

I did not! I created it exactly as in the bug report.

And if you do, k3d should try to detect the docker-machine IP (Docker VM IP) and put it in the Kubeconfig.

Specifying --api-port to something other than `localhost:6443 fixed it! For example:

❯ k3d create --api-port localhost:6444
INFO[0000] Created cluster network with ID e94d9c3caa6af8498a2957b53266adc00abd25f84225a653d41166f8498b0adc
INFO[0000] As of v2.0.0 --port will be used for arbitrary port mapping. Please use --api-port/-a instead for configuring the Api Port
INFO[0000] Add TLS SAN for localhost
INFO[0000] Created docker volume  k3d-k3s-default-images
INFO[0000] Creating cluster [k3s-default]
INFO[0000] Creating server using docker.io/rancher/k3s:v0.10.0...
INFO[0000] SUCCESS: created cluster [k3s-default]
INFO[0000] You can now use the cluster with:

export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl cluster-info

❯ export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
FATA[0000]  Couldn't copy kubeconfig.yaml from server container c43cbe5187673e2417978e3a6e4b05fba4a93d4a5d68c9409a8c6b9b83562434
Error: No such container:path: c43cbe5187673e2417978e3a6e4b05fba4a93d4a5d68c9409a8c6b9b83562434:/output/kubeconfig.yaml

❯ export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"

❯ kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:6444
CoreDNS is running at https://127.0.0.1:6444/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Is this a bug in k3d or did I misunderstand some docks somewhere? Either way thanks for the help @iwilltry42 !

[iwilltry42]

Docker for Mac is running in a VM (Docker machine). Usually the issue there is the missing SAN in the certificate used for the cluster, because the cert was issued on a virtually different host.
I'm not really used to the Docker for Mac stuff, since I've never worked with it, but I know that we've added some functionality for it.

Anyway, after all this is not a bug in k3d 👍
I'll have a look tomorrow if we need more clear logs on this.
I'm bad at browsing through the code on my phone..

[rjshrjndrn]

Can you try this @jazzdan
k3d create --server-arg --tls-san="127.0.0.1"

[jazzdan]

Hey @rjshrjndrn,

That resulted in the same problem, unfortunately:

❯ k3d delete
INFO[0000] Removing cluster [k3s-default]
INFO[0000] ...Removing server
INFO[0000] ...Removing docker image volume
INFO[0000] Removed cluster [k3s-default]

❯ k3d create --server-arg --tls-san="127.0.0.1"
INFO[0000] Created cluster network with ID 257749f41cb95b47701db290c6dbf0fa9c7fc916a8158879ae6d5e5528643f53
INFO[0000] Created docker volume  k3d-k3s-default-images
INFO[0000] Creating cluster [k3s-default]
INFO[0000] Creating server using docker.io/rancher/k3s:v0.10.0...
INFO[0001] SUCCESS: created cluster [k3s-default]
INFO[0001] You can now use the cluster with:

export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl cluster-info

❯ export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"

❯ kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: x509: certificate signed by unknown authority

[nabadger]

Also being affected by this - on MacOS. The workaround at the moment is the same as the reporter (specify --api-port 6444).

[u007]

this works on my ubuntu:

k3d create --api-port localhost:6444 --publish 8080:80 --server-arg --tls-san="127.0.0.1" 

[lewismc]

@nabadger do you mean --api-port localhost:6444 rather than --api-port 6444?

[developer-guy]

@nabadger do you mean --api-port localhost:6444 rather than --api-port 6444?

this one works for me.

$ k3d create --api-port 6444