Minikube: Unable to read client-cert/key Post Minikube Update OSX

Created on 3 Jun 2020  ·  3Comments  ·  Source: kubernetes/minikube

Updated Minikube on my Mac using 

brew upgrade minikube
brew link minikube

Then I started Minikube anew:

minikube start    
😄  minikube v1.11.0 on Darwin 10.15.5
✨  Using the hyperkit driver based on existing profile
💾  Downloading driver docker-machine-driver-hyperkit:
    > docker-machine-driver-hyperkit.sha256: 65 B / 65 B [---] 100.00% ? p/s 0s
    > docker-machine-driver-hyperkit: 10.90 MiB / 10.90 MiB  100.00% 2.41 MiB p
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/jasper/.minikube/bin/docker-machine-driver-hyperkit 
    $ sudo chmod u+s /Users/jasper/.minikube/bin/docker-machine-driver-hyperkit 


Password:
🆕  Kubernetes 1.18.3 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.18.3
🆕  Kubernetes 1.18.3 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.18.3
🆕  Kubernetes 1.18.3 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.18.3
💿  Downloading VM boot image ...
    > minikube-v1.11.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.11.0.iso: 174.99 MiB / 174.99 MiB [] 100.00% 4.43 MiB p/s 39s
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.17.0 preload ...
    > preloaded-images-k8s-v3-v1.17.0-docker-overlay2-amd64.tar.lz4: 522.40 MiB
🔄  Restarting existing  VM for "minikube" ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
💡  Existing disk is missing new features (lz4). To upgrade, run 'minikube delete'
🐳  Preparing Kubernetes v1.17.0 on Docker 19.03.5 ...
    > kubectl.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubeadm.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubelet: 106.39 MiB / 106.39 MiB [-------------] 100.00% 4.93 MiB p/s 22s
    > kubectl: 41.48 MiB / 41.48 MiB [---------------] 100.00% 1.39 MiB p/s 30s
    > kubeadm: 37.52 MiB / 37.52 MiB [-------------] 100.00% 977.04 KiB p/s 40s
🔎  Verifying Kubernetes components...
❗  Enabling 'default-storageclass' returned an error: running callbacks: [Error getting storagev1 interface client config: invalid configuration: [unable to read client-cert /Users/jasper/.minikube/profiles/minikube/client.crt for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.crt: no such file or directory, unable to read client-key /Users/jasper/.minikube/profiles/minikube/client.key for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.key: no such file or directory] : client config: invalid configuration: [unable to read client-cert /Users/jasper/.minikube/profiles/minikube/client.crt for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.crt: no such file or directory, unable to read client-key /Users/jasper/.minikube/profiles/minikube/client.key for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.key: no such file or directory]]
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

The main issues indicated being

Enabling 'default-storageclass' returned an error: running callbacks: [Error getting storagev1 interface client config: invalid configuration: [unable to read client-cert /Users/jasper/.minikube/profiles/minikube/client.crt for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.crt: no such file or directory, unable to read client-key /Users/jasper/.minikube/profiles/minikube/client.key for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.key: no such file or directory] : client config: invalid configuration: [unable to read client-cert /Users/jasper/.minikube/profiles/minikube/client.crt for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.crt: no such file or directory, unable to read client-key /Users/jasper/.minikube/profiles/minikube/client.key for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.key: no such file or directory]]

Client Cert and Key Missing

They are not there. I checked:

  minikube pwd
/Users/jasper/.minikube/profiles/minikube
➜  minikube ll 
total 8
-rw-------  1 jasper  staff   1.6K Jun  3 06:34 config.json

Json Config

Only the config file:

cat config.json 
{
        "Name": "minikube",
        "KeepContext": false,
        "EmbedCerts": false,
        "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.11.0.iso",
        "KicBaseImage": "",
        "Memory": 2000,
        "CPUs": 2,
        "DiskSize": 20000,
        "Driver": "",
        "HyperkitVpnKitSock": "",
        "HyperkitVSockPorts": [],
        "DockerEnv": null,
        "InsecureRegistry": null,
        "RegistryMirror": null,
        "HostOnlyCIDR": "192.168.99.1/24",
        "HypervVirtualSwitch": "",
        "HypervUseExternalSwitch": false,
        "HypervExternalAdapter": "",
        "KVMNetwork": "default",
        "KVMQemuURI": "qemu:///system",
        "KVMGPU": false,
        "KVMHidden": false,
        "DockerOpt": null,
        "DisableDriverMounts": false,
        "NFSShare": [],
        "NFSSharesRoot": "/nfsshares",
        "UUID": "",
        "NoVTXCheck": false,
        "DNSProxy": false,
        "HostDNSResolver": true,
        "HostOnlyNicType": "virtio",
        "NatNicType": "virtio",
        "KubernetesConfig": {
                "KubernetesVersion": "v1.17.0",
                "ClusterName": "",
                "APIServerName": "minikubeCA",
                "APIServerNames": null,
                "APIServerIPs": null,
                "DNSDomain": "cluster.local",
                "ContainerRuntime": "docker",
                "CRISocket": "",
                "NetworkPlugin": "",
                "FeatureGates": "",
                "ServiceCIDR": "10.96.0.0/12",
                "ImageRepository": "",
                "LoadBalancerStartIP": "",
                "LoadBalancerEndIP": "",
                "ExtraOptions": null,
                "ShouldLoadCachedImages": true,
                "EnableDefaultCNI": false,
                "NodeIP": "",
                "NodePort": 8443,
                "NodeName": ""
        },
        "Nodes": [
                {
                        "Name": "minikube",
                        "IP": "192.168.64.3",
                        "Port": 8443,
                        "KubernetesVersion": "v1.17.0",
                        "ControlPlane": true,
                        "Worker": true
                }
        ],
        "Addons": {
                "default-storageclass": true,
                "storage-provisioner": true
        },
        "VerifyComponents": null
}%

Optional: Full output of minikube logs command:

Jun 02 23:35:33 minikube kubelet[2735]: W0602 23:35:33.917720    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for laravel6/nginx-ingress-controller-69d5dc598f-zfpwd through plugin: invalid network status for
Jun 02 23:35:33 minikube kubelet[2735]: W0602 23:35:33.924427    2735 pod_container_deletor.go:75] Container "5d44edd8367c4c968409531eb90ec96c5ae7f00975f612859591164be137f29b" not found in pod's containers
Jun 02 23:35:33 minikube kubelet[2735]: W0602 23:35:33.927601    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/nginx-ingress-controller-6fc5bcc8c9-f72c8 through plugin: invalid network status for
Jun 02 23:35:34 minikube kubelet[2735]: W0602 23:35:34.975809    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/nginx-ingress-controller-6fc5bcc8c9-f72c8 through plugin: invalid network status for
Jun 02 23:35:45 minikube kubelet[2735]: W0602 23:35:45.211982    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for laravel6/cert-manager-cainjector-76f7596c4-v8n6c through plugin: invalid network status for
Jun 02 23:35:45 minikube kubelet[2735]: E0602 23:35:45.348749    2735 pod_workers.go:191] Error syncing pod d07c7ee2-eceb-4f81-8873-a4fdbceb0e03 ("cert-manager-cainjector-76f7596c4-v8n6c_laravel6(d07c7ee2-eceb-4f81-8873-a4fdbceb0e03)"), skipping: failed to "StartContainer" for "cert-manager" with CrashLoopBackOff: "back-off 10s restarting failed container=cert-manager pod=cert-manager-cainjector-76f7596c4-v8n6c_laravel6(d07c7ee2-eceb-4f81-8873-a4fdbceb0e03)"
Jun 02 23:35:45 minikube kubelet[2735]: W0602 23:35:45.358343    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kubernetes-dashboard/kubernetes-dashboard-79d9cd965-gpgzb through plugin: invalid network status for
Jun 02 23:35:46 minikube kubelet[2735]: W0602 23:35:46.387883    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for laravel6/cert-manager-cainjector-76f7596c4-v8n6c through plugin: invalid network status for
Jun 02 23:35:49 minikube kubelet[2735]: E0602 23:35:49.492229    2735 pod_workers.go:191] Error syncing pod b9485329-1eda-4eed-a97c-690c4336df47 ("storage-provisioner_kube-system(b9485329-1eda-4eed-a97c-690c4336df47)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b9485329-1eda-4eed-a97c-690c4336df47)"
Jun 02 23:35:58 minikube kubelet[2735]: W0602 23:35:58.736324    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for laravel6/cert-manager-cainjector-76f7596c4-v8n6c through plugin: invalid network status for
Jun 02 23:36:33 minikube kubelet[2735]: W0602 23:36:33.475119    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/nginx-ingress-controller-6fc5bcc8c9-f72c8 through plugin: invalid network status for
Jun 02 23:36:34 minikube kubelet[2735]: W0602 23:36:34.526508    2735 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for laravel6/cert-manager-cainjector-76f7596c4-v8n6c through plugin: invalid network status for

==> kubernetes-dashboard [6e69bf566f83] <==
2020/06/02 23:34:53 Starting overwatch
panic: Get https://10.96.0.1:443/api/v1/namespaces/kubernetes-dashboard/secrets/kubernetes-dashboard-csrf: dial tcp 10.96.0.1:443: i/o timeout

goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/clientsing in/cclsursft.e(r* ccorffiTgo kteon Mcaonnangeecrt) .tion iatp(i0sxecr0v00e3rd
500)
2020    //0h6/0o2m e2/3t:r3a4v:i5s4/ bUusiilndg/ ksuebcerrente tteosk/edna sfhobro acrsdr/fs rsci/anpi/nbga
ckend/2l0i2e0n/t0/6c/s0r2f /m3n:a3g4e:r.4o :I4n0i t+i0xlbi4z
ing csithub.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenMandasrbfo atrodken /fsrrocm/ akpupb/ebrancekteensd-/dcalsihebnota/rcds-rcfs/rmfa nsaegcer.groe:t6
5
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc000220080)
        /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:494 +0xc7
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0xc000220080)
        /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:462 +0x47
github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...)
        /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:543
main.main()
        /home/travis/build/kubernetes/dashboard/src/app/backend/dashboard.go:105 +0x212

==> kubernetes-dashboard [870cd48af713] <==
2020/06/02 23:35:45 Starting overwatch
2020/06/02 23:35:45 Using namespace: kubernetes-dashboard
2020/06/02 23:35:45 Using in-cluster config to connect to apiserver
2020/06/02 23:35:45 Using secret token for csrf signing
2020/06/02 23:35:45 Initializing csrf token from kubernetes-dashboard-csrf secret
2020/06/02 23:35:45 Empty token. Generating and storing in a secret kubernetes-dashboard-csrf
2020/06/02 23:35:45 Successful initial request to the apiserver, version: v1.17.0
2020/06/02 23:35:45 Generating JWE encryption key
2020/06/02 23:35:45 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2020/06/02 23:35:45 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2020/06/02 23:35:46 Initializing JWE encryption key from synchronized object
2020/06/02 23:35:46 Creating in-cluster Sidecar client
2020/06/02 23:35:46 Serving insecurely on HTTP port: 9090
2020/06/02 23:35:46 Successful request to sidecar

==> storage-provisioner [07d9163c5e77] <==

==> storage-provisioner [31fa9f9694c2] <==
F0602 23:35:48.583181       1 main.go:37] Error getting server version: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout

How to get the client.key and client.crt for local testing of deployments and so on?

picture jasperf

Most helpful comment

Did a nano ~/.kube/config and adjusted the path

cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/jasper/.minikube/ca.crt
    server: https://192.168.64.3:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /Users/jasper/.minikube/profiles/client.crt
    client-key: /Users/jasper/.minikube/profiles/client.key

Now all is well again:

kubectl cluster-info
Kubernetes master is running at https://192.168.64.3:8443
KubeDNS is running at https://192.168.64.3:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
picture jasperf on 3 Jun 2020
👍3

All 3 comments

Doing a brew uninstall minikube, brew install minikube nor brew reinstall minikube changed the situation. Still miss the client.cert and client.key:

kubectl cluster-info   
Error in configuration: 
* unable to read client-cert /Users/jasper/.minikube/profiles/minikube/client.crt for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.crt: no such file or directory
* unable to read client-key /Users/jasper/.minikube/profiles/minikube/client.key for minikube due to open /Users/jasper/.minikube/profiles/minikube/client.key: no such file or directory

Also config as shown here does try to load the certs.. but they are just not there.

cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/jasper/.minikube/ca.crt
    server: https://192.168.64.3:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /Users/jasper/.minikube/profiles/minikube/client.crt
    client-key: /Users/jasper/.minikube/profiles/minikube/client.key

A similar issue was mentioned at https://github.com/kubernetes/kubernetes/issues/76477 but closed and referred to the Minikube repository

jasperf picture jasperf on 3 Jun 2020

Hmm it seems they are located elsewhere

➜  minikube pwd
/Users/jasper/.minikube/profiles/minikube
➜  minikube cd ..
➜  profiles pwd
/Users/jasper/.minikube/profiles
➜  profiles ll
total 64
-rw-r--r--  1 jasper  staff   1.3K Jun  3 06:34 apiserver.crt
-rw-r--r--  1 jasper  staff   1.3K Jun  3 06:34 apiserver.crt.40f8bebd
-rw-------  1 jasper  staff   1.6K Jun  3 06:34 apiserver.key
-rw-------  1 jasper  staff   1.6K Jun  3 06:34 apiserver.key.40f8bebd
-rw-r--r--  1 jasper  staff   1.1K Jun  3 06:34 client.crt
-rw-------  1 jasper  staff   1.6K Jun  3 06:34 client.key
drwx------  3 jasper  staff    96B Jun  3 06:32 minikube
-rw-r--r--  1 jasper  staff   1.1K Jun  3 06:34 proxy-client.crt
-rw-------  1 jasper  staff   1.6K Jun  3 06:34 proxy-client.key

Will see how I can make Minikube load from correct location. Or move them. Also wonder how on earth this then changed?!

jasperf picture jasperf on 3 Jun 2020
👍1

Did a nano ~/.kube/config and adjusted the path

cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/jasper/.minikube/ca.crt
    server: https://192.168.64.3:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /Users/jasper/.minikube/profiles/client.crt
    client-key: /Users/jasper/.minikube/profiles/client.key

Now all is well again:

kubectl cluster-info
Kubernetes master is running at https://192.168.64.3:8443
KubeDNS is running at https://192.168.64.3:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
jasperf picture jasperf on 3 Jun 2020
👍3
Was this page helpful?
0 / 5 - 0 ratings

Related issues

alber70g picture alber70g  ·  3Comments
jroucheton picture jroucheton  ·  3Comments
StasPerekrestov picture StasPerekrestov  ·  3Comments
dkoston picture dkoston  ·  3Comments
hank-cp picture hank-cp  ·  3Comments