Minikube: --embed-certs=true in v1.9.0: client.crt: The system cannot find the file specified.

[Larswa]

This is a new bug in 1.9.0. Earlier versions did not exhibit this behavior.
I get this failure when producing a new minikube instance with the --embed-certs=true param/value.

If I run this command to get the full cert in the kubeconfig on an existing minikube instance it works just fine, and I get the cert written into my kube config file.
I suspect this could be related to #7125 ?

The exact command to reproduce the issue:
Run this to create a new Minikube instance:
minikube start --embed-certs=true

The full output of the command that failed:

* minikube v1.9.0 on Microsoft Windows 10 Pro 10.0.19041 Build 19041
* Automatically selected the hyperv driver. Other choices: docker, virtualbox
* Creating hyperv VM (CPUs=8, Memory=20000MB, Disk=20000MB) ...
* Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
*
X Failed to setup kubeconfig: reading ClientCertificate C:\Users\lars\.minikube\profiles\minikube\client.crt: open C:\Users\
lars\.minikube\profiles\minikube\client.crt: The system cannot find the file specified.
*
* minikube is exiting due to an error. If the above message is not useful, open an issue:
  - https://github.com/kubernetes/minikube/issues/new/choose

The output of the minikube logs command:

* ==> Docker <==
* -- Logs begin at Fri 2020-03-27 23:29:11 UTC, end at Sat 2020-03-28 00:29:10 UTC. --
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.240746627Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.240770927Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.240825027Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.240995827Z" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241023527Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241034827Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241039027Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/4.19.107\n": exit status 1"
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241043727Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin"
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241091127Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241101627Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241140327Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241170727Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241180327Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241187827Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241194727Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241201527Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241207627Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241213827Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241255327Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241302827Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241545627Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241587627Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241624327Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241632227Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241639027Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241644827Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241651727Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241659727Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241666027Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241671727Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241677927Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241699227Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241707327Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241713527Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241732727Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241836327Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock"
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241884127Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock"
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.241893827Z" level=info msg="containerd successfully booted in 0.002802s"
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.253996027Z" level=info msg="parsed scheme: \"unix\"" module=grpc
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.254033227Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.254050227Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.254060027Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.255249227Z" level=info msg="parsed scheme: \"unix\"" module=grpc
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.255291127Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.255307427Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
* Mar 27 23:29:36 minikube dockerd[2782]: time="2020-03-27T23:29:36.255325427Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.928908927Z" level=warning msg="Your kernel does not support cgroup blkio weight"
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.928978427Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.928996727Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.929001427Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.929005927Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.929009927Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.929156527Z" level=info msg="Loading containers: start."
* Mar 27 23:29:37 minikube dockerd[2782]: time="2020-03-27T23:29:37.979575927Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
* Mar 27 23:29:38 minikube dockerd[2782]: time="2020-03-27T23:29:38.002073427Z" level=info msg="Loading containers: done."
* Mar 27 23:29:38 minikube dockerd[2782]: time="2020-03-27T23:29:38.016991727Z" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8
* Mar 27 23:29:38 minikube dockerd[2782]: time="2020-03-27T23:29:38.017049127Z" level=info msg="Daemon has completed initialization"
* Mar 27 23:29:38 minikube dockerd[2782]: time="2020-03-27T23:29:38.027196427Z" level=info msg="API listen on /var/run/docker.sock"
* Mar 27 23:29:38 minikube dockerd[2782]: time="2020-03-27T23:29:38.027223927Z" level=info msg="API listen on [::]:2376"
* Mar 27 23:29:38 minikube systemd[1]: Started Docker Application Container Engine.
* 
* ==> container status <==
* time="2020-03-27T23:31:59Z" level=fatal msg="failed to connect: failed to connect, make sure you are running as root and the runtime has been started: context deadline exceeded"
* CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
* 
* ==> describe nodes <==
* 
* ==> dmesg <==
* [Mar27 23:29] You have booted with nomodeset. This means your GPU drivers are DISABLED
* [  +0.000000] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly
* [  +0.000001] Unless you actually understand what nomodeset does, you should reboot without enabling it
* [  +0.076462] Spectre V2 : Spectre mitigation: LFENCE not serializing, switching to generic retpoline
* [  +0.009619] Decoding supported only on Scalable MCA processors.
* [  +0.477942] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
*               * this clock source is slow. Consider trying other clock sources
* [  +1.923247] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
* [  +0.000049] Unstable clock detected, switching default tracing clock to "global"
*               If you want to keep using the local clock, then add:
*                 "trace_clock=local"
*               on the kernel command line
* [  +0.445835] psmouse serio1: trackpoint: failed to get extended button data, assuming 3 buttons
* [  +0.916789] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
* [  +0.002364] systemd-fstab-generator[1303]: Ignoring "noauto" for root device
* [  +0.001854] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
* [  +0.000001] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
* [  +1.050584] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
* [  +0.231407] vboxguest: loading out-of-tree module taints kernel.
* [  +0.002750] vboxguest: PCI device not found, probably running on physical hardware.
* [ +14.543353] systemd-fstab-generator[2537]: Ignoring "noauto" for root device
* [ +12.294299] kauditd_printk_skb: 65 callbacks suppressed
* [Mar27 23:31] NFSD: Unable to end grace period: -110
* 
* ==> kernel <==
*  23:31:59 up 2 min,  0 users,  load average: 0.20, 0.11, 0.04
* Linux minikube 4.19.107 #1 SMP Thu Mar 26 11:33:10 PDT 2020 x86_64 GNU/Linux
* PRETTY_NAME="Buildroot 2019.02.10"
* 
* ==> kubelet <==
* -- Logs begin at Fri 2020-03-27 23:29:11 UTC, end at Sat 2020-03-28 00:29:10 UTC. --
* -- No entries --

! unable to fetch logs for: describe nodes

The operating system version:
Windows 10 Pro 10.0.19041 Build 19041

[tstromberg]

Sorry about that. I believe you are correct, that this is likely is an unintended consequence of #7125 - It seems like there are no integration tests for --embed-certs=true.

We should definitely fix the last of tests before marking this issue as closed, otherwise it will be broken again in the future.

[Larswa]

Hi @tstromberg yeah, I took a look at #7125 but I can't find my way around Go well enough to track it down.
If it is of any help to you, embed-certs works fine, if I run it after the cluster has been spun up without that flag. It only failed if included in the initial minkube start

Also ... thats an easy workaround untill this is tracked down and fixed.

[tstromberg]

@Larswa - to be honest, I wrote this PR, and I still don't understand how the --embed-certs feature ever worked. The two packages seem to have a very loose coupling with one another, so I assume it just worked via implicit assumptions. Your hint is very useful though, as it leads me to believe that the assumption may have been in the order-of-operations.

Tracking the issue down will take a bit of local testing and likely a few new log messages. Likely just a one-liner to fix.

Thank you for bringing it to our attention!

[linkvt]

I guess this problem exists already since a bit longer but only shows now, see e.g. with a fresh v1.4.0:

 $ make && rm -rf ~/.minikube && out/minikube start --embed-certs
🙄  minikube v1.4.0 on Darwin 10.15.3
💿  Downloading VM boot image ...
    > minikube-v1.4.0.iso.sha256: 65 B / 65 B [--------------] 100.00% ? p/s 0s
    > minikube-v1.4.0.iso: 135.73 MiB / 135.73 MiB [-] 100.00% 8.55 MiB p/s 16s
🔥  Creating virtualbox VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.16.0 on Docker 18.09.9 ...

💣  Failed to setup kubeconfig: reading CertificateAuthority /Users/vincent/.minikube/ca.crt: open /Users/vincent/.minikube/ca.crt: no such file or directory

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new/choose

I guess it worked for you back then because there were some certificate inside of ~/.minikube from a start without --embed-certs, but since they are being picked up now from the new profiles directory there aren't any causing it to fail always.
I will take a look at it but don't want to block anyone of the official maintaners here if they're faster.

[Larswa]

@tstromberg Thanks for making me feel a litle less stupid. 😂

@linkvt I think this has been working in earlier version as I have had a script for a couple of months, running a minkube delete and a minikube start --embed-certs=true with the
At least for the 1.8.x versions I didnt have this issue. I think.

[linkvt]

@Larswa just try it with deleting the ~/.minikube directory first as I did, minikube delete doesn't remove the certificates inside of ~/.minikube.
I just opened a PR, you could try it out.

/assign linkvt

[Larswa]

Hey @linkvt Thanks. The easiest workaround I found was to run minikube start with other options, and then once that was complete, run minikube start --embed-certs=true on the running instance. That works fine.

[tstromberg]

@Larswa - if you want to try a Windows binary built from linkvt's PR:

https://storage.googleapis.com/minikube-builds/7309/minikube-windows-amd64.exe

Having confirmation would be useful.

[Larswa]

@tstromberg @linkvt I can confirm that it worked! Very cool!

I removed existing minkube instance, removed the .minikube folder in my user profile and did a minikube start --embed-certs=true and got a new minikube instance with no cert errors.

[Larswa]

Whoops .. dunno if this should have been closed already without linking it to the pr.

[tstromberg]

Thank you @Larswa for the confirmation! This issue will auto-merge once the PR is merged.