Minikube: hyperkit missing runtime dependency

[aecolley]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): v0.24.1

• OS (e.g. from /etc/os-release): macOS Sierra 10.12.6
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperkit
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): v0.23.6
• Install tools: homebrew cask
• Others:
  docker-machine-driver-hyperkit downloaded and installed as per instructions

What happened:
minikube start fails with error message Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not find hyperkit executable. This is almost certainly because the hyperkit driver depends on something else being installed (such as the "Docker for Mac" app), but the dependency is not documented.

What you expected to happen:
Normal start as with now-deprecated xhyve vm-driver.

How to reproduce it (as minimally and precisely as possible):

$ minikube version
minikube version: v0.24.1
$ minikube config view
- WantReportErrorPrompt: false
- vm-driver: hyperkit
$ minikube status
minikube:
cluster:
kubectl:
$ command -v docker-machine-driver-hyperkit
/usr/local/bin/docker-machine-driver-hyperkit
$ shasum -a 256 /usr/local/bin/docker-machine-driver-hyperkit
2c355e637c7acfd1b87bf575833902cf124332b0637d640abf474f93423c9c3f  /usr/local/bin/docker-machine-driver-hyperkit
$ ls -ld /usr/local/bin/docker-machine-driver-hyperkit
-rwsr-xr-x  1 root  wheel  25305604  4 Dec 11:49 /usr/local/bin/docker-machine-driver-hyperkit
$ command -v hyperkit
$ command -v com.docker.hyperkit
$ command -v /usr/local/bin/hyperkit
$ command -v /Applications/Docker.app/Contents/Resources/bin/hyperkit
$ command -v /Applications/Docker.app/Contents/MacOS/com.docker.hyperkit
$ minikube start
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
E1204 12:59:53.054546   16073 start.go:150] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not find hyperkit executable.

 Retrying.
E1204 12:59:53.057590   16073 start.go:156] Error starting host:  Error creating host: Error creating machine: Error in driver during machine creation: Could not find hyperkit executable
$ minikube status
minikube: Stopped
cluster:
kubectl:
$

Output of minikube logs (if applicable):

F1204 13:01:03.926269   16103 logs.go:50] Error getting cluster bootstrapper: getting localkube bootstrapper: getting ssh client: Error dialing tcp via ssh client: dial tcp :22: getsockopt: connection refused

Anything else do we need to know:
The error message comes from checkHyperKit in the go/hyperkit.go file of the moby/hyperkit repo, when it has failed to find any of the elements of the defaultHyperKits array as executable commands on the runtime PATH. In the to-reproduce section of this report, all 5 elements are checked for and they are not present.

[jwandrews]

@aecolley I also ran into this issue. I ended up going to moby/hyperkit and followed the instructions there to build the hyperkit executable and then moved it into my PATH.

Not sure if this is the recommended direction. If not, maybe the recommended way should be prescribed in the docs with the driver installation here

[southwolf]

@jwandrews @aecolley You do not have to install hyperkit yourself. It is included in Docker for Mac. Just install it from HERE

[jwandrews]

@southwolf Yeah that’s another way, I don’t use the Docker for Mac daemon for my use case, so it wasn’t my first thought.

[r2d4]

You should download the hyperkit driver from our releases at https://github.com/kubernetes/minikube/releases/download/v0.24.1/docker-machine-driver-hyperkit

You can find the full instructions here:
https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver

[roychowdhuryrohit-dev]

@jwandrews @aecolley You do not have to install hyperkit yourself. It is included in Docker for Mac. Just install it from HERE

I have Docker for Mac daemon running. Running minikube start --vm-driver=hyperkit results into following error

Error starting host: Error creating new host: Driver "hyperkit" not found. Do you have the plugin binary "docker-machine-driver-hyperkit" accessible in your PATH?.

I am following this official doc https://kubernetes.io/docs/tutorials/hello-minikube/

[ttelfer]

Also running into the same error as @roychowdhuryrohit-dev.

[jwandrews]

@roychowdhuryrohit-dev @ttelfer

as per @r2d4 comment above
you also need to follow the steps to get the docker-machine driver outlined here

[ttelfer]

@jwandrews

I followed these instructions and they worked but I'm a bit confused as I now have two hyperkit processes running, one is hyperkit and the other is com.docker.hyperkit. When I disable kubernetes in Docker for Mac it stops com.docker.hyperkit, but that seems to not affect my minikube cluster. Am I right to assume then that minikube is not using the Docker for Mac hyperkit driver?

Thanks.

[jwandrews]

@ttelfer there are two things happening here.

The docker for Mac app is only required for the hyperkit executable to be installed and linked into your environment. Once you’ve ran it once, it doesn’t need to be running anymore.

The docker machine driver is what minikube is actually using to start the docker machine vm needed.

I believe that you can also just install the hyperkit executable yourself from https://github.com/moby/hyperkit or via homebrew. Without the need to install the docker for Mac app itself.

[tjanson]

Thanks for the explanations. The Hyperkit driver is available via brew too (brew install docker-machine-driver-hyperkit, read and follow Caveats), so maybe add that to the tutorial and/or the driver install instructions?

To be honest, if you’re just looking to try a quick hello world tutorial, suddenly being exposed to gory details of virtualization tech is a bit daunting. Hiding it behind a brew command feels much better. ;)

[vineethvijay]

@aecolley I also ran into this issue. I ended up going to moby/hyperkit and followed the instructions there to build the hyperkit executable and then moved it into my PATH.

Not sure if this is the recommended direction. If not, maybe the recommended way should be prescribed in the docs with the driver installation here

Works for me!

[roychowdhuryrohit-dev]

Alright I am posting the solution for those facing the issue in future.

curl -Lo docker-machine-driver-hyperkit https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit \
&& chmod +x docker-machine-driver-hyperkit \
&& sudo cp docker-machine-driver-hyperkit /usr/local/bin/ \
&& rm docker-machine-driver-hyperkit \
&& sudo chown root:wheel /usr/local/bin/docker-machine-driver-hyperkit \
&& sudo chmod u+s /usr/local/bin/docker-machine-driver-hyperkit

[oshev]

Didn't help. I'm still getting "Error creating host: Error creating machine: Error in driver during machine creation: new-ing Hyperkit: Could not find hyperkit executable."

[jwandrews]

If you just need the hyperkit executable and not the full Docker for Mac app, you can install via homebrew: brew install hyperkit

[pratiks]

Hope these steps help someone else that runs into dependency issues and are in the middle of similar errors posted here:

Install Steps

$ brew cask install minikube
$ brew install hyperkit
$ brew install docker-machine-driver-hyperkit
$ sudo chown root:wheel /usr/local/bin/docker-machine-driver-hyperkit && sudo chmod u+s /usr/local/bin/docker-machine-driver-hyperkit
$ minikube config set vm-driver hyperkit
$ minikube start --vm-driver hyperkit -v 10
$ minikube dashboard # to see your cluster

Reset Steps I took

$ minikube delete
$ rm -rf ~/.minikube # or you can mv ~/.minikube ~/.minikube.bak
restart docker from your menu
$ sudo rm /var/db/dhcpd_leases # if existed
$ minikube start --vm-driver hyperkit -v 10

[SantoshSah]

@pratiks , thanks. It worked but another new process hyperkit is running along with com.docker.hyperkit process. Can't we use com.docker.hyperkit for minikube?

[afbjorklund]

If I understand correctly, one is used for running the Minikube VM and one is used for running the Docker VM

[jwandrews]

@afbjorklund correct. @SantoshSah The com.docker.hyperkit process is started by Docker for Mac. If you do use Docker for Mac for other things already, there is no need to install the hyperkit executable from homebrew since Docker for Mac bundles the executable for you and puts it in your path.

[SantoshSah]

@jwandrews , this was my assumption that already installed hyperkit by Docker for Mac should work with Minikube but it did not so installed again as mentioned by @pratiks and it is working now.

[afbjorklund]

It is probably a good idea to keep two binaries (of the "same" thing), for exactly this reason...

It's a problem when it gets put in the path, we have the same issue with runc (and containerd)
Normally we try to work around it by choosing "compatible" versions, but sometimes that fails.
And then you are left with hacks such as the above com.docker.hyperkit, or docker-runc
Ideally they should work together with the "upstream" projects, and everyone use the same one.