minikube with xhyve: Error starting stopped host: Something went wrong running an SSH command!
BUG REPORT
Minikube version: v0.18.0
Environment:
- OS: Mac OS Sierra 10.12.4
- VM Driver: xhyve
- ISO version: v0.18.0
What happened:
After upgrading to latest docker-machine-driver-xhyve today, version 0.3.2, minikube doesn't work properly anymore. When I initially setup minikube (minikube delete; minikube start), everything seems to works fine. But if I minikube ssh and then exit I get:
E0422 07:21:08.706220 14811 ssh.go:44] Error attempting to ssh/run-ssh-command: exit status 127
If I stop minikube and start it again, I get:
cb@Sirhc ~ $ minikube start
Starting local Kubernetes cluster...
Starting VM...
E0422 07:22:10.969284 14844 start.go:116] Error starting host: Error starting stopped host: Something went wrong running an SSH command!
command : echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=cb -o dfltuid=$(id -u docker) -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
err : exit status 32
output : #/bin/bash
sudo mkdir -p /Users
sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=cb -o dfltuid=1001 -o dfltgid=50 -o access=any host /Users
mount: host is already mounted or /Users busy
host is already mounted on /Users
Retrying.
E0422 07:22:10.970169 14844 start.go:122] Error starting host: Error starting stopped host: Something went wrong running an SSH command!
command : echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=cb -o dfltuid=$(id -u docker) -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
err : exit status 32
output : #/bin/bash
sudo mkdir -p /Users
sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=cb -o dfltuid=1001 -o dfltgid=50 -o access=any host /Users
mount: host is already mounted or /Users busy
host is already mounted on /Users
ChrisBuchholz
All 30 comments
Same here:
Minikube version: v0.18.0
Environment:
- OS: Mac OS Sierra 10.12.4
- VM Driver: xhyve
Installed minikube and docker-machine-driver-xhyve yesterday. Starting minikube for the first time went smoothly. Today, when I wanted to restart minikube I got:
[12:04|obergner@Olafs-iMac|tmp ]$ minikube start
Starting local Kubernetes cluster...
Starting VM...
E0423 12:04:37.164969 6696 start.go:116] Error starting host: Error starting stopped host: Something went wrong running an SSH command!
command : echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=obergner -o dfltuid=$(id -u docker) -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
err : exit status 32
output : #/bin/bash
sudo mkdir -p /Users
sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=obergner -o dfltuid=1001 -o dfltgid=50 -o access=any host /Users
mount: host is already mounted or /Users busy
host is already mounted on /Users
.
Retrying.
E0423 12:04:37.233056 6696 start.go:122] Error starting host: Error starting stopped host: Something went wrong running an SSH command!
command : echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=obergner -o dfltuid=$(id -u docker) -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
err : exit status 32
output : #/bin/bash
sudo mkdir -p /Users
sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=obergner -o dfltuid=1001 -o dfltgid=50 -o access=any host /Users
mount: host is already mounted or /Users busy
host is already mounted on /Users
Reverting to minikube's default VM driver fixes the problem:
[12:18|obergner@Olafs-iMac|tmp ]$ minikube config unset vm-driver
[12:18|obergner@Olafs-iMac|tmp ]$ minikube start
Starting local Kubernetes cluster...
Starting VM...
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
[12:19|obergner@Olafs-iMac|tmp ]$
obergner
on 23 Apr 2017
Same here - cannot start after stop. xhyve driver.
ash2k
on 1 May 2017
Same.
neverfox
on 1 May 2017
Are you running minikube mount as well? If the minikube mount daemon is still running, I could see this being an error. You should be able to kill that process and then start minikube normally.
Things we should probably do here:
- Make this not a fatal error in the xhyve driver.
- Turn off the 9p mount in the xhyve driver and only use minikube mount.
r2d4
on 1 May 2017
@r2d4 I don't. With fresh install:
minikube start --vm-driver=xhyve
// do some work for several hours
minikube stop
// next day or just later
minikube start --vm-driver=xhyve
// this error happens
Same problem here. Versions same as stated above.
ambis
on 2 May 2017
Ah, this was actually my fault from a PR in the xhyve driver. Previously we were calling
//Start()
...
d.setupMounts
return d.waitForIP()
...
I switched these since there was sometimes a race condition in setting up the mounts before we were able to get an IP. However, it looks like on a start/stop, the mount already exists and d.setupMounts was silently throwing an error, which we are now catching.
https://github.com/zchee/docker-machine-driver-xhyve/pull/161/files
We should probably ignore the error from setup mounts, even though that would mask actual errors.
r2d4
on 2 May 2017
Same here, Was using kube-solo then switched to minikube. However, if I run
sudo chown root:wheel /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
and
sudo chmod u+s /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
before the
minikube start --vm-driver xhyve
command, I can spin up the cluster.
samhalpa
on 4 May 2017
I can confirm that @samhalpa workaround works as I was having the same problem with the same versions reported in the original issue above.
davidthewatson
on 8 May 2017
I also see this issue pop up if I forget to reset my kubectl config to the minikube context since I have multiple contexts around... so remember to jump back to minikube context, i.e.
kubectl config use-context minikube
minikube start --vm-driver=xhyve
belldina
on 12 May 2017
@belldina You might want to check out https://github.com/ahmetb/kubectx by @ahmetb. Pretty cool tool to help you switch between your contexts.
r2d4
on 12 May 2017
@belldina
I run this command, and it works.
kubectl config use-context minikube
crayhuang
on 18 May 2017
@crayhuang I see this happen even when I'm confirmed on the minikube context.
neverfox
on 30 May 2017
Anytime I've run into this issue, I've been able to successfully work around it via the context switch posted a above:
kubectl config use-context minikube
minikube start --vm-driver=xhyve
and if that doesn't work, then ( more drastically, since it deletes all state, containers, etc... ):
minikube delete
followed by the commands above...
Of course, this is likely more destructive that other possible solutions, but I mainly use minikube and its embedded docker repo as a staging area, so deleting/re-creating them is rarely a problem for that use case.
belldina
on 30 May 2017
@belldina Right, deleting the cluster always solves it but, like you said, while practically often easy to deal with, isn't really a solution to this problem in the true sense. It's the minikube equivalent of "unplug it and plug it back in." Obviously, it shouldn't happen in the first place. :)
neverfox
on 30 May 2017
Minikube version v0.19.1 still affected by this issue.
h0tbird
on 31 May 2017
I encountered this with minikube v0.19.1 while working through the Hello Minikube tutorial and trying to debug #1442.
ceridwen
on 1 Jun 2017
I did @samhalpa workaround, after @davidthewatson confirmed it. But why does this work? And has the team located the issue? I got this problem with v.0.19.1 aswell
vongohren
on 2 Jun 2017
Please read https://github.com/kubernetes/minikube/issues/1400#issuecomment-298506787. The issue is fixed in the xhyve driver, however there has not been a release of for that binary yet.
r2d4
on 2 Jun 2017
So workaround so far is @samhalpa s suggestion? Or deleting the cluster? Or nothing? Until the binary is released
vongohren
on 2 Jun 2017
Can confirm pulling latest docker xhyve driver and following manual install / build instructions fixed issue for me.
jezell
on 5 Jun 2017
Even on latest minikube version: v0.20.0. This problem is still easy to reproduce:
- Install xhyve driver per the install docs
minikube start --vm-driver=xhyveminikube stopminikube start --vm-driver=xhyve
Same results (click to expand)
```sh
$ minikube start --vm-driver=xhyve
Starting local Kubernetes v1.6.4 cluster...
Starting VM...
E0626 15:56:44.756006 20938 start.go:127] Error starting host: Error starting stopped host: Something went wrong running an SSH command!
command : echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=USERNAME_OBFUSCATED -o dfltuid=$(id -u docker) -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
err : exit status 32
output : #/bin/bash
sudo mkdir -p /Users
sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=USERNAME_OBFUSCATED -o dfltuid=1001 -o dfltgid=50 -o access=any host /Users
mount: host is already mounted or /Users busy
host is already mounted on /Users
.
Retrying.
E0626 15:56:44.762210 20938 start.go:133] Error starting host: Error starting stopped host: Something went wrong running an SSH command!
command : echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=USERNAME_OBFUSCATED -o dfltuid=$(id -u docker) -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
err : exit status 32
output : #/bin/bash
sudo mkdir -p /Users
sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=USERNAME_OBFUSCATED -o dfltuid=1001 -o dfltgid=50 -o access=any host /Users
mount: host is already mounted or /Users busy
host is already mounted on /Users
```
We're using this gist as a workaround in the meantime: https://gist.github.com/scottrigby/0d5e79afbc00a3650f043cb7d380080d
And after waiting to re-download the binary, we must helm init again, do any additional setup, etc. Doesn't help the case for adoption of Minikube organizationally, but still working on it 馃檹 猬涳笍 鈿欙笍 馃敤
scottrigby
on 27 Jun 2017
Also note #1623 (from Slack convo).
scottrigby
on 27 Jun 2017
v0.3.3 was released; Homebrew PR submitted: https://github.com/Homebrew/homebrew-core/pull/15079
Result after installing from Homebrew via the PR:
位 minikube start
Starting local Kubernetes v1.6.4 cluster...
Starting VM...
Moving files into cluster...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
位 minikube stop
Stopping local Kubernetes cluster...
Machine stopped.
位 minikube start
Starting local Kubernetes v1.6.4 cluster...
Starting VM...
Moving files into cluster...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
:thumbsup:
tristanpemble
on 29 Jun 2017
minikube 0.20.0 still affected by the issue; however, after attempting a subsequent start it seems to work.
githubchic
on 29 Jun 2017
I can confirm the new release of docker-machine-driver-xhyve has completely fixed it for me 馃憤
scottrigby
on 4 Jul 2017
Closing this, it should be fixed with the latest version of docker-machine-driver-xhyve https://github.com/zchee/docker-machine-driver-xhyve/releases/tag/v0.3.3
r2d4
on 4 Jul 2017
The latest version of docker-machine-driver-xhyve still has the same issue.
kube-addon-manager-minikube pod can't be running.
$ cat /usr/local/opt/docker-machine-driver-xhyve/INSTALL_RECEIPT.json
{"homebrew_version":"1.2.3-72-g4fb60d898","used_options":[],"unused_options":["--without-qcow2","--without-docker-machine"],"built_as_bottle":true,"poured_from_bottle":true,"installed_as_dependency":false,"installed_on_request":true,"changed_files":[".brew/docker-machine-driver-xhyve.rb","INSTALL_RECEIPT.json"],"time":1502647700,"source_modified_time":1498701473,"HEAD":null,"stdlib":null,"compiler":"clang","aliases":[],"runtime_dependencies":[{"full_name":"docker-machine","version":"0.12.0"},{"full_name":"opam","version":"1.2.2"},{"full_name":"libev","version":"4.24"}],"source":{"path":"/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/docker-machine-driver-xhyve.rb","tap":"homebrew/core","spec":"stable","versions":{"stable":"0.3.3","devel":null,"head":"HEAD","version_scheme":0}}}%
$ ls -l $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
-r-sr-xr-x 1 root wheel 10828084 6 29 10:57 /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
$ minikube status
minikube: Running
localkube: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.64.3
$ cat ~/.minikube/machines/minikube/config.json | grep DriverName
"DriverName": "xhyve",
$ kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-addon-manager-minikube 0/1 ContainerCreating 0 40m
kube-system kubernetes-dashboard-3313488171-chktz 0/1 ContainerCreating 0 34m
$ kubectl describe pod kube-addon-manager-minikube --namespace kube-system
Name: kube-addon-manager-minikube
Namespace: kube-system
Node: minikube/192.168.64.3
Start Time: Wed, 16 Aug 2017 00:10:11 +0900
Labels: component=kube-addon-manager
kubernetes.io/minikube-addons=addon-manager
version=v6.4
Annotations: kubernetes.io/config.hash=c654b2f084cf26941c334a2c3d6db53d
kubernetes.io/config.mirror=c654b2f084cf26941c334a2c3d6db53d
kubernetes.io/config.seen=2017-08-15T15:10:06.567373808Z
kubernetes.io/config.source=file
Status: Pending
IP: 192.168.64.3
Containers:
kube-addon-manager:
Container ID:
Image: gcr.io/google-containers/kube-addon-manager:v6.4-beta.2
Image ID:
Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 5m
memory: 50Mi
Environment: <none>
Mounts:
/etc/kubernetes/ from addons (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
addons:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoExecute
Events: <none>
$ minikube logs
Aug 15 15:23:59 minikube localkube[3557]: W0815 15:23:59.990274 3557 helpers.go:432] Couldn't load Docker cofig. If sandbox image "gcr.io/google_containers/pause-amd64:3.0" is in a private registry, this will cause further errors. Error: couldn't find valid .dockercfg after checking in [/.docker .docker]
Aug 15 15:23:59 minikube localkube[3557]: E0815 15:23:59.990366 3557 remote_runtime.go:91] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = unable to pull sandbox image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: {"message":"Get https://gcr.io/v1/_ping: dial tcp: lookup gcr.io on 192.168.64.1:53: server misbehaving"}
Aug 15 15:23:59 minikube localkube[3557]: E0815 15:23:59.990392 3557 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c654b2f084cf26941c334a2c3d6db53d)" failed: rpc error: code = 2 desc = unable to pull sandbox image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: {"message":"Get https://gcr.io/v1/_ping: dial tcp: lookup gcr.io on 192.168.64.1:53: server misbehaving"}
Aug 15 15:23:59 minikube localkube[3557]: E0815 15:23:59.990403 3557 kuberuntime_manager.go:618] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c654b2f084cf26941c334a2c3d6db53d)" failed: rpc error: code = 2 desc = unable to pull sandbox image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: {"message":"Get https://gcr.io/v1/_ping: dial tcp: lookup gcr.io on 192.168.64.1:53: server misbehaving"}
Aug 15 15:23:59 minikube localkube[3557]: E0815 15:23:59.990432 3557 pod_workers.go:182] Error syncing pod c654b2f084cf26941c334a2c3d6db53d ("kube-addon-manager-minikube_kube-system(c654b2f084cf26941c334a2c3d6db53d)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c654b2f084cf26941c334a2c3d6db53d)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c654b2f084cf26941c334a2c3d6db53d)\" failed: rpc error: code = 2 desc = unable to pull sandbox image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: {\"message\":\"Get https://gcr.io/v1/_ping: dial tcp: lookup gcr.io on 192.168.64.1:53: server misbehaving\"}"
$ minikube ssh
$ nslookup gcr.io
Server: 192.168.64.1
Address 1: 192.168.64.1
nslookup: can't resolve 'gcr.io'
$ nslookup gcr.io 8.8.8.8
Server: 8.8.8.8
Address 1: 8.8.8.8 google-public-dns-a.google.com
Name: gcr.io
Address 1: 64.233.189.82 tl-in-f82.1e100.net
Address 2: 2404:6800:4008:c07::52 tl-in-x52.1e100.net
kamusis
on 15 Aug 2017
minikube delete -> Install VirtualBox -> minikube start, then everything is fine.
$ cat ~/.minikube/machines/minikube/config.json | grep DriverName
"DriverName": "virtualbox",
$ kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-addon-manager-minikube 1/1 Running 0 3m
kube-system kube-dns-910330662-sp9k0 3/3 Running 0 3m
kube-system kubernetes-dashboard-xhm5x 1/1 Running 0 3m
$ minikube ssh
$ nslookup gcr.io
Server: 10.0.2.3
Address 1: 10.0.2.3
Name: gcr.io
Address 1: 74.125.23.82 tg-in-f82.1e100.net
@kamusis solution worked for me 馃憤
amitoj
on 17 Dec 2017
Related issues
mrtkp9993
路
3Comments
RaeesBhatti
路
3Comments
djschny
路
3Comments
jroucheton
路
3Comments
tstromberg
路
3Comments
Most helpful comment
Same here:
Minikube version: v0.18.0
Environment:
Installed
minikubeanddocker-machine-driver-xhyveyesterday. Startingminikubefor the first time went smoothly. Today, when I wanted to restartminikubeI got:Reverting to
minikube's default VM driver fixes the problem: