Minikube: No dashboard available when using the xhyve driver

[moondev]

When I use the xhyve driver on OSX, the dashboard does not run or launch. The dashboard works fine using the default virtualbox driver.

[pacuna]

Must be an issue with your setup. I just launched a cluster using that driver I didn't have any problem.

[dlorenc]

This might be a duplicate of #288. If you run:

kubectl get pods --all-namespaces

Do you see a dashboard pod? What's the status?

[Toddzio]

I am also having this issue.
It is identical in behavior to #288 , but deleting the token has no effect. The token is re-created, but doesn't get sent to the pod.
I updated to 0.60, and after stopping and starting minikube, I am back to the same problem.

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-addon-manager-minikubevm 1/1 Running 2 2h
kube-system kubernetes-dashboard-98ody 0/1 CrashLoopBackOff 30 2h
key-tk-mbp% kubectl logs kubernetes-dashboard-98ody --namespace=kube-system
Starting HTTP server on port 9090
Creating API server client for https://10.0.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: the server has asked for the client to provide credentials
key-tk-mbp%

[moondev]

Here is my output of kubectl get pods --all-namespaces

NAMESPACE     NAME                             READY     STATUS              RESTARTS   AGE
kube-system   kube-addon-manager-boot2docker   0/1       ContainerCreating   0          8m

[dlorenc]

This looks like two separate issues. @Toddzio , have you tried deleting, then starting again with 0.6.0?

@moondev, what shows up when you run:

kubectl describe pod kube-addon-manager-boot2docker --namespace=kube-system

[moondev]

Thanks @dlorenc for helping investigate.
describe pod kube-addon-manager-boot2docker --namespace=kube-system

Name:       kube-addon-manager-boot2docker
Namespace:  kube-system
Node:       boot2docker/192.168.64.9
Start Time: Wed, 13 Jul 2016 18:12:14 -0600
Labels:     <none>
Status:     Pending
IP:     192.168.64.9
Controllers:    <none>
Containers:
  kube-addon-manager:
    Container ID:
    Image:      gcr.io/google-containers/kube-addon-manager-amd64:v2
    Image ID:
    Port:
    Requests:
      cpu:          5m
      memory:           50Mi
    State:          Waiting
      Reason:           ContainerCreating
    Ready:          False
    Restart Count:      0
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  addons:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/kubernetes/
QoS Tier:   Burstable
No events.

Is there anyway to "reset" minikube other than minikube delete?

[dlorenc]

Re-running minikube start will reset most things, but won't stop/delete the VM. minikube stop will stop the VM but not clear state. delete deletes the VM and clears all state.

It looks like the VM is having trouble pulling the docker image. If you run: minikube ssh, are you able to contact the internet? Something like curl google.com should be sufficient to tell.

[moondev]

Yep that must be it:

docker@boot2docker:~$ ping google.com
ping: bad address 'google.com'
docker@boot2docker:~$ curl google.com
curl: (6) Could not resolve host: google.com
docker@boot2docker:~$

I can access the internet fine on the host. For what it's worth I also have the docker for mac beta which uses xhyve as well I believe. Networking is fine in there too.

[dlorenc]

Actually @Toddzio, I read your post more closely. If you delete the secret using kubectl, a new one will get created with a new name. That secret will be valid, but it won't make it into any pods that reference the old secret. If you delete the dashboard pod a new one will get created that does have the right secret.

To do it automatically, you have to update teh existing secret and delete the token entry.

[dlorenc]

@moondev, does the docker-machine-xhyve-driver work for you in docker-machine? https://github.com/zchee/docker-machine-driver-xhyve

or does that not have internet access either?

[dlorenc]

@moondev are you using a proxy or VPN by any chance?

[moondev]

No proxy or vpn. As a test I uninstalled docker for mac beta, and installed
docker toolkit. I then tested docker machine via xhyve and still no
connection. I'm going to try and reinstall all the components next, do you
run the docker for mac beta on your machine?

On Jul 19, 2016 4:15 PM, "dlorenc" [email protected] wrote:

@moondev https://github.com/moondev are you using a proxy or VPN by any
chance?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/minikube/issues/328#issuecomment-233783242,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdw0jHsZbZciCdKaYHnXkJzyDctETDRks5qXUyOgaJpZM4JLZ4U
.

[dlorenc]

Sorry for the delay. I do use Docker for Mac alongside minikube, and it works fine for me. Not sure what could be causing the xhyve networking to not work. We should figure this out before making it default. #270

[numbsafari]

@moondev - I'm seeing the same thing. Debugging it now. How are you installing the xhyve driver, etc? Do you know what version you are using?

[numbsafari]

In my case, at least, seems to be DNS-related:

docker@boot2docker:/etc$ nslookup google.com 8.8.8.8
Server:    8.8.8.8
Address 1: 8.8.8.8 google-public-dns-a.google.com

Name:      google.com
Address 1: 216.58.217.78 iad23s41-in-f78.1e100.net
Address 2: 2607:f8b0:4004:80b::200e iad23s41-in-x0e.1e100.net
docker@boot2docker:/etc$ curl google.com -4 216.58.217.78
curl: (6) Could not resolve host: google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

[numbsafari]

Not sure if this helps:

21:49 $ kubectl get pods --namespace=kube-system
NAME                             READY     STATUS              RESTARTS   AGE
kube-addon-manager-boot2docker   0/1       ContainerCreating   0          3m

and

21:46 $ minikube dashboard --logtostderr=True --show-libmachine-logs=True
W0830 21:46:55.408807    5938 root.go:93] Error reading config file at /Users/swilson/.minikube/config: Unsupported Config Type ""
I0830 21:46:55.408908    5938 notify.go:102] Checking for updates...
Could not find finalized endpoint being pointed to by %s: %s kubernetes-dashboard endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found
endpoints "kubernetes-dashboard" not found

[numbsafari]

FWIW ... I was unable to create a docker-machine with the xhyve driver without it having the same issue. I added virtualbox to my machine and it is working fine.

Going to finish what I'm doing and come back to this. It _was_ working for me previously.

[numbsafari]

Circled back to do some more testing on this, I believe this is related to my use of dnsmasq on my laptop. When I disable dnsmasq, the problem goes away. Going to tinker with it a bit more to come up with a work around, but I believe I just need to create a separate loopback interface for dnsmasq to listen on, other than 127.0.0.1. It's just weird because I'm only using dnsmasq for .test domains, so I'm not sure why this would interfere with normal operation.

[dlorenc]

Thanks for following up! I'll close this out for now, let me know if you run into any more trouble.

[mssio]

I think this problem has not been resolved yet.
When I am installing http://pow.cx on my Mac, the dns resolution on minikube (using xhyve driver) is not working. But when I am uninstalling pow, it will works.

[mezis]

Bumping this thread as I had an identical issue: no dashboard, containers no set up, images wouldn't download, confirmed by lack of DNS resolution inside the VM:

$ minikube ssh
# curl google.com
Could not resolve host: google.com

Resolution was, for me, to disable the tools that run interference in terms of VM networking:

• Shut down Docker for Mac (and tell it not to auto start at login)
• Idem for Virtualbox
• Remove the Virtualbox launch agent at /Library/LaunchDaemons/org.virtualbox.startup.plist
• Reboot

Both Virtualbox _and_ Docker for Mac had to be off for minikube to start properly.

[johnraz]

@numbsafari did you find a workaround or the reason dnsmasq running on the host is getting in the way?

[shinzui]

Why is this issue closed? It would be nice to at least have a workaround of running minikube without shutting down dnsmasq.

[gautamkpai]

Looks like dnsmasq was refusing connections on the Host IP address(in this case 192.168.64.1) assigned by minikube.

➜  ~ minikube ssh
$ docker search alpine
Error response from daemon: Get https://index.docker.io/v1/search?q=alpine&n=25: dial tcp: 
lookup index.docker.io on 192.168.64.1:53: read udp 192.168.64.30:33718->192.168.64.1:53: 
read: connection refused

Configuring dnsmasq to specifically listen on the minikube host IP address did the trick for me. Add the following 2 lines in your dnsmasq.conf.

➜  ~ vim /usr/local/etc/dnsmasq.conf
# If you want dnsmasq to listen for DHCP and DNS requests on
#  by address (remember to include 127.0.0.1 if you use this.)
# Repeat the line for more than one interface.
#listen-address=
listen-address=192.168.64.1
listen-address=127.0.0.1

You will need to restart your macbook after updating dnsmasq config.
(Strangely enough, only restarting minikube VM and dnsmasq did not work.)