minikube dashboard doesn't display in browser

[pswenson]

I installed minikube version: v0.5.0 yesterday.

when I type minikube dashboard it starts a browser tab as expected at http://192.168.99.100:30000/

however, the dashboard just hangs and nothing displays.

any ideas?

thanks

[dlorenc]

Hey,

The dashboard can take a minute to start up. If you run:

kubectl get pods --all-namespaces

you should see a kubernetes-dashboard-abcde pod. The STATUS should be RUNNING.

[pswenson]

kubectl get pods --all-namespaces

NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
default       hello-minikube-3383150820-1ua6r         1/1       Running            0          40m
kube-system   kube-addon-manager-minikubevm           1/1       Running            4          22h
kube-system   kubernetes-dashboard-3717423461-ws7e8   0/1       Error              25         21h
kube-system   kubernetes-dashboard-d9wep              0/1       CrashLoopBackOff   25         22h

[dlorenc]

That looks like the issue. Could you attach the output of

kubectl describe pod kubernetes-dashboard-3717423461-ws7e8 --namespace=kube-system
kubectl describe pod kubernetes-dashboard-d9wep --namespace=kube-system
kubectl logs kubernetes-dashboard-d9wep --namespace=kube-system
kubectl logs kubernetes-dashboard-3717423461-ws7e8 --namespace=kube-system

For some reason the dashboard is crashing.

[dlorenc]

OK, I think I found the issue. The API server generates a secret at startup time, which contains a ca.crt allowing the dashboard to connect to the API server.

If you stop and start the minikube VM, the IP can change. This invalidates the ca.crt, but the API server doesn't generate another one automatically. If you run
'kubectl get secret --namespace=kube-system`

Then take the secret name and pass it into:

kubectl delete secret NAME --namespace=kube-system

the API server will recreate the secret, and you should be all set.

We should figure out a way to keep the cert valid or regenerate it automatically.

@luxas, any ideas?

[pswenson]

Still not working.... I did you suggested and restarted minikube.

now I see this (in out.txt) when running
kubectl describe pod kubernetes-dashboard-3717423461-ws7e8 --namespace=kube-system
out.txt

the interesting part is

5m  1m  3   {kubelet minikubevm}        Warning FailedMount Unable to mount volumes for pod "kubernetes-dashboard-3717423461-ws7e8_kube-system(f71ffdb8-43b5-11e6-92c1-52866b0174ef)": timeout expired waiting for volumes to attach/mount for pod "kubernetes-dashboard-3717423461-ws7e8"/"kube-system". list of unattached/unmounted volumes=[default-token-y885c]
5m  1m  3   {kubelet minikubevm}        Warning FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kubernetes-dashboard-3717423461-ws7e8"/"kube-system". list of unattached/unmounted volumes=[default-token-y885c]

I'm guessing if I get to a clean state I can get this thing going again.

A couple things:

1. I installed minikube yesterday and the dashboard did work.
2. I have rebooted since yesterday

[ghost]

After you delete the secret, you must also delete the pod:
kubectl get po --namespace=kube-system
kubectl delete po NAME --namespace=kube-system

The replication controller will spin up a new pod that will use the new secret.

[ghost]

@dlorenc: For xhyve, you can set it to use the same IP across reboots:
https://github.com/mist64/xhyve#networking

[ghost]

@dlorenc: Although, I encountered the same problem even when my IP never changed. I checked the SANs in the cert and it listed my IP address, per usual.

[pswenson]

thanks, all good now

[dlorenc]

More debugging:

After the first minikube start, the kubernetes-system service account gets created with a secret that has a token.
This token is valid, and the dashboard can communicate with the API server.

After a minikube stop, then a start, the kubernetes-system service account does not get updated. The secret references the same token as before.

Now the token is not valid though, so the dashboard cannot communicate with the API server.

It looks like the tokens are encrypted using the TLS certificate we pass to the API server, which gets regenerated every time the user runs minikube start.

We need to regenerate this in case the IP changes (although if the IP didn't change we could skip the regeneration, but we'd still be in trouble when it does change).

This means we need to somehow update the secret and service account, or figure out a way to keep the token valid.

[dlorenc]

Doing a kubectl edit $SECRET --namespace=kube-system, and deleting the token works. This triggers a regeneration of the token to a valid value, then this propagates automatically to the pods that reference the secret.

We should do this during cluster start.

[dlorenc]

The fix for this should go out in the next release.