Dashboard: can not login to dashboard again

[HaveF]

Environment

Dashboard version: 1.7.1
Kubernetes version: 1.8.2
Operating system: debian 9.1
Node.js version:
Go version:

Steps to reproduce

At first, dashboard work, but when I what to try to give dashboard privileges, I find I can not login again. Even I delete all dashboard related things and reinstall:

1. delete dashboard related things

kubectl -n kube-system get secret | grep kubernetes-dashboard | awk '{print $1}'|  xargs kubectl -n kube-system delete secrets  &&
kubectl -n kube-system get serviceaccounts | grep kubernetes-dashboard | awk '{print $1}' | xargs kubectl -n kube-system delete serviceaccounts &&
kubectl -n kube-system get roles.rbac.authorization.k8s.io | grep kubernetes-dashboard | awk '{print $1}' | xargs kubectl -n kube-system delete roles.rbac.authorization.k8s.io &&
kubectl -n kube-system get rolebindings.rbac.authorization.k8s.io | grep kubernetes-dashboard | awk '{print $1}' | xargs kubectl -n kube-system delete rolebindings.rbac.authorization.k8s.io &&
kubectl -n kube-system get clusterrolebindings.rbac.authorization.k8s.io | grep kubernetes-dashboard | awk '{print $1}' | xargs kubectl -n kube-system delete clusterrolebindings.rbac.authorization.k8s.io &&
kubectl -n kube-system get pods | grep kubernetes-dashboard  | awk '{print $1}' | xargs kubectl -n kube-system delete pods &&
kubectl -n kube-system get svc | grep kubernetes-dashboard  | awk '{print $1}'  | xargs kubectl -n kube-system delete svc &&
kubectl -n kube-system get deployments.extensions | grep kubernetes-dashboard  | awk '{print $1}'  | xargs kubectl -n kube-system delete deployments.extensions 

1. reinstall
   kubectl create -f ~/k8s/kubernetes-dashboard.yaml, not error

2. use kubectl proxy --address="192.168.5.58" -p 8001 --accept-hosts='^*$' and view the web, login page appear,

3. get the token, but when paste the token to login page, can click sign in, nothing appear.
   kubectl -n kube-system get secret | grep kubernetes-dashboard-token | awk '{print "secret/"$1}' | xargs kubectl describe -n kube-system

4. view the pod, nothing strange

$ kubectl -n kube-system get pods | grep kubernetes-dashboard | awk '{print $1}' | xargs kubectl describe pods -n kube-system
Name:           kubernetes-dashboard-747c4f7cf-72m79
Namespace:      kube-system
Node:           k8s-192.168.5.79/192.168.5.79
Start Time:     Thu, 26 Oct 2017 15:10:10 +0800
Labels:         k8s-app=kubernetes-dashboard
                pod-template-hash=303709379
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kubernetes-dashboard-747c4f7cf","uid":"afadb27f-ba1d-11e7-80...
Status:         Running
IP:             172.16.228.5
Created By:     ReplicaSet/kubernetes-dashboard-747c4f7cf
Controlled By:  ReplicaSet/kubernetes-dashboard-747c4f7cf
Init Containers:
  kubernetes-dashboard-init:
    Container ID:   docker://a89f493cdeb3a04d939dfd442643493ed35ccf77d3d6d171dab9982e4d08eb2d
    Image:          gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.0.1
    Image ID:       docker-pullable://gcr.io/google_containers/kubernetes-dashboard-init-amd64@sha256:7e0356253d780007a39820b24676676dd40902cc1ee02ba28d77697149b090a8
    Port:           <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 26 Oct 2017 15:10:11 +0800
      Finished:     Thu, 26 Oct 2017 15:10:12 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-pxf9b (ro)
Containers:
  kubernetes-dashboard:
    Container ID:  docker://7b1a01c34f27b52eaa90e692072a2369f712ad2452cfa61bddc937a66744b753
    Image:         gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.1
    Image ID:      docker-pullable://gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:327cfef378e88ffbc327f98dd24adacf6c9363c042db78e922d050f2bdcf6f78
    Port:          8443/TCP
    Args:
      --tls-key-file=/certs/dashboard.key
      --tls-cert-file=/certs/dashboard.crt
    State:          Running
      Started:      Thu, 26 Oct 2017 15:10:13 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (ro)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-pxf9b (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  kubernetes-dashboard-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-certs
    Optional:    false
  tmp-volume:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  kubernetes-dashboard-token-pxf9b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-token-pxf9b
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age   From                       Message
  ----     ------                 ----  ----                       -------
  Normal   SuccessfulMountVolume  18m   kubelet, k8s-192.168.5.79  MountVolume.SetUp succeeded for volume "tmp-volume"
  Normal   SuccessfulMountVolume  18m   kubelet, k8s-192.168.5.79  MountVolume.SetUp succeeded for volume "kubernetes-dashboard-certs"
  Normal   SuccessfulMountVolume  18m   kubelet, k8s-192.168.5.79  MountVolume.SetUp succeeded for volume "kubernetes-dashboard-token-pxf9b"
  Normal   Pulled                 18m   kubelet, k8s-192.168.5.79  Container image "gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.0.1" already present on machine
  Normal   Created                18m   kubelet, k8s-192.168.5.79  Created container
  Normal   Started                18m   kubelet, k8s-192.168.5.79  Started container
  Warning  FailedMount            18m   kubelet, k8s-192.168.5.79  MountVolume.SetUp failed for volume "kubernetes-dashboard-certs" : secrets "kubernetes-dashboard-certs" not found
  Normal   Pulled                 18m   kubelet, k8s-192.168.5.79  Container image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.1" already present on machine
  Normal   Created                18m   kubelet, k8s-192.168.5.79  Created container
  Normal   Started                18m   kubelet, k8s-192.168.5.79  Started container
  Normal   Scheduled              11m   default-scheduler          Successfully assigned kubernetes-dashboard-747c4f7cf-72m79 to k8s-192.168.5.79

1. view the log, nothing strange,

$ kubectl -n kube-system get pods | grep kubernetes-dashboard | awk '{print $1}' | xargs kubectl -n kube-system logs
2017/10/26 07:10:13 Starting overwatch
2017/10/26 07:10:13 Using in-cluster config to connect to apiserver
2017/10/26 07:10:13 Using service account token for csrf signing
2017/10/26 07:10:13 No request provided. Skipping authorization
2017/10/26 07:10:13 Successful initial request to the apiserver, version: v1.8.2
2017/10/26 07:10:13 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2017/10/26 07:10:13 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2017/10/26 07:10:13 Initializing secret synchronizer synchronously using secret kubernetes-dashboard-key-holder from namespace kube-system
2017/10/26 07:10:13 Initializing JWE encryption key from synchronized object
2017/10/26 07:10:13 Creating in-cluster Heapster client
2017/10/26 07:10:13 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/26 07:10:13 Serving securely on HTTPS port: 8443
2017/10/26 07:10:43 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/26 07:11:13 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/26 07:11:27 Getting application global configuration
2017/10/26 07:11:27 Application configuration {"serverTime":1509001887739}
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Incoming HTTP/2.0 GET /api/v1/thirdpartyresource request from 172.16.162.192:34768: {}
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 172.16.162.192:34768: {}
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Incoming HTTP/2.0 GET /api/v1/rbac/status request from 172.16.162.192:34768: {}
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Outcoming response to 172.16.162.192:34768 with 200 status code
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Incoming HTTP/2.0 GET /api/v1/login/modes request from 172.16.162.192:34768: {}
2017/10/26 07:11:28 Getting list of third party resources
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Outcoming response to 172.16.162.192:34768 with 200 status code
2017/10/26 07:11:28 Non-critical error occurred during resource retrieval: thirdpartyresources.extensions is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list thirdpartyresources.extensions at the cluster scope
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Outcoming response to 172.16.162.192:34768 with 200 status code
2017/10/26 07:11:28 [2017-10-26T07:11:28Z] Outcoming response to 172.16.162.192:34768 with 200 status code
2017/10/26 07:11:31 [2017-10-26T07:11:31Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/login request from 172.16.162.192:34768: {}
2017/10/26 07:11:31 [2017-10-26T07:11:31Z] Outcoming response to 172.16.162.192:34768 with 200 status code
2017/10/26 07:11:31 [2017-10-26T07:11:31Z] Incoming HTTP/2.0 POST /api/v1/login request from 172.16.162.192:34768: {
  "kubeConfig": "",
  "password": "",
  "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1weGY5YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImFmYTg2MzI3LWJhMWQtMTFlNy04MDdlLTAwNTA1NmI3NzJkZCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.DoS9a3X03rqFnCpgTHgi7cPacoWd_hUmRuNdPRHiy4Q0OIIoTVLMGKDI7Zsw3m-Okvj0jMyZ9gKcn6J_wdNiVkXKxftDnO0Bho22aY4cCTwfjS52wkFM8FuOlzCz_ZxQQBvSil2k-GTrKHcPNg2n08pFAb9nhXUlZ98otiC8lYbWTma2SycZ6vsUv9S20zVwI81112msOQoaENq3KXOuxjHKVjF7mV8GHBpbZEFTOJ5gm69Jst2QbLPaVOXG6lYUUgK1cwd8kEJXHf1jFyQeHsdgzbNFq1LdOp9q_HPs4j8RsTBQa6qaHQB3A_xJiANvUMG4jvgz_UEdgenSU-UMVA",
  "username": ""
}
2017/10/26 07:11:31 [2017-10-26T07:11:31Z] Outcoming response to 172.16.162.192:34768 with 200 status code
2017/10/26 07:11:31 [2017-10-26T07:11:31Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 172.16.162.192:34768: {}
2017/10/26 07:11:31 [2017-10-26T07:11:31Z] Outcoming response to 172.16.162.192:34768 with 200 status code
2017/10/26 07:11:43 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/26 07:12:13 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/26 07:12:43 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/26 07:13:13 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/26 07:13:43 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.

what can I do now?

Observed result

Expected result

Comments

[floreks]

Did you read accessing Dashboard guide on our wiki? In your case sign in won't work and it is explained why.

Works as intended.

[HaveF]

@floreks Hi, thanks for your replay. I can login before, but after reinstall can not login. I do suppose I use bearer-token.

Have any suggestion? thanks!

[zhengyiqun]

I also encountered the same problem.
version：v1.8.5 + coreos.0
dashboard：1.8.1

[cauwulixuan]

have you solve this problem? I meet the same issue.... same log.
k8s version: 1.8.5
dashboard: 1.8.0

[floreks]

https://github.com/kubernetes/dashboard/wiki/Accessing-Dashboard---1.7.X-and-above

NOTE: Dashboard should not be exposed publicly using kubectl proxy command as it only allows HTTP connection. For domains other than localhost and 127.0.0.1 it will not be possible to sign in. Nothing will happen after clicking Sign in button on login page.

[Fireworks643]

How dashboard should be published then ? (when i changing ClusterIP option to NodePort my dash is not accessable at all ! Why is kubectl proxy existing in the first place ? Why it is so hard to start with kubernetes, you got luck of examples in configuration.

[ekhaydarov]

@floreks it would be great if you could list some possible solutions as to how to gain access to dashboard. essentially it renders the dashboard pointless if it can only be accessed from localhost

[fathepurm]

https://github.com/kubernetes/dashboard/wiki/Accessing-Dashboard---1.7.X-and-above

NOTE: Dashboard should not be exposed publicly using kubectl proxy command as it only allows HTTP connection. For domains other than localhost and 127.0.0.1 it will not be possible to sign in. Nothing will happen after clicking Sign in button on login page.

What are the steps to access externally using NodePort. Because i have the same issue, when i enter the token and click on sign in nothing happens? Please can you provide clear working steps if you know. Because i see so many solutions but nothing seems working.

[floreks]

If you don't care about security then there is enable-insecure-login argument. https://github.com/kubernetes/dashboard/blob/master/docs/common/dashboard-arguments.md

If you do care, then use some kind of reverse proxy to secure the traffic between user and Dashboard. There are many ways how to expose it. It's Kubernetes that offers ways to do this. Dashboard is just an app.