Dashboard: Not able to access dashboard from external browser

[saikatgithub]

Steps:

1. Installed kubernetes using kubeadm (1.6) from the following - "https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/"

2. Installed successfully in master and other nodes (all are Ubuntu 16 VM in local cluster)

3. Started the kubectl proxy using the following way in Master to access API server externally -
   kubectl proxy --address --port=9999 --accept-hosts='^*$'

4. Installed Dashboard in the master node with ARG as "--apiserver-host=http://:9999" and type as "NodePort" so that it can be accessed externally

administrator@Kube1:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-kube1 1/1 Running 1 3d
kube-system kube-apiserver-kube1 1/1 Running 1 3d
kube-system kube-controller-manager-kube1 1/1 Running 1 3d
kube-system kube-dns-3913472980-mp0tz 3/3 Running 3 3d
kube-system kube-flannel-ds-0mk7x 2/2 Running 1 3d
kube-system kube-proxy-2pg76 1/1 Running 0 3d
kube-system kube-scheduler-kube1 1/1 Running 1 3d
kube-system kubernetes-dashboard-2680428461-8rjnt 1/1 Running 0 3d
.....

administrator@Kube1:~$ kubectl describe services kubernetes-dashboard --namespace=kube-system
Name: kubernetes-dashboard
Namespace: kube-system
Labels: k8s-app=kubernetes-dashboard
Annotations:
Selector: k8s-app=kubernetes-dashboard
Type: NodePort
IP: 10.101.228.92
Port: 80/TCP
NodePort: 31706/TCP
Endpoints: 10.244.0.7:9090
Session Affinity: None

administrator@Kube1:~$ kubectl --namespace=kube-system logs kubernetes-dashboard-2680428461-8rjnt
Using HTTP port: 8443
Using apiserver-host location: http://:9999
Creating API server client for http://:9999
Successful initial request to the apiserver, version: v1.6.3
Creating in-cluster Heapster client
Using service account token for csrf signing

Status:

1. Able to browse almost all the APIs of Server from external browser outside the cluster as follows -
   Chrome ==> http://:9999/version
   {
   "major": "1",
   "minor": "6",
   "gitVersion": "v1.6.3",
   "gitCommit": "0480917b552be33e2dba47386e51decb1a211df6",
   "gitTreeState": "clean",
   "buildDate": "2017-05-10T15:38:08Z",
   "goVersion": "go1.7.5",
   "compiler": "gc",
   "platform": "linux/amd64"
   }

2. BUT, NOT able to access dashboard ui from external browser -
   Chrome ==> http://:9999/ui
   Chrome ==> http://:31706
   Chrome ==> http://:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/
   All the above requests are hanging for long and later failed with connection timed out. But I am not getting any "unauthorized" error

3. Curl to dashboard ui is passing from the master node (ONLY), but, failing from other nodes inside kube cluster ["curl -v http://:31706"]. So looks like everything (including dashboard) is working fine except accessing dashboard from outside the master node

Problem:

Not able to access dashboard externally (Fine with insecure / non-safe way as of now)

Looking for urgent help to resolve this. Thanks in advance!

[floreks]

Did you try checking apiserver logs for unusual messages? It looks like some kind of networking issue to me. Could you also try using in-cluster config and see if issue still occurs?

I have just checked similar setup (kubeadm + kubectl proxy) and I was able to access dashboard from my phone that is in the same network as my --address <interface> passed to kubectl proxy. Only difference is that I am using in-cluster config not --apiserver-host argument.

[saikatgithub]

Thanks floreks!

Changes:

I have deleted and created the dashboard again without having '--apiserver-host' as ARG. Dashboard started fine connecting API server in a different IP. No other change has been done

administrator@Kube1:~$ kubectl --namespace=kube-system log kubernetes-dashboard-3750890696-bdlng
Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Successful initial request to the apiserver, version: v1.6.3
Creating in-cluster Heapster client
Using service account token for csrf signing

administrator@IoTKube1:~$ kubectl describe services kubernetes-dashboard --namespace=kube-system
Name: kubernetes-dashboard
Namespace: kube-system
Labels: k8s-app=kubernetes-dashboard
Annotations:
Selector: k8s-app=kubernetes-dashboard
Type: NodePort
IP: 10.111.158.210
Port: 80/TCP
NodePort: 32521/TCP
Endpoints: 10.244.0.8:9090
Session Affinity: None
Events:

Status:

1. Cluster state looks ok except I have some errors in API Server logs repetitively coming every 3-5 min or so from long. Not sure what this error means and if this has any impact on dashboard exposing outside

administrator@IoTKube1:~$ kubectl --namespace=kube-system log kube-apiserver-iotkube1
....
E0620 14:31:14.123435 1 watcher.go:188] watch chan error: etcdserver: mvcc: required revision has been compacted
I0620 14:32:12.087201 1 compact.go:159] etcd: compacted rev (4334422), endpoints ([http://127.0.0.1:2379])
I0620 14:37:12.118791 1 compact.go:159] etcd: compacted rev (4334844), endpoints ([http://127.0.0.1:2379])
E0620 14:40:12.181587 1 watcher.go:188] watch chan error: etcdserver: mvcc: required revision has been compacted
....

1. Status remains same on the access part. Not able to access dashboard ui from external browser. However able to access almost all the APIs of Server from the same external browser.
   Chrome ==> http://:9999/version/ >>>>> Passing
   Chrome ==> http://:9999/ui/ >>>>> Failing
   Chrome ==> http://:32521 >>>>> Failing
   Chrome ==> http://:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/ >>>>> Failing

2. One thing I have forgot to mention that even from MASTER node, I am NOT able to access dashboard ui using "curl -v http://:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/". Call returning "HTTP/1.1 503 Service Unavailable" error after long time. However from MASTER node I am getting success on these curl calls -
   http://:9999/version/
   http://:32521

[floreks]

One thing I have forgot to mention that even from MASTER node, I am NOT able to access dashboard ui using "curl -v http://:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/". Call returning "HTTP/1.1 503 Service Unavailable" error after long time. However from MASTER node I am getting success on these curl calls -
http://:9999/version/
http://:32521

This definitely looks like some kind of networking issue to me. There seem to be 2 issues actually.

1. When accessing dashboard using service proxy kubernetes fails to resolve correct address. That's why you are getting Service Unavailable. Request times out somewhere on the way (maybe it's kube-proxy fault).
2. If MASTER_EXTERNAL_IP is indeed external then it shouldn't matter if you are accessing application using NodePort from the same machine or from external one as long as you are accessing the node on which application is deployed. Sometimes VMs have 2 network bridges set. 1 for internal communication between VMs and second one for outside. There are many potential issues here so you have to investigate that further.

[saikatgithub]

"If MASTER_EXTERNAL_IP is indeed external ..."

>
MASTER_EXTERNAL_IP is the node IP that is accessible from outside. Do I need to do anything to make it external? I am able access APIs of the server from the external browser like -
External Chrome ==> http://:9999/version/ >>>>> Passing

Now my question is that -

• why the below call is failing from the same external browser -
  From external Chrome ==> http://:32521 >>>>> Failing
• while curl call on the same URI is passing from master node -
  From Master node ==> curl -v http://:32521 >>>>> Passing

Is there any firewall rule that is preventing from accessing dashboard from outside (not going through kube proxy)? Did I miss any configuration step?

Also is this anything to do with the below error?
administrator@IoTKube1:~$ kubectl --namespace=kube-system log kube-apiserver-iotkube1
....
E0620 14:31:14.123435 1 watcher.go:188] watch chan error: etcdserver: mvcc: required revision has been compacted
I0620 14:32:12.087201 1 compact.go:159] etcd: compacted rev (4334422), endpoints ([http://127.0.0.1:2379])
I0620 14:37:12.118791 1 compact.go:159] etcd: compacted rev (4334844), endpoints ([http://127.0.0.1:2379])
E0620 14:40:12.181587 1 watcher.go:188] watch chan error: etcdserver: mvcc: required revision has been compacted
....

[floreks]

Is there any firewall rule that is preventing from accessing dashboard from outside (not going through kube proxy)? Did I miss any configuration step?

We are not blocking any traffic or enforcing any rules. Dashboard is just an application. if kube-proxy fails to create some firewall rules then it is not our fault but rather system or kubernetes.

Also is this anything to do with the below error?
...

No. This has nothing to do with dashboard. Quite common etcd messages.

If you can access dashboard on NodePort from the node and not from the outside then it has to be networking issue (possibly firewall or routing).

[saikatgithub]

Thanks floreks for quick help!

2 small questions -

1. MASTER_EXTERNAL_IP is the node IP that is accessible from outside. Do I need to do anything extra to make it an external IP for that node?
2. Which component should I raise the issue instead of dashboard?

[floreks]

1. I can't really answer that without knowing all the details. It depends on the configuration. It might be required.
2. As this looks like networking issue I'd recommend trying on kubernetes repository directly.

[cheld]

I assume that you are using a virtual machine in the cloud. You must certainly do some manual steps in your cloud provider
to expose a Nodeport manually. Typically, in the cloud a LoadBalancer is used to export nodeports to the outside.
(But you should not expose Dashboard with Nodeport unless you have secured access yourself, somehow)

Use kubectl proxy to access dashboard.

I don't think the problem is related to dashboard or kubernetes

[saikatgithub]

It is bare metal, not cloud.

[cheld]

I wonder what you mean with master _EXTERNAL_ IP. On bare metal you have only a single IP, right? Or do you have more complicated network setup?

Chrome ==> http://<MASTER_EXTERNAL_IP>:9999/ui/ >>>>> Failing

this will only work if the apiserver is configured to accept username and password. Does

http://<MASTER_EXTERNAL_IP>:9999/api
work?

The URL below is obviously wrong, because of the .../proxy/ suffix does not exist. Please check and use the correct url.

Chrome ==> http://<MASTER_EXTERNAL_IP>:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/ >>>>> Failing

[floreks]

http://127.0.0.1:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/workload?namespace=default
It is correct. This is the url to which you are redirected after using http://<MASTER_IP>:<MASTER_PORT>/ui

http://<MASTER_EXTERNAL_IP>:9999/ui
This should also work if you use kubectl proxy --port 9999 --interface <EXTERNAL_INTERFACE> --accept-hosts="^*$" to proxy the requests and expose apiserver.

[cheld]

interesting... in 1.5.x the url is /api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#

[floreks]

AFAIR it was changed for security reasons to use proxy subresource after service.
https://github.com/kubernetes/kubernetes/commit/88a876b1d0d4d1e71d51c2e540bd3850caffea4b#diff-9d7366044517a2a0407ea473a1fa366a

[saikatgithub]

When I query with "http://:9999/ui" URI, then chrome is showing the redirection to "http://:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy". However it times out.

Ubuntu 16 VMs constitute the nodes of my local cluster. MASTER_EXTERNAL_IP is the node IP of the Kubernetes master node and this IP accessible from outside. Not sure if I need to do any extra configuration to make it an external IP for that node

[nerusnayleinad]

There is a hack I discovered this morning. You can create a user called "system:anonymous". The con is that it needs to have cluster-admin role. Any other doesn't work. So you would connect to your cluster and do...

$ kubectl create clusterrolebinding NAME --clusterrole=cluster-admin --user=system:anonymous

Now go to https://master_ip/ui

Cheers

[pgnaleen]

if you want to just access dashboard without any authentication just edit your dashboard yaml file like below. find
kind: Service
and change like this

kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
ports:

• port: 80
  targetPort: 9090
  selector:
  k8s-app: kubernetes-dashboard
  externalIPs:
• enter the ip address of your master
  type: LoadBalancer

you will be able to access dashboard just using your master ip if you didn't used
masterip:80 port for any other pod

[vglisin]

Problem:
Environment is setup as two virtual machines (one master and one node).

I was using kubedadm to set up this... thing. Master node and one minion node looks ok:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
masternode Ready master 4h v1.8.0
minionone Ready 2h v1.8.0
kubernetes dashboard installed by the book:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

Problem is:
I am not able to get dashboard ui. BTW: where to start kubectl proxy? On my machine, master node or node? This is totally dark for newbee by reading what you have written (documentation is still a problematic one). So, any help to make dashboard working from my machine which is not master or minion in kubernetes in a simple user/pass way? I don't give mind about possible authentication perversion and security issues, just is it a simple way to make this working in a test environment?
Hopefully somebody will answer.

[cheld]

1. Run 'kubectl get nodes' to make sure kubectl is configured properly
2. Run kubectl proxy on your local workstation
3. Open browser and type URL: http://localhost:xxxxxxxxx as documented

With these steps you should be able to see at least the dashboard login screen.

[pgnaleen]

@vglisin why you gave me negative reply.

[cheld]

@pgnaleen from your description it seems you have given full control of the cluster to the public internet. Maybe you have some additional firewall rule, but you are hopefully aware of it

[jfmwz]

I can access using this
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/deploy/app?namespace=default

[floreks]

Closing as stale. Reopen if issue still occurs.

[chongyaowang]

@jfmwz I have similar issue as you described that my browser can't access dashboard. how did you solve the problem?

[saikatgithub]

For quick turn around (not advisable for production), you can follow the below steps -

1. Use "Alternative setup" to install dashboard for quick but non-secure way -

2. kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/alternative/kubernetes-dashboard.yaml

3. Then configure 'NodePort' way of accessing it. Change 'type: ClusterIP' to 'type: NodePort' and save -

4. kubectl -n kube-system edit service kubernetes-dashboard

5. Find out the node and exposed port where dashboard is running -

6. kubectl -n kube-system describe pod kubernetes-dashboard-*

7. kubectl -n kube-system get service kubernetes-dashboard

8. Grant cluster-admin permissions to Dashboard -

9. kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

10. Access Dashboard through the link - "http://:" that you got as part of step 3

Enjoy!

[yosiasz]

This worked for me. To find your master ip address run the following
kubectl config view
you will see the master.ip.address in following entry
server: https://master.ip.address:6443
Then run the following
kubectl proxy --port=9999 --address='master.ip.address' --accept-hosts="^*$"
Go to your browser and put following
http://master.ip.address:9999/ui

[kamarajuPrathi]

@yosiasz if you master doesnt have a public ip , how will you access it

[natiki]

@kamarajuPrathi This will vary based on your installation. But generally you will need to make a change to your firewall to allow external.ip --> internal.ip

[prashantcfc]

@yosiasz
Thanks a bunch mate, I bootstrapped a kubernetes cluster with kubeadm and finally your solution worked for me. Just that in order to access the UI, one needs to point to the following :

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/.

[saddique164]

Problem:
Environment is setup as two virtual machines (one master and one node).

I was using kubedadm to set up this... thing. Master node and one minion node looks ok:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
masternode Ready master 4h v1.8.0
minionone Ready 2h v1.8.0
kubernetes dashboard installed by the book:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

Problem is:
I am not able to get dashboard ui. BTW: where to start kubectl proxy? On my machine, master node or node? This is totally dark for newbee by reading what you have written (documentation is still a problematic one). So, any help to make dashboard working from my machine which is not master or minion in kubernetes in a simple user/pass way? I don't give mind about possible authentication perversion and security issues, just is it a simple way to make this working in a test environment?
Hopefully somebody will answer.

Please remove slave nodes while configuring it. keep only master active in the cluster. It will resolve you issues

[saddique164]

http://master.ip.address:9999/ui
This is my browser shows when I open your shared URL using the IP. Note I followed your steps...
{
"paths": [
"/apis",
"/apis/",
"/apis/apiextensions.k8s.io",
"/apis/apiextensions.k8s.io/v1beta1",
"/healthz",
"/healthz/etcd",
"/healthz/log",
"/healthz/ping",
"/healthz/poststarthook/crd-informer-synced",
"/healthz/poststarthook/generic-apiserver-start-informers",
"/healthz/poststarthook/start-apiextensions-controllers",
"/healthz/poststarthook/start-apiextensions-informers",
"/metrics",
"/openapi/v2",
"/version"
]
}

[saddique164]

The problem has been resolved with your idea. I just changed the URL to this..

http://My_Master_Ip :9999/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

[amitkumarjain4u2u]

I had installed dashboard by following

Below link:

https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html

till step kubectl proxy i am getting results as expected but below URL is not accessible:

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#!/login

I have done all these steps on master.
my machine is an AWS machine.
Can anyone suggest what needs to be done for accessing above URL?

[Wowgreat]

I have solved by follow command
sudo kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'

[srivasthava]

Kubernetes dashboard can't accessed across the cluster, Think you have 3 master and 3 worker as cluster and one 1 machine for admin user where you setup to execute the kubectl commands, you can use run kube proxy and assign the private IP and port to run ,

kubectl proxy --address=**10.x.x.x** --port 8001 --accept-hosts=".*"

Prerequisites: Dashboard-service exposed as Nodeport, if any master or any node is Load balanced, make sure your inbound rule accepting the port 8001 is opened, Kubectl config view should show your admin user

Example :

If you're using host machine (admin user) in Cloud(Azure/AWS/Google) VM which has Private IP (10.1.0.4) and Public-IP or External IP, since kubectl-proxy listening only on localhost:8001,try run the command below

1) kubectl proxy --address=**10.1.0.4** --port 8001 --accept-hosts=".*"

2) Access the URL using the public ip from any laptop browser (outside of your cluster):
http://_public-IP_ :8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Note:
If you have 3 Nodes serving one External IP, you have to run the kubectl proxy in all nodes