Dashboard: Unable to access dashboard
Issue details
Unable to access dashboard on http://master_ip/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
Message "no endpoints available for service "kubernetes-dashboard""
I did the steps given in http://kubernetes.io/docs/user-guide/ui-access/, but still no result.
kubectl create -f cluster/addons/dashboard/dashboard-controller.yaml --namespace=kube-system
kubectl create -f cluster/addons/dashboard/dashboard-service.yaml --namespace=kube-system
#kubectl version
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
When using v0.19.3, I was able to access the dashboard.
Dashboard version: v1.0.1
Kubernetes version: v1.2.4
Operating system: Gnu-linux/Ubuntu
Node.js version: -
Go version: -
Observed result
Unable to access UI
Rahul91
All 71 comments
Is the Dashboard UI running? Can you show me logs of the dashboard pod?
bryk
on 20 Jun 2016
I got the same error.
Starting HTTP server on port 9090
Creating API server client for https://10.0.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.0.0.1:443/version: dial tcp 10.0.0.1:443: i/o timeout
Name: kubernetes-dashboard
Namespace: kube-system
Labels: k8s-app=kubernetes-dashboard,kubernetes.io/cluster-service=true Selector: k8s-app=kubernetes-dashboard
Type:
ClusterIP IP: 10.0.0.55
Port:
Endpoints: 172.17.0.5:9090
Session Affinity: None
No events.
$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1
arhided
on 23 Jun 2016
@bryk Gist: https://gist.github.com/Rahul91/f443e58dd730e0571bcea6409adb5761
I am getting this error in http://master_ip/ui
Error: 'dial tcp 10.100.22.2:9090: i/o timeout'
Trying to reach: 'http://10.100.22.2:9090/'
I am running my master on a server with publicly accessible IP and minion on my local machine running in a local network. Is that the reason I am getting this error?
Rahul91
on 23 Jun 2016
@Rahul91 As your master is located on external server try to provide apiserver-host parameter to the dashboard. It is commented inside the yaml file. Without providing it, dashboard tries to discover master node locally.
@arhided how are you running your cluster? Locally? AWS/GCE?
floreks
on 23 Jun 2016
Kube is running locally.
~$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1
I think Kube service is there and kube-dash autodiscovery also locationg it properly.
arhided
on 24 Jun 2016
@arhided
Is it local-up-cluster script or kube-up?
local-up-cluster does not offer https endpoint as far as i know. It is only exposed using http protocol and443 which is https.
To make it work with dashboard locally you may want to use hyperkube script that we have on our repository in build folder. Other issue may be that docker daemon is not configured properly and dashboard container can not access kubernetes master.
@bryk any other ideas?
floreks
on 24 Jun 2016
hack/local-up-cluster.sh
Go version: go version go1.6.2 linux/amd64
+++ [0622 17:56:05] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
+++ [0622 17:56:05] Building go targets for linux/amd64:
cmd/kubectl
cmd/hyperkube
+++ [0622 17:56:07] Placing binaries
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused
API SERVER port is free, proceeding...
Detected host and ready to start services. Doing some housekeeping first...
Using GO_OUT /home/user/kubernetes/_output/local/bin/linux/amd64
Starting services now!
Starting etcd
etcd -addr 127.0.0.1:4001 -data-dir /tmp/tmp.hXTnA9YpPY --bind-addr 127.0.0.1:4001 >/dev/null 2>/dev/null
Waiting for etcd to come up.
+++ [0622 17:56:08] On try 2, etcd: :
{"action":"set","node":{"key":"/_test","value":"","modifiedIndex":3,"createdIndex":3}}
Waiting for apiserver to come up
+++ [0622 17:56:09] On try 2, apiserver: : {
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/pods",
"resourceVersion": "10"
},
"items": []
}
Local Kubernetes cluster is running. Press Ctrl-C to shut it down.
Logs:
/tmp/kube-apiserver.log
/tmp/kube-controller-manager.log
/tmp/kube-proxy.log
/tmp/kube-scheduler.log
/tmp/kubelet.log
To start using your cluster, open up another terminal/tab and run:
cluster/kubectl.sh config set-cluster local --server=http://127.0.0.1:8080 --insecure-skip-tls-verify=true
cluster/kubectl.sh config set-context local --cluster=local
cluster/kubectl.sh config use-context local
cluster/kubectl.sh
Http it there
http://127.0.0.1:8080/
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/apps",
"/apis/apps/v1alpha1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/batch",
"/apis/batch/v1",
"/apis/batch/v2alpha1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/apis/policy",
"/apis/policy/v1alpha1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1alpha1",
"/healthz",
"/healthz/ping",
"/logs/",
"/metrics",
"/swaggerapi/",
"/ui/",
"/version"
]
}
https://127.0.0.1:8080/
An error occurred during a connection to 127.0.0.1:8080. SSL received a record that exceeded the maximum permissible length. (Error code: ssl_error_rx_record_too_long)
arhided
on 24 Jun 2016
Hypercube.sh - don`t change much, pretty much nothing.
gcr.io/google_containers/heapster:v1.0.2 "/heapster -port 8082"
http://127.0.0.1:8082/
404 page not found
arhided
on 24 Jun 2016
@floreks, @bryk I ran kubernetes-dashboard.yaml on my master server, with --apiserver-host commented and even ran by uncommenting and gave public IP of my server. Both of above didn't work.
On my minion server(running in a private network)
$ps -ef | grep kube
root 17834 1 1 14:24 ? 00:00:06 /opt/bin/kubelet --address=0.0.0.0 --port=10250 --hostname_override=0.0.0.0 --api_servers=http://163.172.128.148:8080 --logtostderr=true
root 17851 1 0 14:24 ? 00:00:01 /opt/bin/kube-proxy --master=163.172.128.148:8080 --logtostderr=true
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
aa4d35d4898f gcr.io/google_containers/kubernetes-dashboard-amd64:v1.1.0 "/dashboard --port=90" About a minute ago Up About a minute k8s_kubernetes-dashboard.9ae56f1c_kubernetes-dashboard-mc01o_kube-system_280da0d1-3956-11e6-9ed6-0007cb03d000_d5712460
2dd21a03d9d2 gcr.io/google_containers/pause:0.8.0 "/pause" About a minute ago Up About a minute k8s_POD.3e8ce8bb_kubernetes-dashboard-mc01o_kube-system_280da0d1-3956-11e6-9ed6-0007cb03d000_b9769263
$ docker logs aa4d35d4898f
Starting HTTP server on port 9090
Creating API server client for http://163.172.128.148:8080
Successful initial request to the apiserver, version: v1.2.4
Creating in-cluster Heapster client
On my master node(running on a server having public accessible IP)
# ps -ef | grep kube
root 16620 1 0 07:26 ? 00:00:01 /opt/bin/kube-scheduler --master=127.0.0.1:8080 --logtostderr=true --v=0
root 16621 1 0 07:26 ? 00:00:19 /opt/bin/kube-apiserver --insecure-bind-address=0.0.0.0 --insecure-port=8080 --etcd_servers=http://10.1.72.219:2379 --logtostderr=true --service-cluster-ip-range=192.168.3.0/24 --secure-port=0 --service-node-port-range=30000-32767 --service_account_lookup=false
root 16623 1 0 07:26 ? 00:00:17 /opt/bin/kube-controller-manager --master=127.0.0.1:8080 --v=0
#kubectl get rc --namespace=kube-system
NAME DESIRED CURRENT AGE
kubernetes-dashboard 1 1 19h
# kubectl get po --namespace=kube-system
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-mc01o 1/1 Running 0 17h
#kubectl logs kubernetes-dashboard-mc01o
Error from server: pods "kubernetes-dashboard-mc01o" not found
root@onehop-for-docker-testing:~# kubectl logs kubernetes-dashboard-mc01o --namespace=kube-system
Error from server: Get https://0.0.0.0:10250/containerLogs/kube-system/kubernetes-dashboard-mc01o/kubernetes-dashboard: dial tcp 0.0.0.0:10250: connection refused
http://master_ip:8080
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/batch",
"/apis/batch/v1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/healthz",
"/healthz/ping",
"/logs/",
"/metrics",
"/resetMetrics",
"/swagger-ui/",
"/swaggerapi/",
"/ui/",
"/version"
]
}
http://master_ip:8080/api/v1/namespaces/kube-system/services/kubernetes-dashboard/
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "kubernetes-dashboard",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/kubernetes-dashboard",
"uid": "b31b3fbf-3943-11e6-9ed6-0007cb03d000",
"resourceVersion": "210445",
"creationTimestamp": "2016-06-23T13:09:21Z",
"labels": {
"app": "kubernetes-dashboard"
}
},
"spec": {
"ports": [
{
"protocol": "TCP",
"port": 80,
"targetPort": 9090,
"nodePort": 31260
}
],
"selector": {
"app": "kubernetes-dashboard"
},
"clusterIP": "192.168.3.63",
"type": "NodePort",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
}
http://master_ip:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/
Error: 'dial tcp 10.100.22.2:9090: i/o timeout'
Trying to reach: 'http://10.100.22.2:9090/'
I am new to kubernetes, please help me resolve this issue.
Rahul91
on 24 Jun 2016
$ docker logs aa4d35d4898f
Starting HTTP server on port 9090
Creating API server client for http://163.172.128.148:8080
Successful initial request to the apiserver, version: v1.2.4
Creating in-cluster Heapster client
This means that the UI successfully connected to the apiserver. From now on it should just work. I guess this is something wrong with your cluster and proxy settings. Can you change dashboard service from "NodePort" to "LoadBalancer". This would give it an external ip address that you can access.
bryk
on 24 Jun 2016
@arhided
which docker version are you using? which kubernetes version?
could you please paste the output of:
kubectl run debain2 --image debian sleep 1000000
kubectl exec debain2-xxxx ls /var/run/secrets/kubernetes.io/serviceaccount
cheld
on 24 Jun 2016
Hi
$ docker version
Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:16:54 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:16:54 UTC 2015
OS/Arch: linux/amd64
http://127.0.0.1:8080/version
{
"major": "1",
"minor": "3+",
"gitVersion": "v1.3.0-alpha.4.893+2976e892a45fb2-dirty",
"gitCommit": "2976e892a45fb2fdaf458af1c90040c095feb0d1",
"gitTreeState": "dirty",
"buildDate": "2016-06-20T14:02:32Z",
"goVersion": "go1.6.2",
"compiler": "gc",
"platform": "linux/amd64"
}
$ kubectl run debain2 --image debian sleep 1000000
deployment "debain2" created
$ kubectl exec debain2-1251972161-9po9g ls /var/run/secrets/kubernetes.io/serviceaccount
ca.crt
namespace
token
arhided
on 27 Jun 2016
ok, please try:
kubectl run test --image {container-with-curl, e.g. gcr.io/google_containers/hyperkube-amd64:v1.3.0-beta.1 } sleep 100000
kubectl exec test.... curl -k -u admin:admin https://10.0.0.1:443
kubectl exec test.... curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -u admin:admin https://10.0.0.1:443
Can you post the result?
Most likely you have inconsistent certificates. Maybe you killed the cluster during boot-up, or something. It should work if you clean up everything:
docker kill $(docker ps -q)
docker rm $(docker ps -aq)
[reboot]
sudo rm -R /var/lib/kubelet
sudo rm -R /var/run/kubernetes
cheld
on 27 Jun 2016
$kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-node-3689398978-2tqvf 1/1 Running 0 29m
test-541238630-gby7g 1/1 Running 0 9m
$ kubectl exec test-541238630-gby7g -- curl -k -u admin:admin https://10.0.0.1:443
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0^C
$ kubectl exec test-541238630-gby7g -- curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -u admin:admin https://10.0.0.1:443
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0^C
arhided
on 27 Jun 2016
Looks like there is no connectivity between containers.
arhided
on 28 Jun 2016
Got it working, i rebuild kuber-cluster and now it's show's web UI:
http://127.0.0.1:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/pod?namespace=default and http://10.0.0.96/#/workload?namespace=default
Thx.
arhided
on 28 Jun 2016
Oh, perfect! I'm closing the issue. Please reopen if needed.
bryk
on 28 Jun 2016
Still having this issue. Dashboard works fine right after K8s installation, but fails to start upon reboot.
kubectl get pods -n kube-system
kubernetes-dashboard-3543765157-4ftml 0/1 CrashLoopBackOff 1 12s
kubectl logs kubernetes-dashboard-3543765157-4ftml -n kube-system
Using HTTP port: 9090
Creating API server client for https://10.3.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.3.0.1:443/version: dial tcp 10.3.0.1:443: getsockopt: no route to host
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
sarastinishi
on 14 Jan 2017
For me, the solution was to loosen up overzealous firewall rules preventing the dashboard from accessing the subnet associated with the flannel interface. Because this subnet changed with every docker service restart, it was a few rounds of whack-a-mole before I realized what was going on.
cachance7
on 25 Jan 2017
open the port 6443 on firewall will be ok.
the follow is the firewall reject log before open the port:
Feb 25 00:04:30 k8s kernel: XFS (dm-32): Mounting V5 Filesystem
Feb 25 00:04:30 k8s kernel: XFS (dm-32): Ending clean mount
Feb 25 00:04:32 k8s kernel: FINAL_REJECT: IN=docker0 OUT= PHYSIN=veth2fd9745 MAC=02:42:cf:c5:2c:da:02:42:0a:01:49:03:08:00 SRC=10.1.73.3 DST=192.168.1.112 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11531 DF PROTO=TCP SPT=38734 DPT=6443 WINDOW=28640 RES=0x00 SYN URGP=0
Feb 25 00:04:33 k8s kernel: FINAL_REJECT: IN=docker0 OUT= PHYSIN=veth2fd9745 MAC=02:42:cf:c5:2c:da:02:42:0a:01:49:03:08:00 SRC=10.1.73.3 DST=192.168.1.112 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11532 DF PROTO=TCP SPT=38734 DPT=6443 WINDOW=28640 RES=0x00 SYN URGP=0
Feb 25 00:04:33 k8s dockerd: time="2017-02-25T00:04:33.935301481+08:00" level=error msg="containerd: deleting container" error="exit status 1: \"container dcb4a44031b96470eaef50eb8ac4ee2b9f958906702d94645c3a45c4852b6335 does not exist\\none or more of the container deletions failed\\n\""
Feb 25 00:04:34 k8s kernel: XFS (dm-32): Unmounting Filesystem
winse
on 24 Feb 2017
I see this issue too with kubernetes 1.5.4 and kubernetes-dashboard image version gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0.
I installed kubeadm referring https://kubernetes.io/docs/getting-started-guides/kubeadm/, and then installed kubernetes-dashboard by doing
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.0/src/deploy/kubernetes-dashboard.yaml
I see the kubernetes-dashboard in CrashLoopBackOff status and the k8s_kubernetes-dashboard.* container on the worker is in Exited state.
Below are the errors. Has anyone successfully installed kubernetes-dashboard on kubeadm?
# kubectl --namespace=kube-system get all
NAME READY STATUS RESTARTS AGE
po/calico-policy-controller-mqsmh 1/1 Running 0 4h
po/canal-etcd-tm2rv 1/1 Running 0 4h
po/canal-node-3nv2t 3/3 Running 0 4h
po/canal-node-5fckh 3/3 Running 1 4h
po/canal-node-6zgq8 3/3 Running 0 4h
po/canal-node-rtjl8 3/3 Running 0 4h
po/dummy-2088944543-09w8n 1/1 Running 0 4h
po/etcd-vhosakot-kolla-kube1.localdomain 1/1 Running 0 4h
po/kube-apiserver-vhosakot-kolla-kube1.localdomain 1/1 Running 2 4h
po/kube-controller-manager-vhosakot-kolla-kube1.localdomain 1/1 Running 0 4h
po/kube-discovery-1769846148-pftx5 1/1 Running 0 4h
po/kube-dns-2924299975-9m2cp 4/4 Running 0 4h
po/kube-proxy-0ndsb 1/1 Running 0 4h
po/kube-proxy-h7qrd 1/1 Running 1 4h
po/kube-proxy-k6168 1/1 Running 0 4h
po/kube-proxy-lhn0k 1/1 Running 0 4h
po/kube-scheduler-vhosakot-kolla-kube1.localdomain 1/1 Running 0 4h
po/kubernetes-dashboard-3203962772-mw26t 0/1 CrashLoopBackOff 11 41m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/canal-etcd 10.96.232.136 <none> 6666/TCP 4h
svc/kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 4h
svc/kubernetes-dashboard 10.100.254.77 <nodes> 80:30085/TCP 41m
NAME DESIRED SUCCESSFUL AGE
jobs/configure-canal 1 1 4h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/kube-discovery 1 1 1 1 4h
deploy/kube-dns 1 1 1 1 4h
deploy/kubernetes-dashboard 1 1 1 0 41m
NAME DESIRED CURRENT READY AGE
rs/calico-policy-controller 1 1 1 4h
rs/dummy-2088944543 1 1 1 4h
rs/kube-discovery-1769846148 1 1 1 4h
rs/kube-dns-2924299975 1 1 1 4h
rs/kubernetes-dashboard-3203962772 1 1 0 41m
# kubectl --namespace=kube-system describe pod kubernetes-dashboard-3203962772-mw26t
20m 5s 89 {kubelet vhosakot-kolla-kube2.localdomain} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203962772-mw26t_kube-system(67b0d69b-0b47-11e7-8c97-7a2ed4192438)"
# kubectl --namespace=kube-system logs kubernetes-dashboard-3203962772-mw26t
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
# docker ps -a | grep -i dash
3c33cf43d5e4 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0 "/dashboard --port=90" 54 seconds ago Exited (1) 22 seconds ago k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4
# docker logs k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
vhosakot
on 17 Mar 2017
I'm not sure what it is about kubeadm specifically, but I was able to get this working by forcing dashboard to run on master. I did this using nodeSelector in the kubernetes-dashboard.yaml file:
nodeSelector:
node-role.kubernetes.io/master:
Once I did that and re-added the service, it worked a charm!
otakumike
on 12 Jun 2017
@otakumike worked for me after i tainted my master node as per the kubeadm doco
kubectl taint nodes --all node-role.kubernetes.io/master-
I think this appears like something to do with with the kubeadm install, but am too noob right now to be able to debug. Any clues about where I should look, happy to start posting anything useful.
timm088
on 12 Jun 2017
By default kubeadm does not allow running applications without specified selector on master node.
First solution is to specify selector as @otakumike did. Second option is to remove taint from the master to allow deploy of applications without this selector as @timm088 did.
- sign at the end of taint removes it.
floreks
on 12 Jun 2017
Facing the same issue, too. Install k8s cluster with kubeadm: v1.6.4 and dashboard.
solved by using @otakumike way and ran on master node successfully. Cheers!
But are there anyway not run applications on master node? or it would just fine to run applications on master node?
Beanking77
on 22 Jun 2017
Kubeadm by default taints master node so all applications without proper toleration set can not be started on master node.
kubectl taint nodes --all node-role.kubernetes.io/master-
This removed the default taint and allows to run applications on master. Dashboard does not have to be run on master node. It might be any node in the cluster. Usually you shouldn't run normal applications on master unless you need to.
floreks
on 22 Jun 2017
@floreks thanks for your quick reply.
But recently I can only run dashboard successfully on master node, same error logs as @vhosakot when dashboard is on another nodes. Is this would be a issue?
Beanking77
on 22 Jun 2017
If there is a connection timeout/refused error then usually it means that there is a problem with networking/dns in your cluster. For in-cluster config dashboard uses service proxy to access apiserver and for that to work service discovery and dns have to work correctly.
The easiest way to check that is to create busybox container and do an nslookup like described here: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
floreks
on 22 Jun 2017
I got some error logs when verify DNS setting, and been solved by add (OS: Centos 7.3)
iptables -I INPUT -p tcp -m tcp --dport 8472 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 9898 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
it looks not good to me,
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes.default'
and still got error when I try to run dashboard on another node.
docker logs k8s_kubernetes-dashboard_kubernetes-dashboard-2039414953-qdbwx_kube-system_812c20bf-572a-11e7-b95d-b8aeed766268_2
Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: getsockopt: no route to host
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
Should I need to delete DNS pod and auto-create it?
Beanking77
on 22 Jun 2017
Definitely you have some kind of networking issue in your cluster and dashboard can't access apiserver. Try checking existing/closed issues on kubernetes repository. I think simple pod deletion won't resolve this. It might be related to system/firewall.
floreks
on 22 Jun 2017
@floreks Thanks you so much for your reply and now dashboard working on minion perfectly.
OS: CentOS 7.3
stop the firewall
$ systemctl stop firewalld
$ systemctl disable firewalld
and make sure
/usr/lib/sysctl.d/00-system.conf config are
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
and iptable rules are
iptables -I INPUT -p tcp -m tcp --dport 8472 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 9898 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
No problem. Great to hear that it works!
floreks
on 23 Jun 2017
Hello,
I am having the same issue, I have created a cluster with one master (server-1) and one minion(server-2) running on ubuntu and using flannel for networking and using kubeadm. And I am trying to run the dashboard from the minion server-2. I am very new to kubernetes and not an expert on linux networking setup so any help would be appreciated.
kubectl and kubeadm : 1.6.6
Docker: 17.03.1-ce
This is the command I followed to install my cluster
1) Installing master
kubeadm init --pod-network-cidr=10.244.0.0/16
2) Installing flannel
kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
Setting-up RBAC
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
3) Joining from minion
kubeadm join --token <token> <master-ip>:<master-port>
4) Installing Dashboard
kubectl create -f https://git.io/kube-dashboard
Now here are all the pods I see :
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-server-1 1/1 Running 0 15h
kube-system kube-apiserver-server-1 1/1 Running 0 15h
kube-system kube-controller-manager-server-1 1/1 Running 0 15h
kube-system kube-dns-692378583-5zj21 3/3 Running 0 15h
kube-system kube-flannel-ds-df0f8 2/2 Running 4 15h
kube-system kube-flannel-ds-dzkqk 2/2 Running 0 15h
kube-system kube-proxy-gwbjj 1/1 Running 0 15h
kube-system kube-proxy-sx8qb 1/1 Running 0 15h
kube-system kube-scheduler-server-1 1/1 Running 0 15h
kube-system kubernetes-dashboard-2039414953-8frcf 0/1 CrashLoopBackOff 179 15h
Here are the services I see
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.96.0.1 <none> 443/TCP 15h
kube-system kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 15h
kube-system kubernetes-dashboard 10.103.141.154 <none> 80/TCP 15h
Dasbard is in error with
Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
Dashboard is running on minion (server-2), here is the desciption of the pod (just removed the real ip)
Name: kubernetes-dashboard-2039414953-8frcf
Namespace: kube-system
Node: server-2/xxx.xxx.xxx.xxx
Start Time: Tue, 27 Jun 2017 16:44:08 +0000
Labels: k8s-app=kubernetes-dashboard
pod-template-hash=2039414953
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kubernetes-dashboard-2039414953","uid":"d7031736-5b57-11e7-8...
Status: Running
IP: 10.244.1.19
Controllers: ReplicaSet/kubernetes-dashboard-2039414953
Containers:
kubernetes-dashboard:
Container ID: docker://1776d490980e860f97ab2ad94783e10c30166a5e6c53f72bfe0d6161e9b10863
Image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1
Image ID: docker-pullable://gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:b537ce8988510607e95b8d40ac9824523b1f9029e6f9f90e9fccc663c355cf5d
Port: 9090/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 28 Jun 2017 08:14:23 +0000
Finished: Wed, 28 Jun 2017 08:14:53 +0000
Ready: False
Restart Count: 180
Liveness: http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-rnvnc (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
kubernetes-dashboard-token-rnvnc:
Type: Secret (a volume populated by a Secret)
SecretName: kubernetes-dashboard-token-rnvnc
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node-role.kubernetes.io/master=:NoSchedule
node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
15h 2m 181 kubelet, server-2 spec.containers{kubernetes-dashboard} Normal Pulled Container image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1" already present on machine
15h 2m 172 kubelet, server-2 spec.containers{kubernetes-dashboard} Normal Created (events with common reason combined)
15h 2m 172 kubelet, server-2 spec.containers{kubernetes-dashboard} Normal Started (events with common reason combined)
15h 11s 3893 kubelet, server-2 spec.containers{kubernetes-dashboard} Warning BackOff Back-off restarting failed container
15h 11s 3863 kubelet, server-2 Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-2039414953-8frcf_kube-system(d708bf94-5b57-11e7-89e4-fa163ebaf0b4)"
Here is iptables from server 1
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-ISOLATION all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-SERVICES (2 references)
target prot opt source destination
REJECT tcp -- anywhere 10.103.141.154 /* kube-system/kubernetes-dashboard: has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable
here are iptables from server-2
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-ISOLATION all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-SERVICES (2 references)
target prot opt source destination
REJECT tcp -- anywhere 10.103.141.154 /* kube-system/kubernetes-dashboard: has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable
I have been looking at the following troubleshooting guide but I could not find a solution (maybe I missed something) : https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
The starting error I got was when trying to run kubectl proxy and open following url from my workstation I get
http://127.0.0.1:8001/ui/
Get redirect (307) to
http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"kubernetes-dashboard\"",
"reason": "ServiceUnavailable",
"code": 503
}
ronanquillevere
on 28 Jun 2017
@ronanquillevere Did you see my update on this from 16 days prior, about running dashboard on the master? I think that should solve your problem.
otakumike
on 28 Jun 2017
@otakumike I am trying to do that right now to move on but still I would like to run it on the minion and understand WHY i is not working right now :)
The point is I have other issues when trying to run other kind of services which are maybe also related, I think there is a networking problem that I do not understand, but again I am a newbee.
ronanquillevere
on 28 Jun 2017
@ronanquillevere Me too! :) If you find a solution to that please do post it here.
otakumike
on 28 Jun 2017
This is not a dashboard issue. First check you cluster networking and dns: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
There is a simple busybox container that allows you to do basic networking/dns check. Run the container and execute kubectl exec -ti busybox -- nslookup kubernetes.default. If this fails to resolve then you have a cluster issue and any application that would need to connect to other app in the cluster through services will fail.
For this kind of issues create one in core repository. They are more experienced in this area.
floreks
on 28 Jun 2017
@otakumike @ronanquillevere I faced the same issue and after I followed @floreks suggest to check dns/firewall and fixed it by some steps I post 5 days ago, and the dashboard can works on minion now.
Maybe you could try the same way in ubuntu.
pod list: (test-kube01 - master)
NAME READY STATUS RESTARTS AGE IP NODE default-http-backend-2198840601-nhj46 1/1 Running 0 4d 10.244.2.7 test-kube03 etcd-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 heapster-2315332064-x7zn5 1/1 Running 0 5d 10.244.2.3 test-kube03 kube-apiserver-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 kube-controller-manager-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 kube-dns-3913472980-hhkkx 3/3 Running 0 5d 10.244.0.25 test-kube01 kube-flannel-ds-9nkxq 2/2 Running 0 5d 10.1.30.15 test-kube02 kube-flannel-ds-jf3bk 2/2 Running 0 5d 10.1.30.16 test-kube03 kube-flannel-ds-k4ctc 2/2 Running 0 5d 10.1.30.14 test-kube01 kube-proxy-l2cjw 1/1 Running 0 5d 10.1.30.14 test-kube01 kube-proxy-lg6b4 1/1 Running 0 5d 10.1.30.15 test-kube02 kube-proxy-mx057 1/1 Running 0 5d 10.1.30.16 test-kube03 kube-scheduler-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 kubernetes-dashboard-2039414953-65392 1/1 Running 0 5d 10.244.2.2 test-kube03 monitoring-grafana-3975459543-c9pl1 1/1 Running 0 5d 10.244.3.3 test-kube02 monitoring-influxdb-3480804314-lnrr3 1/1 Running 0 5d 10.244.3.4 test-kube02 nginx-ingress-controller-429549519-q8jvl 1/1 Running 0 4d 10.1.30.15 test-kube02 weave-scope-agent-6chk2 1/1 Running 0 4d 10.1.30.16 test-kube03 weave-scope-agent-dvgz7 1/1 Running 0 4d 10.1.30.15 test-kube02 weave-scope-agent-zqpfb 1/1 Running 0 4d 10.1.30.14 test-kube01 weave-scope-app-879505699-4lwx9 1/1 Running 0 4d 10.244.2.8 test-kube03
dns test with kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
Beanking77
on 28 Jun 2017
@Beanking77
added the iptables rules on master (server-1). But on my servers, no firewalld is running (see list below of enabled services) and no such file : /usr/lib/sysctl.d/00-system.conf
sudo ufw status
Status: inactive
It is not working better but I am probably missing something (I am a newbie on linux networking config).
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp dpt:9898
ACCEPT tcp -- anywhere anywhere tcp dpt:6443
ACCEPT tcp -- anywhere anywhere tcp dpt:8472
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
...
ubuntu@server-1:~$ systemctl list-unit-files | grep enabled
acpid.path enabled
accounts-daemon.service enabled
apparmor.service enabled
atd.service enabled
[email protected] enabled
cloud-config.service enabled
cloud-final.service enabled
cloud-init-local.service enabled
cloud-init.service enabled
console-setup.service enabled
cron.service enabled
dbus-org.freedesktop.resolve1.service enabled
docker.service enabled
ebtables.service enabled
friendly-recovery.service enabled
[email protected] enabled
iscsi.service enabled
iscsid.service enabled
keyboard-setup.service enabled
kubelet.service enabled
lvm2-monitor.service enabled
lxcfs.service enabled
lxd-containers.service enabled
networking.service enabled
ondemand.service enabled
open-iscsi.service enabled
open-vm-tools.service enabled
pollinate.service enabled
resolvconf.service enabled
rsyslog.service enabled
setvtrgb.service enabled
snapd.autoimport.service enabled
snapd.service enabled
snapd.system-shutdown.service enabled
ssh.service enabled
sshd.service enabled
syslog.service enabled
systemd-resolved.service enabled
systemd-timesyncd.service enabled
ufw.service enabled
unattended-upgrades.service enabled
ureadahead.service enabled
acpid.socket enabled
apport-forward.socket enabled
dm-event.socket enabled
docker.socket enabled
lvm2-lvmetad.socket enabled
lvm2-lvmpolld.socket enabled
lxd.socket enabled
snapd.socket enabled
uuidd.socket enabled
remote-fs.target enabled
apt-daily.timer enabled
snapd.refresh.timer enabled
@ronanquillevere According to https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ at limitation section, only RHEL/CentOS 7 user might to edit that file (I'm CentOS 7.3).
Have you test your network, like test dns with kubectl exec -ti busybox -- nslookup kubernetes.default.
Make sure network should work properly and the dashboard should work on minion.
Beanking77
on 28 Jun 2017
@Beanking77 @floreks I am looking at that right now, try to fix my DNS :)
kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes.default'
But everything looks fine
kubectl exec busybox cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local local
options ndots:5
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
kube-dns-692378583-5zj21 3/3 Running 0 17h
I got not error/warnings in the logs of kubedns and dnsmasq. I do not have the healthz container but I believe this is not an issue.
And my DNS service is up an exposing endpoins
ubuntu@server-1:~$ kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 17h
kubernetes-dashboard 10.97.135.242 <none> 80/TCP 1h
ubuntu@server-1:~$ kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns 10.244.0.4:53,10.244.0.4:53 17h
Created a new issue
https://github.com/kubernetes/dns/issues/118
ronanquillevere
on 28 Jun 2017
@ronanquillevere In my experienced in CentOS, it might be firewall or ip table rule issue.
But I'm not quite sure if it happen in ubuntu, too. It might be related to system/firewall.
Beanking77
on 28 Jun 2017
I have a similar issue with kubernetes dashboard. I have used kubeadm for k8s install and then installed dashboard using kubectl apply -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml. But from a long time ( more than 40 minutes) the state is still in ContainerCreating. I don't seem to understand the error. Can someone help me with this please.
When i try to check the logs with _kubectl logs --namespace=kube-system kubernetes-dashboard-xxxxx_ i get Error from server (Bad Request): container "kubernetes-dashboard" in pod "kubernetes-dasshboard-xxx" is waiting to start: ContainerCreating
@floreks i did install busybox as per the link you mentioned. But even there the pod busybox remains in pending state. Could you please suggest on how I can proceed? With this i get pod busybox does not have a host assigned
spnzig
on 3 Aug 2017
Did you taint the node to allow application deployment on master?
kubectl taint nodes --all node-role.kubernetes.io/master-
floreks
on 3 Aug 2017
@floreks it gives error: at least one taint update is required
spnzig
on 3 Aug 2017
The instead of --all specify node name you want to taint.
floreks
on 3 Aug 2017
@floreks I get the same error with kubectl taint nodes "master_node_name" node-role.kubernetes.io/master. I am a newbie to K8S. So could you please elaborate
spnzig
on 3 Aug 2017
Did you forgot - sign at the end of command?
floreks
on 3 Aug 2017
Oh i didn't notice that. Now i get node "master-node-name" untainted. Now after running the kube-proxy command i get 503 service unavailable error
spnzig
on 3 Aug 2017
This is not a dashboard issue but cluster. As you have used kubeadm please try to ask for help on https://github.com/kubernetes/kubeadm or https://github.com/kubernetes/kubernetes. We are dealing here only with dashboard specific problems.
floreks
on 3 Aug 2017
@floreks thanks for your help. I will post this on kubeadm channel.
spnzig
on 3 Aug 2017
Yes, this would I alsko hope to know: "Has anyone successfully installed kubernetes-dashboard on kubeadm?" I am wasting more then week looking into kubernetes documentation on kubernetes site but there is no explanation and real how-to to do that. Looks like guys are having some issues with know-how and know-how-to-write-how-to.
vglisin
on 2 Oct 2017
kubectl -n kube-system get service kubernetes-dashboard
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard 10.105.94.132
kubectl get nodes
NAME STATUS AGE VERSION
masternode.local.com Ready 4d v1.7.5
minionone.local.com Ready 3d v1.7.5
miniontwo.local.com Ready 3d v1.7.5
http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
Getting response:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"kubernetes-dashboard\"",
"reason": "ServiceUnavailable",
"code": 503
}
Anybody idea?
vglisin
on 2 Oct 2017
"Has anyone successfully installed kubernetes-dashboard on kubeadm?"
I believe, that thousands of people did.
The whole setup is explained. You should read our wiki pages like @floreks recommended. If you have any specific questions ask them here or in a new issue.
maciaszczykm
on 2 Oct 2017
Well, somehow you have found your way to this repository then I can only assume that it is too difficult to read Getting started section from our main README, execute 2 commands and open 1 link to access Dashboard.
And yes, it works on kubeadm as we are (dev team) using it...
PS. Instead of going through issues and blaming us you could have just create issue, describe in-detail your problem and wait for help.
floreks
on 2 Oct 2017
See related issue.
https://github.com/kubernetes/dashboard/issues/1303
claudiuchis
on 3 Oct 2017
sysctl net.ipv4.ip_forward=1
https://kubernetes.io/docs/concepts/cluster-administration/networking/
claudiuchis
on 3 Oct 2017
Actually, what worked for me was to run this command on the nodes:
sudo iptables -P FORWARD ACCEPT
The problem was that packets where not leaving nodes, so none of the pods that were running on the nodes (and not the master) had any connectivity.
Found the solution in this related post:
https://github.com/kubernetes/kubernetes/issues/45022
To make this change persistent, add this line to /etc/sysctl.conf (I'm using Ubuntu 16.04):
net.ipv4.ip_forward=1
Then, if you run "sudo iptables-save", you should see ip forwarding enabled:
*filter
:FORWARD ACCEPT [4:1088]
claudiuchis
on 16 Oct 2017
"Has anyone successfully installed kubernetes-dashboard on kubeadm?"
I believe, that thousands of people did.
I'm willing to bet thousands more have not
mabushey
on 18 Apr 2018
i'm the thousands and one that not getting it work, even follow all @floreks did.
still getting the below messages, been trying it out 48hours.
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "no endpoints available for service \"https:kubernetes-dashboard:\"",
"reason": "ServiceUnavailable",
"code": 503
}
thyeun
on 4 Aug 2018
Getting the following:
http://MASTERIP:9999/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
Getting response:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service "kubernetes-dashboard"",
"reason": "ServiceUnavailable",
"code": 503
}
shanekorbel
on 21 Oct 2018
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "kubernetes-dashboard",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/kubernetes-dashboard",
"uid": "830b32c4-d9fe-11e8-b980-025000000001",
"resourceVersion": "1220904",
"creationTimestamp": "2018-10-27T15:39:37Z",
"labels": {
"k8s-app": "kubernetes-dashboard"
},
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"kubernetes-dashboard\"},\"name\":\"kubernetes-dashboard\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"port\":443,\"targetPort\":8443}],\"selector\":{\"k8s-app\":\"kubernetes-dashboard\"}}}\n"
}
},
"spec": {
"ports": [
{
"protocol": "TCP",
"port": 443,
"targetPort": 8443,
"nodePort": 31019
}
],
"selector": {
"k8s-app": "kubernetes-dashboard"
},
"clusterIP": "10.101.12.65",
"type": "NodePort",
"sessionAffinity": "None",
"externalTrafficPolicy": "Cluster"
},
"status": {
"loadBalancer": {
"ingress": [
{
"hostname": "localhost"
}
]
}
}
}
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "no endpoints available for service \"https:kubernetes-dashboard:\"",
"reason": "ServiceUnavailable",
"code": 503
}
kids-return
on 27 Oct 2018
My problem is solved
Because docker didn't pull successfully k8s.gcr.io/kubernetes-dashboard-amd64
Check with docker images to make sure there is k8s.gcr.io/kubernetes-dashboard-amd64
Inspection records：
kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
etcd-docker-for-desktop 1/1 Running 0 30d
kube-apiserver-docker-for-desktop 1/1 Running 0 30d
kube-controller-manager-docker-for-desktop 1/1 Running 2 30d
kube-dns-86f4d74b45-p2xmk 3/3 Running 0 30d
kube-proxy-mbfbb 1/1 Running 0 30d
kube-scheduler-docker-for-desktop 1/1 Running 0 30d
kubernetes-dashboard-7b9c7bc8c9-pkhqk 0/1 ImagePullBackOff 0 1h
or
kubernetes-dashboard-7b9c7bc8c9-pkhqk 0/1 ErrImagePull 0 1h
kubectl describe pod kubernetes-dashboard-7b9c7bc8c9-pkhqk --namespace=kube-system
Warning Failed 18m (x4 over 21m) kubelet, docker-for-desktop Failed to pull image "k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Normal Pulling 19m (x4 over 21m) kubelet, docker-for-desktop pulling image "k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0"
Warning Failed 18m (x4 over 21m) kubelet, docker-for-desktop Failed to pull image "k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Same here.
I found that one of my nodes was unhealthy. Scaled down to kill the node and scaled up to create a new one.
yianni-ververis
on 29 Oct 2018
This is not a dashboard issue. First check you cluster networking and dns: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
There is a simple busybox container that allows you to do basic networking/dns check. Run the container and execute
kubectl exec -ti busybox -- nslookup kubernetes.default. If this fails to resolve then you have a cluster issue and any application that would need to connect to other app in the cluster through services will fail.For this kind of issues create one in core repository. They are more experienced in this area.
@floreks I have still this error:
kubectl logs kubernetes-dashboard-5f7b999d65-pgdgr -n kube-system
2019/03/31 16:30:47 Starting overwatch
2019/03/31 16:30:47 Using in-cluster config to connect to apiserver
2019/03/31 16:30:47 Using service account token for csrf signing
2019/03/31 16:31:17 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ
But I am able to nslookup kubernetes.default from an ubuntu pod:
root@my-shell-75b487f578-hhkkc:/# nslookup kubernetes.default
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
lvthillo
on 31 Mar 2019
Dashboard not working... and isn't issue of firewall , the check using curl work fine
kube-master 1.14.1
kube-slave1 1.14.1
kube-slave2 1.14.1
Every 1,0s: kubectl -n kube-system get all -o wide Sun Apr 28 16:51:28 2019
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/coredns-fb8b8dccf-cnlz4 1/1 Running 0 19m 172.16.0.13 k8s-n1
pod/coredns-fb8b8dccf-h5sxc 1/1 Running 0 19m 172.16.0.12 k8s-n1
pod/etcd-k8s-n1 1/1 Running 0 18m 192.168.200.10 k8s-n1
pod/kube-apiserver-k8s-n1 1/1 Running 0 18m 192.168.200.10 k8s-n1
pod/kube-controller-manager-k8s-n1 1/1 Running 0 18m 192.168.200.10 k8s-n1
pod/kube-flannel-ds-amd64-24zrk 1/1 Running 0 19m 192.168.200.10 k8s-n1
pod/kube-flannel-ds-amd64-8pwtr 1/1 Running 0 12m 192.168.200.12 k8s-n3
pod/kube-flannel-ds-amd64-mjbh6 1/1 Running 0 12m 192.168.200.11 k8s-n2
pod/kube-proxy-9rg7q 1/1 Running 0 19m 192.168.200.10 k8s-n1
pod/kube-proxy-h65j5 1/1 Running 0 12m 192.168.200.11 k8s-n2
pod/kube-proxy-ssrq9 1/1 Running 0 12m 192.168.200.12 k8s-n3
pod/kube-scheduler-k8s-n1 1/1 Running 0 18m 192.168.200.10 k8s-n1
pod/kubernetes-dashboard-5d5958d7b5-jhx2d 1/1 Running 4 4m2s 172.16.2.45 k8s-n3
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kube-dns ClusterIP 10.96.0.10
service/kubernetes-dashboard NodePort 10.99.124.181
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
daemonset.apps/kube-flannel-ds-amd64 3 3 3 3 3 beta.kubernetes.io/arch=amd64 19m kube-flannel quay.io/coreos/flannel:v0.11.0-amd64 app=flannel,tier=node
daemonset.apps/kube-flannel-ds-arm 0 0 0 0 0 beta.kubernetes.io/arch=arm 19m kube-flannel quay.io/coreos/flannel:v0.11.0-arm app=flannel,tier=node
daemonset.apps/kube-flannel-ds-arm64 0 0 0 0 0 beta.kubernetes.io/arch=arm64 19m kube-flannel quay.io/coreos/flannel:v0.11.0-arm64 app=flannel,tier=node
daemonset.apps/kube-flannel-ds-ppc64le 0 0 0 0 0 beta.kubernetes.io/arch=ppc64le 19m kube-flannel quay.io/coreos/flannel:v0.11.0-ppc64le app=flannel,tier=node
daemonset.apps/kube-flannel-ds-s390x 0 0 0 0 0 beta.kubernetes.io/arch=s390x 19m kube-flannel quay.io/coreos/flannel:v0.11.0-s390x app=flannel,tier=node
daemonset.apps/kube-proxy 3 3 3 3 3
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/coredns 2/2 2 2 19m coredns k8s.gcr.io/coredns:1.3.1 k8s-app=kube-dns
deployment.apps/kubernetes-dashboard 1/1 1 1 4m2s kubernetes-dashboard k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1 k8s-app=kubernetes-dashboard
[root@k8s-n1 dashboard]# kubectl logs pod/kubernetes-dashboard-5d5958d7b5-jhx2d -n kube-system
2019/04/28 14:49:42 Starting overwatch
2019/04/28 14:49:42 Using apiserver-host location: https://10.96.0.1:443
2019/04/28 14:49:42 Skipping in-cluster config
2019/04/28 14:49:42 Using random key for csrf signing
2019/04/28 14:50:12 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ
when start pod running fine till 1m passed after time out.
CrashLoopBackOff
[root@k8s-n1 dashboard]# ssh root@k8s-n3
Last login: Sun Apr 28 16:49:47 2019 from k8s-n1
[root@k8s-n3 ~]# curl https://10.96.0.1:443/version -k
{
"major": "1",
"minor": "14",
"gitVersion": "v1.14.1",
"gitCommit": "b7394102d6ef778017f2ca4046abbaa23b88c290",
"gitTreeState": "clean",
"buildDate": "2019-04-08T17:02:58Z",
"goVersion": "go1.12.1",
"compiler": "gc",
"platform": "linux/amd64"
emper0r
on 28 Apr 2019
After trying out every fix I found, what finally granted me access to the dashbord was this URL:
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:https/proxy/
Notice this part: /https:kubernetes-dashboard:https/
Without adding the https it didn't work for me and I always got "no endpoints available for service "kubernetes-dashboard"".
Found the working link in the readme here: https://github.com/helm/charts/tree/master/stable/kubernetes-dashboard
c-stoeckl
on 28 May 2019
@Rahul91 As your master is located on external server try to provide
apiserver-hostparameter to the dashboard. It is commented inside the yaml file. Without providing it, dashboard tries to discover master node locally.@arhided how are you running your cluster? Locally? AWS/GCE?
@floreks,
Thanks a ton for your suggestion. I was stuck with the same issue with no endpoint available for dashboard. My pod was not up as my apiserver-host which is my master is located on external ip. After i provided this parameter in recommended.yaml, my dashboard pod and its endpoint is up on pod IP.
Cheers,
Sriram
sriramperumalla
on 26 Dec 2019
In case this helps someone (after being incredibly frustrated trying to get this working)... Thanks to all those who commented above!
(I was getting an error similar to the OP, with no endpoints available for the service when accessing the URL, and the logging showing:
Error: 'dial tcp 10.100.22.2:9090: i/o timeout'
Trying to reach: 'http://10.100.22.2:9090/')
Raspbian Buster, 3x raspberry pi 4 cluster. Wasn't able to access dashboard by following the instructions - dashboard pod not running on the master, using flannel, setup mostly following the guide here: teamserverless/k8s-on-raspbian Guide (with some badly formatted notes on my fork here )
This worked for me to get dashboard working after running
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc7/aio/deploy/recommended.yaml
per the instructions...
On each node, edit /etc/sysctl.d/99-sysctl.conf
sudo nano /etc/sysctl.d/99-sysctl.conf
uncomment the line
net.ipv4.ip_forward=1
add the lines
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
reboot
run kubectl proxy on the master
on the master (gui desktop), use your browser to navigate to
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login
That takes you to the token login page.... :)
chrisbu
on 1 Apr 2020
kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='.*'
This won't work with recommended config. You can see the login page, but you won't be able to log in. Please remove this part, because people will try it and then create an issue as login will not work. This is not a bug though.
floreks
on 1 Apr 2020
Awesome - thanks (edited and removed that bit)
chrisbu
on 1 Apr 2020
Related issues
shu-mutou
·
3Comments
kasunsjc
·
3Comments
Eddman
·
4Comments
maciaszczykm
·
3Comments
donspaulding
·
5Comments
Most helpful comment
After trying out every fix I found, what finally granted me access to the dashbord was this URL:
Notice this part: /https:kubernetes-dashboard:https/
Without adding the https it didn't work for me and I always got "no endpoints available for service "kubernetes-dashboard"".
Found the working link in the readme here: https://github.com/helm/charts/tree/master/stable/kubernetes-dashboard