Dashboard: Cannot access dashboard using kubectl proxy

[montanaflynn]

Steps to reproduce

1. Run kubectl proxy
2. Visit http://127.0.0.1:8001/ui/ which redirects to http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/workload?namespace=default and shows the dashboard HTML sidebar and header except with the following error in the main viewport:

error_outline Service Unavailable (503)

Error: 'EOF'
Trying to reach: 'http://10.12.6.4:9090/api/v1/workload/default?itemsPerPage=10&page=1'

Environment

Installation method: using google container engine
Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.4-gke.0", GitCommit:"b8463befa7d1cdc2eba7452b7608a31f4087e79d", GitTreeState:"clean", BuildDate:"2017-05-22T18:58:26Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Dashboard version: v1.6.0
Commit: bfab10151f012d1acc5dfb1979f3172e2400aa3c

Observed result

Dashboard reported Service Unavailable (503):

Error: 'ssh: rejected: connect failed (Connection refused)'
Trying to reach: 'http://10.12.6.4:9090/api/v1/workload/default?itemsPerPage=10&page=1'```

[montanaflynn]

Also sometimes I don't get the html page with the error above but this JSON instead:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "no endpoints available for service \"kubernetes-dashboard\"",
  "reason": "ServiceUnavailable",
  "code": 503
}

[floreks]

It means that container has crashed. Probably the reason is Out of memory. Try increasing memory and request limits for dashboard.

[maciaszczykm]

@montanaflynn Did @floreks' advice help?

[montanaflynn]

I just tried:

kubectl edit deployment kubernetes-dashboard --namespace kube-system

increased both requests and limit from 50m to 200m and the dashboard works again. I'm not sure if it was increasing the limit or the restart that fixed the problem though.

[maciaszczykm]

@montanaflynn Increase fixed it, 50m is too low. Actually, as far as I remember we set 300m defaults in Kubernetes core repository few weeks ago.

[jingpengw]

I changed to 300 Mi, but it went back to 50Mi automatically a few seconds later...

[floreks]

For sure it wasn't changed by dashboard. Maybe kubernetes addon manager reconfigured it.

[derluke]

I have the same problem - if I change the limits it gets reset right away. I'm using kubernetes on google cloud. Is there a workaround? Do I have to raise it with google?

[floreks]

It gets reset because on GCP Dashboard is installed as cluster addon. There is a yaml on the disk that is synced from time to time and it makes sure that you can not change addons configuration.

[rf232]

On GCP you can disable the addon
https://cloud.google.com/sdk/gcloud/reference/container/clusters/update
gcloud container clusters update --update-addons=KubernetesDashboard=DISABLED
and then install it yourself.

[derluke]

Thank you both for the swift reply - this indeed seems to work, it does make my cluster creation more tedious though. I need to check with google on this I guess.
On the plus side this gives me a more up-to-date version of the dashboard