BUG REPORT
Version of Helm and Kubernetes:
2.6.1
Which chart:
kubernetes-dashboard
What happened:
Unable to use nginx-ingress
What you expected to happen:
Nginx-Ingress is unable to route to https pod of dashboard.
I should be able to access the dashboard via aws LB
How to reproduce it (as minimally and precisely as possible):
Spawn a kubernetes dashboard using helm. Spawn nginx-ingress. Hit the endpoint via aws loadbalancer. We are getting a 502
Anything else we need to know:
I also had issues getting the kubernetes-dashboard chart working in a similar environment that uses Traefik as an ingress controller. To get it running on http as an internal service:
values.yaml file:extraArgs:
- --enable-insecure-login
- --port=8444 # By default, https uses 8443 so we move it away to something else
- --insecure-port=8443 # The chart has 8443 hard coded as a containerPort in the deployment spec so we must use this internally for the http service
- --insecure-bind-address=0.0.0.0
# Not actually necessary, but makes it more intuitive that this is now just a http service (this `externalPort` is used by the service and ingress resources to expose the dashboard)
service:
externalPort: 80
--auto-generate-certificates from spec.template.spec.containers[0].args (this is again hard coded in the chart and it seems that the dashboard application automatically goes into HTTPS mode if this argument is passed to it)spec.template.spec.containers[0].livenessProbe.httpGet.scheme to HTTP instead of HTTPS (the pod will keep restarting due to failing the liveness check otherwise)Hopefully with the above instructions you can get the dashboard running as an http service, and have it exposed via your own internal ingress setup. Unfortunately this is a bit of a faff and ideally the stable chart would have this customisation built in - if I get some time I will try and submit a PR that fixes the hardcoded issues, but in the meantime hopefully this is a sufficient workaround/some fairly direct steps if someone wants to beat me to fixing the chart.
@MattLaw0 Thanks for the steps. Running the same setup, running into the same issues.
Deployed all this 7 months or so ago with same setup 0 issues.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
This issue is being automatically closed due to inactivity.
I had the same issue with dashboard and nginx-ingress. The nginx-ingress-controller pods had the following logs:
2018/09/27 15:51:21 [error] 816#816: *1873 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 1.2.3.4, server: dashboard.custom-domain.com, request: "GET / HTTP/2.0", upstream: "http://100.96.1.2:8443/", host: "dashboard.custom-domain.com"
2018/09/27 15:51:21 [error] 816#816: *1873 recv() failed (104: Connection reset by peer) while reading upstream, client: 1.2.3.4, server: dashboard.custom-domain.com, request: "GET / HTTP/2.0", upstream: "http://100.96.1.2:8443/", host: "dashboard.k8s-stg.getaws.arubanetworks.com"
1.2.3.4 - [1.2.3.4] - - [27/Sep/2018:15:51:21 +0000] "GET / HTTP/2.0" 009 7 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" 30 0.001 [kube-system-kubernetes-dashboard-443] 100.96.1.2:8443 7 0.000 200 75c8185f0cde08796abc524da904b471
Note the upstream: "http://100.96.1.2:8443/" line where it's trying to talk to the upstream dashboard over HTTP. I followed @MattLaw0 exact steps to run the dashboard over HTTP and it worked perfectly for me. However, a less invasive solution is to keep running dashboard over HTTPS and then configuring the ingress to communicate with it over HTTPS instead of HTTP. With nginx-ingress v18+, you can simply add an annotation to support this: nginx.ingress.kubernetes.io/backend-protocol: "HTTPS". No other changes on dashboard are necessary.
This is what my kubernetes-dashboard's values.yaml looks like:
ingress:
enabled: true
annotations:
聽 kubernetes.io/ingress.class: nginx
聽 kubernetes.io/tls-acme: "true"
聽 ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
hosts:
- $RELEASE_HOSTNAME
tls:
- secretName: $RELEASE_NAME-tls
hosts:
- $RELEASE_HOSTNAME
I think there should be an HTTP version of Kubernetes-dashboard, what you @ravishivt suggested work only if other services are also handling https which is a huge bummer as the whole point of ingress is to take off that logic from downstream services
Ah, It's possible to run it on just http with this file https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/alternative.yaml but not with helm :/
Most helpful comment
I had the same issue with dashboard and
nginx-ingress. The nginx-ingress-controller pods had the following logs:Note the
upstream: "http://100.96.1.2:8443/"line where it's trying to talk to the upstream dashboard over HTTP. I followed @MattLaw0 exact steps to run the dashboard over HTTP and it worked perfectly for me. However, a less invasive solution is to keep running dashboard over HTTPS and then configuring the ingress to communicate with it over HTTPS instead of HTTP. With nginx-ingress v18+, you can simply add an annotation to support this:nginx.ingress.kubernetes.io/backend-protocol: "HTTPS". No other changes on dashboard are necessary.This is what my kubernetes-dashboard's values.yaml looks like: