Zero-to-jupyterhub-k8s: Proxy redirects to unexpected port

[ryanlovett]

With v0.7-578b3a2 and

https:
  hosts:
    - "my.hostname.edu"
  type: manual
  manual:
    key: |
      ...
    cert: |
      ...

Connecting to the proxy on port 80 was redirecting me to 8443:

--2018-07-31 16:02:50--  http://aa.bb.cc.dd/
Connecting to aa.bb.cc.dd:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://aa.bb.cc.dd:8443/ [following]
--2018-07-31 16:02:50--  https://aa.bb.cc.dd:8443/
Connecting to aa.bb.cc.dd:8443... ^C

This would then hang since I only had 80/443 open in my firewall.

[choldgraf]

wasn't there also a recent issue about how the notebook server stopped using localhost? could this be related to the same thing?

[betatim]

Maybe @yuvipanda can chime in on this as he was the last person to touch that line

[gytlinux]

I also have the same problem, as long as I enable my certificate and enter my domain name access, it will automatically jump to port 8443, resulting in inaccessibility, unless directly typing https://my-domain-name access is successful.

[dkipping]

Same problem here. As far as I can see adding a field for overriding the targetPort might help. But I am not well informed enough on the configuration and setup of the ports to judge which defaults in the service.yaml make sense.

Tried to manually alter the 8443 port assignment on the proxy deployment as well as on the service (via kubectl edit deployment proxy -n jhub or kubectl edit svc proxy-public -n jhub respectively) but that just breaks accessing the url completely

[clancychilds]

I've had this problem and have had a bit of a poke around to see what is going on.
When manual HTTPS is used, you can see that in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/jupyterhub/templates/proxy/deployment.yaml it sets the redirect_port on the configurable-http-proxy:
command: - configurable-http-proxy - --ip=0.0.0.0 - --api-ip=0.0.0.0 - --api-port=8001 - --default-target=http://$(HUB_SERVICE_HOST):$(HUB_SERVICE_PORT) - --error-target=http://$(HUB_SERVICE_HOST):$(HUB_SERVICE_PORT)/hub/error {{- if $manualHTTPS }} - --port=8443 - --redirect-port=8000 - --ssl-key=/etc/chp/tls/tls.key - --ssl-cert=/etc/chp/tls/tls.crt

The problem here is that in https://github.com/jupyterhub/configurable-http-proxy/blob/master/bin/configurable-http-proxy#L348 if someone attempts to connect to that port, they are 301'd to the listen port, which is 8443 internally, but usually NAT'd to that by the loadbalancer from 443.

The easiest work around is to add an 8443->8443 in your running proxy-public port forwarding, but that is probably not ideal.
To properly fix this, I think it would require adding an option in CHP to allow a "redirect to" option so that users hitting port 8000 aren't just by default redirected to the listen port, but can be redirected to another port (i.e. 443). Then this option needs to be implemented into the helm chart here and use that image.

[clancychilds]

Ah, it looks like https://github.com/jupyterhub/configurable-http-proxy/issues/120 is trying to address this problem?

[clancychilds]

I have submitted a PR on the configurable-http-proxy that, once in place, could be used to address this problem: https://github.com/jupyterhub/configurable-http-proxy/pull/187

[clancychilds]

That PR has been merged and pushed to docker hub. I will find some time to update this helm chart to use that feature.

[jcatana]

Fwiw, in the helm chart using type: secret creates the same issue.

[chicocvenancio]

1341 fixes this. In the meantime anyone facing this can manually edit the proxy deployment and add - --redirect-to=443 to the chp command. Note this needs chp to be version 4.1.0 or higher.

[Nop0x]

@chicocvenancio Thanks for the hint. Could you maybe point into the direction where exactly that parameter needs to be added? Deploying over the 0.8.2 helm chart.

[sunmakrules]

@chicocvenancio @clancychilds I have been tracking this issue and have been on every link possible to find a solution or a workaround but can't figure it out yet.

Tried editing the deployment and changing the ports kubectl edit deployment proxy -n jhub and kubectl edit svc proxy-public -n jhub.
Also tried adding a command in the values file to change redirection to port 443 command: "configurable-http-proxy --redirect-port 80 --redirect-to 443" which doesn't work either.

If the issue is fixed and the PR is merged why don't we have a new tag for it? TIA

[consideRatio]

@sunmakrules we don't have a quick flow of releases. I'm working to make it more fluent as we speak, and I actually made a tagged release yesterday that you can use if you want, named 0.9.0-alpha.1

[sunmakrules]

@consideRatio Thanks for your response. Can we atleast document or just put down here a working workaround for this issue?

[consideRatio]

For now, the suitable workaround is to use 0.9-1743b85 @sunmakrules.

(https://jupyterhub.github.io/helm-chart/)

[sunmakrules]

@consideRatio Thanks. But I still have an issue accessing the proxy since it gives me an error saying

jupyter.example.com redirected you too many times

[consideRatio]

This could happen for various reasons, and if you are using a fresh version of the helm chart, then the issue may not be this specific issue causing that.

Perhaps you could open a new issue about that? If you do, please provide information about your config.yaml with sensitive stuff redacted, the jupyterhub helm chart version used, notes about kubernetes is setup, and perhaps the most relevant part may be notes about what kind of authenticator you are using and how it is setup.

Also information from the hub pod logs and proxy pod logs after a redirect bounce event would be relevant I bet.

[sunmakrules]

@consideRatio Thanks for your help! Have created a new issue as per your suggestion. #1466