Jupyterhub: Authorization form must be sent from authorization page

[elgalu]

Hi! after upgrading to master branch (around 40 days of new commits) admins are no longer able to impersonate users, we get this error right after clicking "Authorize" on the new page:

https://github.com/jupyterhub/jupyterhub/blob/5b4f0d4304b1ea57b4d398609f2777bd93131ef4/jupyterhub/apihandlers/auth.py#L255

403 : Forbidden
Authorization form must be sent from authorization page

We are still using GitHubOAuthenticator and Postgres as backend.

Also other strange thing is happening, the admin page is not properly listing all users, but this is probably related to some of our custom page.html templates:

[elgalu]

Update I disabled all custom templates and the listing users issue persists.

Also, when clicking "Add Users" the new user doesn't appear. It will only appear if the user himself logs into the Hub independently of the admin.

Is this an intended changed behavior?

[elgalu]

Found the problem, is related to being using http and https in different parts of the config settings

[elgalu]

It seems I should switch to c.JupyterHub.internal_ssl = True in order to keep http_method consistency with my external TLS termination.

[hnykda]

Interesting. We should add it to https://github.com/jupyterhub/jupyterhub/blob/master/docs/source/getting-started/security-basics.rst#if-ssl-termination-happens-outside-of-the-hub, no?

[minrk]

I think this means something is wrong in the detection of what URL is being used to make the request. How is SSL termination being implemented? Since all of these requests are made by the client with HTTPS, it shouldn't be relevant whether internal SSL is used. That would only come up if the proxy-protocol is not properly forwarding that the request is made via HTTPS.

Is this perhaps on Amazon? I believe folks have seen incorrect proxy headers coming from Amazon load-balancers. I think we need to provide the ability to override public URLs in cases where the proxy-protocol doesn't properly preserve the request protocol.

[hnykda]

We have seen this on GCP with roughly this

[minrk]

thanks for the pointer. It would be helpful if someone had a reproducible test case where this occurs (I think it's somewhere in the proxy protocol reporting http when the client request is actually https). Dumping all of the headers for the rejected request would be useful, I think.

[hnykda]

So when I try to impersonate user:

I click on authorize and the following HAR (annonymized) request/response 403 is returned:
https://gist.github.com/hnykda/1e7bad208c53971b04077a88bf01f313

[ahkui]

any solution for this ?

[alpine-digger]

Is there some progress or solution? We use jupyterhub on GCP behind proxy which provides SSL connection so jupyterhub does not use internal SSL and it is quite complicated to use internal SSL in a such case.

[vnijs]

This issue seems related to https://github.com/jupyterhub/jupyterhub/issues/2604

[raethlein]

We faced the same issue today and did the following in our JupyterHub container: we have an nginx proxy (openresty version with lua support) started and there we added following lines as a workaround for now to the location / directive:

set $ref "";
access_by_lua_block {
  if (ngx.var.http_referer ~= nil and ngx.var.http_referer ~= "") then
      ngx.var.ref = string.gsub(ngx.var.http_referer, "https", "http")
  end
}

proxy_set_header Referer $ref;

You can see the full nginx.conf here: https://github.com/ml-tooling/ml-hub/blob/develop/docker-res/nginx.conf#L71

The solution comes from as the Hub compares the Referer with an internally-saved url here.
To pass this test, we modify the Referer header by replacing "https" with "http".

[cw6515]

Is there any solution to this? We recently upgraded jupyterhub from 0.7.2 -> 1.0.0 and admins can no longer access user servers. When navigating to a user server, the Authorize Access page appears, followed by:

```403 : Forbidden
Authorization form must be sent from authorization page

We are running the hub behind a proxy/single user servers in docker containers. Having read around, I tried:

c.JupyterHub.internal_ssl = True
c.JupyterHub.allow_named_servers = True
```

Neither of which worked. Are there any other fixes/suggestions?

[limimiking]

We faced the same issue today and did the following in our JupyterHub container: we have an nginx proxy (openresty version with lua support) started and there we added following lines as a workaround for now to the location / directive:

set $ref "";
access_by_lua_block {
  if (ngx.var.http_referer ~= nil and ngx.var.http_referer ~= "") then
      ngx.var.ref = string.gsub(ngx.var.http_referer, "https", "http")
  end
}

proxy_set_header Referer $ref;

You can see the full nginx.conf here: https://github.com/ml-tooling/ml-hub/blob/develop/docker-res/nginx.conf#L71

The solution comes from as the Hub compares the Referer with an internally-saved url here.
To pass this test, we modify the Referer header by replacing "https" with "http".

I also found that the cause of this problem is http referer, but I'm using apache2 instead of nginx.
There's no lua_block in apache2, This is how I solve this problem in apache2:
Note: better solutions see comments by @Carreau, @bitnik and @ianalis
also see https://jupyterhub.readthedocs.io/en/stable/reference/config-proxy.html

RewriteEngine on
<Location /jupyter>
    ProxyPass http://localhost:8000/jupyter
    ProxyPassReverse http://localhost:8000/jupyter

    # important config 
    RewriteCond %{HTTP_REFERER} /jupyter/hub/api/oauth2/ [NC] 
    RewriteRule /jupyter/(.*) - [E=REFERER:%{SERVER_NAME}/jupyter/$1]

    # thanks to @Carreau, fix a mistake here
    RequestHeader set Referer http://%{REFERER}e env=REFERER

    # important config end

    RewriteCond %{HTTP:UPGRADE} WebSocket$ [NC]
    RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
    RewriteRule /jupyter/(.*) ws://localhost:8000/jupyter/$1 [P,L]
</Location>

Rewrite engine can set environment virable by [E=REFERER:%{SERVER_NAME}/jupyter/$1]
RequestHeader is similar to "proxy_set_header" in niginx

[Carreau]

@limimiking solution worked for me.

[Carreau]

Well I spoke too soon, that broke other stuff.

[W 2019-08-28 11:16:29.068 JupyterHub base:60] Blocking Cross Origin API request.  Referer: 
 http://(null), Host: <mydomain>/jupyter/hub/

[Carreau]

I fixed it with

RequestHeader set Referer http://%{REFERER}e env=REFERER

So the RequestHeader is set only if REFERER has been defined by the previous rule.

[bitnik]

From tornado source code (https://github.com/tornadoweb/tornado/blob/v6.0.3/tornado/httputil.py#L314-L318):

The protocol used, either "http" or "https". If HTTPServer.xheaders
is set, will pass along the protocol used by a load balancer if
reported via an X-Scheme header.

So I set the X-Scheme header in my nginx config and this fixed the problem:

    # Managing literal requests to the JupyterHub front end
    location / {
        ...
        proxy_set_header X-Scheme $scheme;
    }

[ianalis]

@limimiking's solution with @Carreau's correction worked for me as well but @bitnik's solution is easier. For apache, I added this single line to the config:

RequestHeader set "X-Scheme" expr=%{REQUEST_SCHEME}

[limimiking]

I fixed it with

RequestHeader set Referer http://%{REFERER}e env=REFERER

So the RequestHeader is set only if REFERER has been defined by the previous rule.

My fault, I also encountered this problem later.
Thank you for your correction, worked for me pretty well.

[ahkui]

@bitnik solution is work for me on my caddy proxy

proxy / jupyterhub {
    header_upstream X-Scheme {scheme}
    transparent
    websocket
    timeout 0
}

[manics]

@yuvipanda this issue was automatically closed by https://github.com/berkeley-dsep-infra/datahub/commit/ae18ef85490dde261981b08d1e8a6b25a4d22dd5
Was that unintentional?

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/authorization-form-must-be-sent-from-authorization-page/3081/1

[consideRatio]

@manics I think it was a mistake, but the same code is now available as part of z2jh 0.9.0-beta.3 though, but that is only related to a z2jh deployment rather than JupyterHub in general. I'm reopening this for now.

[matteoipri]

With latest JupyterHub I also had issues in accessing other users' servers while being an admin.
I use JupyterHub with SwarmSpawner and OAuth authentication.
I have an HAproxy in front of JupyterHub handling TLS termination.
I added this to HAproxy config to fix the scheme mismatch:

http-request add-header X-Scheme https if { ssl_fc }

To note that I already had the following, but it was not enough:

http-request add-header X-Forwarded-Proto https if { ssl_fc }

Now with both headers added, things do work without breaking other pieces.
Thank you all for the tips!

[duongnt]

If you're running istio like we do, just add something like

headers:
  request:
    set:
      "x-scheme": "https"

[zifeo]

Anyone solved that issue using the jupyterhub helm chart jupyterhub/zero-to-jupyterhub-k8s?

[itssimon]

We're also running into this issue with the "Zero to JupyterHub" Helm chart (Kubernetes). SSL is terminated at the AWS load balancer (proxy.https.type: offload).
I don't think any of the above workarounds apply in that scenario?

[consideRatio]

@itssimon what version of the helm chart is used?

[itssimon]

Chart version: jupyterhub-0.9.0
App version: 1.1.0

[stefanvangastel]

Same issue here with:

Chart version: jupyterhub-0.9.0
App version: 1.1.0

Using LDAP auth. Worked with chart 0.8.2

Using NGINX reverse proxy for https

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/use-caddy-as-a-proxy/4452/1

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/use-caddy-as-a-proxy/4452/2

[manics]

@yuvipanda your commit autoclosed this again!

[covrig]

Same problem. Using Traefik as a reverse proxy. Posting just to keep thread open.

[gadiener]

Same here with GCP Loadbalancer

[leosunmo]

Managed to get this working by adding this config to my Gloo (Envoy wrapper) config:

<snip>
    routes:
      - matchers:
          - methods: []
            prefix: /
        options:
          headerManipulation:
            requestHeadersToAdd:
              - header:
                  key: X-Scheme
                  value: https
                append: false
              - header:
                  key: X-Forwarded-Proto
                  value: https
                append: false
<snip>

[0nebody]

@zifeo The issue with jupyterhub/zero-to-jupyterhub-k8s for me is related to the proxy. The proxy adds its own x-forward headers to the ones created by the service offloading SSL. I offload SSL at the load balancer.

x-forwarded-port=443,80
x-forwarded-proto=https,http

Editing the proxy deployment adding --no-x-forward passes the headers to the hub unaltered by the proxy.

[stefanvangastel]

@zifeo The issue with jupyterhub/zero-to-jupyterhub-k8s for me is related to the proxy. The proxy adds its own x-forward headers to the ones created by the service offloading SSL. I offload SSL at the load balancer.

x-forwarded-port=443,80
x-forwarded-proto=https,http

Editing the proxy deployment adding --no-x-forward passes the headers to the hub unaltered by the proxy.

@0nebody and this fixed it for you? (we are also offloading ssl at the NGINX reverse proxy)

[0nebody]

@stefanvangastel Yes, it works for me as the AWS ALB adds the correct headers. Assuming your Nginx configuration sets the x-forwarded headers than it should be the same.

If the proxy doesn't have --no-x-forward set it appends its own x-forwarded-proto to the one added by the ALB for http. I assume this is why people are having success setting x-scheme.

[stefanvangastel]

Hmm. Nginx sets the correct forwarded scheme and proto (https), deployment of proxy modified to use the --no-x-forward flag. Still getting the "Authorization form must be sent from authorization page".

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/gateway-error-with-haproxy-in-front-of-kubernetes/5821/5

[ryanlovett]

@bitnik's tip worked for me when using nginx for ssl termination.

[aneagoe]

@0nebody adding --no-x-forward doesn't work for OKD 4.5 (haproxy underneath), throwing too many redirects with this in unfortunately.
@elgalu did this commit fix it for you? https://github.com/elgalu/jupyterhub/commit/5e687c095542be6799d282ed23a1fa130e0c6f23 If yes, could you submit a PR with it?

[elgalu]

yes, my commit fixed the problem for us but it's a desperate workaround, isn't it?

[aneagoe]

@elgalu I think we're all desperate here... seems to be an issue plaguing us since 2018. In OKD 3.11 I've fixed it by patching the openshift router template as per https://github.com/jupyterhub/jupyterhub/issues/2284#issuecomment-578118761 but that's an even worse approach. I see quite some folks implementing workarounds for this left and right when hopefully it can be fixed here.

[elgalu]

@aneagoe #3219 fixes the problem in my setup, feel free to test it in yours if you have time.