Jupyterhub: Deploying behind a reverse proxy

[danielballan]

Has anyone worked on deploying jupyterhub behind a Reverse Proxy?

I am trying to deploy it on a system that communicates with the open network through an Apache reverse proxy. Authentication works fine, and kernels are created when users open notebooks, but the notebooks cannot connect to their kernels. It looks like requests are coming in, but communication out through the Apache server is not working correctly.

[minrk]

You need to make sure that websocket connections work, which I think is not Apache's default behavior. We've deployed it behind nginx a few times. Here is an example nginx config.

[danielballan]

Thanks. I'm working with our busy network admins on this, and I'll revisit the issue with anything we that might be helpful to others.

[danielballan]

If anyone else is looking to do deploy behind an Apache reverse proxy, here's our intrepid sysadmin's summary of she got this working:

Older versions of Apache such as 2.2.x do not provide WebSocket support, therefore a newer version of Apache (i,e 2.4) included in the upcoming Debian Jessie release, supplies proxy_wstunnel for WebSocket applications. For Jupyterhub, for any URL spaces /(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?, forward to ws(s)://servername:port_number, all other standard spaces, forward to http(s)://servername:port_number, that will do the trick!

[franga2000]

I know I'm a bit late, but if anyone needs it, here's the config that works for me (basically what @danielballan quoted):

ProxyPass / http://localhost:8000/
ProxyPassReverse / http://localhost:8000/

Header edit Origin {external address} localhost:8000
RequestHeader edit Origin {external address} localhost:8000

Header edit Referer {external address} localhost:8000
RequestHeader edit Referer {external address} localhost:8000
<Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
    ProxyPass ws://localhost:8000
    ProxyPassReverse ws://localhost:8000
</Location>

[iocampomx]

Hello franga2000, what means {external address}? What is the value that I need to put in this place?

[franga2000]

@nafiux it's the address that you want to use to access jupyter from outside (like py.example.com)

[chandragaajula]

@nafiux I have the following entry in "/etc/apache2/sites-enabled" on my Ubuntu 15.10

<VirtualHost *:80>
        ServerName "jupyter.xxxxxxxxxxxx.com"
        ProxyPass / http://192.168.254.23:8888/
        ProxyPassReverse / http://192.168.254.23:8888/
        Header edit Origin "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888
        RequestHeader edit Origin "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888

        Header edit Referer "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888
        RequestHeader edit Referer "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888
        <Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
                ProxyPass ws://192.168.254.23:8888
                ProxyPassReverse ws://192.168.254.23:8888
        </Location>
</VirtualHost>

After making the above entry I restarted apache: "sudo service apache2 restart"

When I attempt to run my script the Kernel is unable to connect.

Console Logs:

The same works when I specify the IP Address though:

How do I get this to work with the domain name?

[franga2000]

@chandragaajula are there any errors in the Apache logs? Also, have you enabled mod_proxy_wstunnel?

[chandragaajula]

@franga2000 - Yes, I have enabled mod_proxy_wstunnel:

 chandra@TheMilkyWay:/etc/apache2$ sudo a2enmod
 Your choices are: access_compat actions alias allowmethods asis auth_basic auth_digest auth_form authn_anon authn_core authn_dbd authn_dbm authn_file authn_socache authnz_fcgi authnz_ldap authz_core authz_dbd authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex buffer cache cache_disk cache_socache cgi cgid charset_lite data dav dav_fs dav_lock dbd deflate dialup dir dump_io echo env expires ext_filter file_cache filter headers heartbeat heartmonitor ident include info lbmethod_bybusyness lbmethod_byrequests lbmethod_bytraffic lbmethod_heartbeat ldap log_debug log_forensic lua macro mime mime_magic mpm_event mpm_prefork mpm_worker negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_express proxy_fcgi proxy_fdpass proxy_ftp proxy_html proxy_http proxy_scgi proxy_wstunnel ratelimit reflector remoteip reqtimeout request rewrite sed session session_cookie session_crypto session_dbd setenvif slotmem_plain slotmem_shm socache_dbm socache_memcache socache_shmcb speling ssl status substitute suexec unique_id userdir usertrack vhost_alias xml2enc
 Which module(s) do you want to enable (wildcards ok)?
 proxy_wstunnel
 Considering dependency proxy for proxy_wstunnel:
 Module proxy already enabled
 Module proxy_wstunnel already enabled
 chandra@TheMilkyWay:/etc/apache2$

I don't see any specific errors in logs, will review them again.

[mbmilligan]

The config for websocket proxying quoted by @franga2000 didn't work for me either. What DID work was instead being explicit about the URL matching:

<LocationMatch "/mypath/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
    ProxyPassMatch ws://localhost:8999/mypath/$1/$2$3
    ProxyPassReverse ws://localhost:8999 # this may be superfluous
</LocationMatch>

For what it's worth, this is using Apache 2.2 on CentOS, with the wstunnel backport patches described here: https://gist.github.com/vitkin/6661683

[chandragaajula]

@mbmilligan - Thanks for the link you provided which led me to my solution. In my case, it turns out to be quite simple:

<VirtualHost *:80>
        ServerName "jupyter.xxxxxxxxxxx.com"
        ProxyPreserveHost On
        ProxyRequests off

        ProxyPass /api/kernels/ ws://192.168.254.23:8888/api/kernels/
        ProxyPassReverse /api/kernels/ http://192.168.254.23:8888/api/kernels/

        ProxyPass / http://192.168.254.23:8888/
        ProxyPassReverse / http://192.168.254.23:8888/
</VirtualHost>

Important thing to note: The entry for WebSockets should be in the beginning.

[mcapuccini]

Same problem on Traefik. Anyone managed to configure it using it? A generic explanation of what causes the problem would be really appreciated, so I can try to configure the proxy myself.

[mcapuccini]

Just found the solution. I needed to add the following to the JSON that I send to Marathon:

"labels": {"traefik.frontend.entryPoints":"http,https,ws"}

I guess this will be useful for many people, as Traefik is getting popular.

[lbiemans]

Had the same issue but thanks to @franga2000 this is solved. I had to enable proxy, proxy_http, headers and proxy_wstunnel.

[TheGodEmperor]

I notice this is old, but I've been trying to figure out how to have a server proxy to the server running JupyterHub.

Basically, one server is test-session that runs Jupyterhub and the other is the test-dev which will need to run so that if one did test-dev/notebook/ it'd link to the test-session:8888/notebooks/ link.

I've got a proxy for test-dev to test-session for other things that works (like Python scripts), but for some reason the proxy for the Jupyterhub isn't working. Wouldn't the proxy be, on the test-dev server, basically?

ProxyPass /notebooks/ http://lle-test-session:8888/
ProxyPassReverse /notebooks/ http://lle-test-session:8888/

Am I missing something? I'm not much of a server administrative guy so I'm still in the process of learning.

[ryanlovett]

Just adding another _works for me_ example. Apache 2.4.7 in Ubuntu 14.04 suffers from a websocket bug that prevents notebook sessions from working properly in this context.

ProxyPreserveHost On
ProxyRequests off

ProxyPass /api/kernels/ ws://127.0.0.1:8888/api/kernels/
ProxyPassReverse /api/kernels/ https://127.0.0.1:8888/api/kernels/

ProxyPass / https://127.0.0.1:8888/
ProxyPassReverse / https://127.0.0.1:8888/

<Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
    ProxyPass ws://localhost:8888
    ProxyPassReverse ws://localhost:8888
</Location>

[mbmilligan]

@TheGodEmperor You should crib from one of the working examples in this thread. You definitely do need to proxy the main paths and the websockets separately. A couple more notes:

1. you spell the url "/notebook" in one place and "/notebooks" in another, check for misspellings in your config;
2. the Hub may need to be configured to listen to an external interface rather than localhost;
3. recent versions will complain if started without SSL, so if you are terminating SSL at your proxy server (which is the common thing to do here) you may need to add the --no-ssl option

[lbiemans]

What works for me is:

    ProxyPass / http://localhost:8000/
    ProxyPassReverse / http://localhost:8000/

    Header edit Origin HOSTNAME.TLD localhost:8000
    RequestHeader edit Origin HOSTNAME.TLD localhost:8000

    Header edit Referer HOSTNAME.TLD localhost:8000
    RequestHeader edit Referer HOSTNAME.TLD localhost:8000
    <Location ~ "/(user/[^/]+)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
            ProxyPass ws://localhost:8000
            ProxyPassReverse ws://localhost:8000
    </Location>

Don't forget to enable proxy, proxy_http, headers and proxy_wstunnel.

[lordloh]

I am getting a "Blocking Cross Origin API request".

[W 2016-10-04 13:53:03.855 bharath handlers:288] Blocking Cross Origin API request.  Origin: https://192.168.144.3:8000, Host: jptr.mydomain.com
[W 2016-10-04 13:53:03.856 bharath log:47] 404 POST /user/bharath/api/contents (129.118.26.12) 1.39ms referer=https://192.168.144.3:8000/user/bharath/tree

what do I need to change in my jupyterhub config file?

[lordloh]

What worked for me was -

<VirtualHost *:443>
        ServerName jptr.example.com
        SSLEngine On
        SSLProxyEngine On
        ProxyPreserveHost On
        DocumentRoot /var/www/html

        <Location ~ "/(user/[^/]+)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
                ProxyPass ws://192.168.144.3:8000
                ProxyPassReverse ws://192.168.144.3:8000
        </Location>

        ProxyPass / http://192.168.144.3:8000/
        ProxyPassReverse / http://192.168.144.3:8000/

        SSLCertificateFile /etc/letsencrypt/live/jptr.example.com/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/jptr.example.com/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/jptr.example.com/chain.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

The websocket part had to come before the the http part.

[slacmshankar]

One more Apache configuration as an example; I'm using an existing httpd-2.4.6-40 instance and placing this within /jupyterhub, i.e c.JupyterHub.base_url = '/jupyterhub' The order of the Location and LocationMatch seemed to be crucial; otherwise, I'd get a various websocket related failures (HTTP 400's) with sessions/kernels etc. Note this is the opposite of what others had experienced. So if you see websocket failures, try changing the order of your Location directives.

  ProxyRequests off
  ProxyPreserveHost On

  <Location /jupyterhub>
    ProxyPass http://jphub:8000/jupyterhub
    ProxyPassReverse http://jphub:8000/jupyterhub
  </Location>

  <LocationMatch "/jupyterhub/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
    ProxyPassMatch ws://jphub:8000/jupyterhub/$1/$2$3
    ProxyPassReverse ws://jphub:8000/jupyterhub/$1/$2$3
  </LocationMatch>

[ianabc]

@slacmshankar that _almost_ works for me. I get the initial connection but whenever I try to restart a running kernel it fails and I see connection failed messages in the console.

kernel.js:453 Starting WebSockets: wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33e1-4d14-bbf9-737798456261
VM681:35 WebSocket connection to 'wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33…d14-bbf9-737798456261/channels?session_id=1BB7A309853142AFB8843CF626F67EE0' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1WrappedWebSocket @ VM681:35WrappedWebSocket @ VM713:35Kernel.start_channels @ kernel.js:455Kernel.reconnect @ kernel.js:349i @ jquery.min.js:4
kernel.js:100 Kernel: kernel_disconnected (3538a47c-33e1-4d14-bbf9-737798456261)
kernel.js:538 WebSocket connection failed:  wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33e1-4d14-bbf9-737798456261 true
kernel.js:556 Connection lost, reconnecting in 2 seconds.
kernel.js:100 Kernel: kernel_connected (3538a47c-33e1-4d14-bbf9-737798456261)
manager.js:112 Widget frontend version does not match the backend.
kernel.js:100 Kernel: kernel_ready (3538a47c-33e1-4d14-bbf9-737798456261)
kernel.js:100 Kernel: kernel_reconnecting (3538a47c-33e1-4d14-bbf9-737798456261)
kernel.js:453 Starting WebSockets: wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33e1-4d14-bbf9-7377984

...

Could you try restarting a running kernel on your instance and see if you get the same behaviour?

Edit: This is only happening to me in Chrome, Firefox seems to handle the restart just fine!

[slacmshankar]

Apologies; noob here. How do I "restart a running kernel on your instance"?

[ianabc]

Hi @slacmshankar me too! I just mean select Kernel->Restart from the menu system (screenshot attached).

When I do that from Chrome it doesn't reconnect to the running kernel and I get a bunch of error messages in the console.

I think I've got the problem narrowed down to something in the way that the websockets are handled in kernel.js (jupyter/notebook) rather than jupyterhub but it would be really helpful to know if you are seeing the same behaviour. I don't see any problem with Firefox.

[slacmshankar]

Aah; confirmed! On Google chrome, I get the same behavior
WebSocket connection to '...' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1

Works just fine in Firefox; no problems there.

[ianabc]

Thanks @slacmshankar I've opened another issue at https://github.com/jupyter/notebook/issues/1835 I think that might be where the root of the problem is (notebook) and this issue is closed. I'll try to play with it this weekend and see if I can make any progress.

[remidebette]

Dear everyone,

Thank you for the usefull discussion.
The .conf file of @slacmshankar made the jupyterhub work for me.

But after I upgraded to jupyterhub 0.7 then suddently apache does not redirect.
"Not Found

The requested URL /jupyterhub was not found on this server."

Would you have any idea of how to update the .conf file?

Thank you

[remidebette]

Dear everyone,

To be more specific:
With the following apache conf (all other things unchanged after the upgrade to 0.7)

ProxyPreserveHost On
ProxyRequests off
#SSLProxyEngine On

<Location /jupyter>
        ProxyPass http://localhost:8000/jupyter
        ProxyPassReverse http://localhost:8000/jupyter
</Location>

<LocationMatch "/jupyter/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
        ProxyPassMatch ws://localhost:8000/jupyter/$1/$2$3
        ProxyPassReverse ws://localhost:8000/jupyter/$1/$2$3
</LocationMatch>

I get the following when calling http://localhost/jupyter

https://debette.noip.me/jupyter/hub/jupyter 404 not found

and no error if I call http://localhost/jupyter/ (with the slash)

On the other hand if I add slashes:

ProxyPreserveHost On
ProxyRequests off
#SSLProxyEngine On

<Location /jupyter/>
        ProxyPass http://localhost:8000/jupyter/
        ProxyPassReverse http://localhost:8000/jupyter/
</Location>


<LocationMatch "/jupyter/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
        ProxyPassMatch ws://localhost:8000/jupyter/$1/$2$3
        ProxyPassReverse ws://localhost:8000/jupyter/$1/$2$3
</LocationMatch>

Then http://localhost/jupyter
gives me:

Not Found

The requested URL /jupyter was not found on this server.

With the slash it also works fine.

All my other proxied servers work fine with the same kind of configuration, with or without slash, and once again this configuration worked just fine before the upgrade to Juptyerhub 0.7
Should I raise a bug?

Remi

[minrk]

@remidebette can you include your JupyterHub configuration? I suspect a base_url is getting duplicated (hence /jupyter/hub/jupyter/), but it's not quite obvious where this is happening.

[goulou]

In my setup, none of the above solutions seemed to work... This might be due to the fact that Apache runs in SSL mode.
However, the following solution works perfectly, and seems cleaner to me :

  ProxyPreserveHost On
  RewriteEngine On
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*)           ws://jupyter_host:8888/$1 [P,L]
  RewriteCond %{HTTP:Upgrade} !=websocket [NC]
  RewriteRule /(.*)           http://jupyter_host:8888/$1 [P,L]

It uses HTTP headers instead of the URL, so should be more generic.

Setup details :
OS : debian Jessie (8)
Apache version : Apache/2.4.10 (Debian)
The version of the notebook server is 4.3.0 and is running on:
Python 3.5.2+ (default, Nov 22 2016, 01:00:20)
[GCC 6.2.1 20161119]

I hope this will help!

[remidebette]

@minrk Sorry about the long delay.

Here are the uncommented parts of the configuration file I use:
(see my former comment for the apache reverse proxy code)

c.JupyterHub.base_url = '/jupyter'
c.JupyterHub.db_url = 'mysql+pymysql://[localconf]'
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'

Others are just user and admin configuration
Jupyter 0.7, apache 2.4.18, php 7.0.13-0ubuntu0.16.10.1, Ubuntu 16.10

Remi

[remidebette]

@goulou Your solution works fine for me.

But this solution requires all the subpaths to be redirected to /jupyter/hub/...
In my configuration I still want to use several applications on the same host, based on the subpaths.
Which brings us back to this error since jupyterhub 0.7

[rgbkrk]

This version works well for the first connection:

  ProxyPreserveHost On
  RewriteEngine On
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*)           ws://jupyter_host:8888/$1 [P,L]
  RewriteCond %{HTTP:Upgrade} !=websocket [NC]
  RewriteRule /(.*)           http://jupyter_host:8888/$1 [P,L]

However, any reconnect or restart attempt ends up with new errors. They happen on close of the original websocket.

[rgbkrk]

Of note, we were experiencing this with Apache 2.4.7 (current Trusty Ubuntu stable)

[rgbkrk]

I can reproduce this really quickly with an Apache proxy fronted notebook by closing the current kernel:

Jupyter.notebook.kernel.ws.close()

The WebSocket.close function is a native function and I can't seem to catch the WebSocket connection to '...' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1 error.

[ianabc]

I'm on apache Apache/2.4.6 (CentOS) and I'm using something like

ProxyPass http://127.0.0.1:8000/jupyter
ProxyPassReverse http://127.0.0.1:8000/jupyter
ProxyPreserveHost on
<LocationMatch "/jupyter/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
    ProxyPassMatch ws://127.0.0.1:8000/jupyter/$1/$2$3
    ProxyPassReverse ws://127.0.0.1:8000/jupyter/$1/$2$3
</LocationMatch>

I see the same behaviour closing websockets but only via the notebook restart action. Selecting kernel->restart leaves my browser in a loop trying to reconnect the kernel as described in (https://github.com/jupyter/notebook/issues/1835).

If I do Jupyter.notebook.kernel.ws.close() in chrome's console I see one instance of the error message you get above, but then I do actually get reconnected without any problem

WebSocket connection to 'wss://ubc.syzygy.ca/jupyter/user/ianatr/api/kernels/a94f074d-c534-4653-8284-22309b590960/channels?session_id=D872FB9D684D451B8802474890B9EBE4' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1
WrappedWebSocket @ VM297:35
WrappedWebSocket @ VM328:35
Kernel.start_channels @ kernel.js:455
Kernel.reconnect @ kernel.js:349
i @ jquery.min.js:4
kernel.js:100 Kernel: kernel_disconnected (a94f074d-c534-4653-8284-22309b590960)
kernel.js:538 WebSocket connection failed:  wss://ubc.syzygy.ca/jupyter/user/ianatr/api/kernels/a94f074d-c534-4653-8284-22309b590960 true
kernel.js:556 Connection lost, reconnecting in 1 seconds.
kernel.js:100 Kernel: kernel_reconnecting (a94f074d-c534-4653-8284-22309b590960)
kernel.js:453 Starting WebSockets: wss://ubc.syzygy.ca/jupyter/user/ianatr/api/kernels/a94f074d-c534-4653-8284-22309b590960
kernel.js:100 Kernel: kernel_connected (a94f074d-c534-4653-8284-22309b590960)
manager-base.js:195 Widget backend and frontend versions are compatible
kernel.js:100 Kernel: kernel_ready (a94f074d-c534-4653-8284-22309b590960)

[Geremia]

If I do Jupyter.notebook.kernel.ws.close() in chrome's console I see one instance of the error message you get above, but then I do actually get reconnected without any problem

Unfortunately, that doesn't solve the issue for me in Firefox or Chromium. I use a similar configuration you did but with Apache 2.4.25.

Update: I was able to fix the issue by adding RequestHeader set Origin "http://localhost:8888" to my webserver config (fix courtesy: this answer).

[ianabc]

I have this problem on a bunch of hubs now, I haven't really figured out how to debug it. I've tried making local modifications to kernel.js within Chrome and saving them, but they never seem to be applied. The most likely thing I've found is this websocket-sharp issue which talks about problems with Chrome's handling of compression. If I can figure out how, I'll try disabling permessage-delate.

Here is a summary of the behaviour in my case:

• Reloading a notebook works fine in Firefox and Chrome, the new websocket connection is created no problem
• Using the reconnect action in Firefox is fine, the socket gets reconnected
• Manually closing the connection in chrome (Jupyter.notebook.kernel.ws.close()) works fine, everything gets reconnected OK.
• Using the reconnect action in chrome triggers the loop. I get an loop of Connection failed notices from the notebook, and a "failed: Invalid frame header" for the websocket in the console.

[kckern]

Here's the configuration that finally worked for me (Apache/2.4.18 (Ubuntu))

<VirtualHost *:80>
    ServerName jupyter.example.com
    <Location />
        RequestHeader unset Accept-Encoding
        ProxyPass        http://127.0.0.1:8888/
        ProxyPassReverse http://127.0.0.1:8888/
        ProxyPreserveHost on
        Order allow,deny
        Allow from all
    </Location>
    <Location /api/kernels/>
        ProxyPass        ws://127.0.0.1:8888/api/kernels/
        ProxyPassReverse ws://127.0.0.1:8888/api/kernels/
    </Location>
</VirtualHost>

[tfabbri]

Here's my configuration that worked on my VPS:

<VirtualHost *:80>
    ServerName ipynb.example.ovh
    ServerAdmin [email protected]
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    ProxyPreserveHost On
    ProxyRequests off

    <Location /api/kernels/>
        ProxyPass        ws://example.ovh:9999/api/kernels/
        ProxyPassReverse ws://example.ovh:9999/api/kernels/
    </Location>

    ProxyPass       /  http://example.ovh:9999/
    ProxyPassReverse / http://example.ovh:9999/
    RequestHeader set Origin "http://example.ovh:9999"
    Redirect permanent / http://ipynb.example.ovh
</VirtualHost>

[gerroon]

@tfabbri

what is the ip here Redirect permanent / http://ipynb.example.ovh

[pkumar8789]

Went through issue #219 and related links but no success till yet.

I have a reverse proxy setup on Jupyterhub using Apache 2.4. The redirection to JupyterHUB url from Apache URL works fine but the internal Websocket connection after Kernel is created fails as it takes the Port from the Apache URL and not the redirected Port.

Apache URL: https://HOST_NAME:31000

JupyterHUB redirect URL: https://HOST_NAME:8000

but websocket connection is attempted at wss://HOST_NAME:31000 which fails. The URL redirection doesn't happen for websocket.

Apache Conf:

Listen HOST_NAME:31000

ServerName HOST_NAME

RewriteEngine On
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule /(.*) ws://HOST_NAME:8000/$1 [P,L]

ProxyPreserveHost on
# proxy to JupyterHub
ProxyPass https://HOST_NAME:8000/
ProxyPassReverse ws://HOST_NAME:8000/

Note: HOST_NAME is the server on which JupyterHub is running. Apache is also running on the same server.

While trying to open Notebook session, I see Kernel Created.

Starting websockets: wss://HOST_NAME:31000/user/d12333/api/kernels/d34343/channels?session...failing to connect. if the URL re-direction for websocket uses 8000 port then this should work.

But not able to configure the redirect through apache cnfiguration.

proxy_wstunnel_module, proxy_http_module and proxy_module are all enabled.

[lbiemans]

Checked one of our configs and found some differences like:

Listen HOST_NAME:31000 ==> We don't use it

Below a snippet from the websocket config:

    RewriteEngine On
    RewriteCond %{HTTP:Connection} Upgrade [NC]
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteRule /(.*) ws://HOST_NAME:8000/$1 [P,L]

    <Location "/">
            ProxyPreserveHost on
            ProxyPassReverse  http://HOST_NAME:8000/
            ProxyPass         http://HOST_NAME:8000/
    </Location>

[pkumar8789]

Checked one of our configs and found some differences like:

Listen HOST_NAME:31000 ==> We don't use it

Below a snippet from the websocket config:

    RewriteEngine On
    RewriteCond %{HTTP:Connection} Upgrade [NC]
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteRule /(.*) ws://HOST_NAME:8000/$1 [P,L]

    <Location "/">
            ProxyPreserveHost on
            ProxyPassReverse  http://HOST_NAME:8000/
            ProxyPass         http://HOST_NAME:8000/
    </Location>

The reason I used Listen host_name:31000 is because Apache URL is setup at that port and I want all incoming calls to be redirected to JupyterHUB URL host_name:8000.

[lbiemans]

Okay makes sense :-).

Have you tried the instead of ?

What does the apache debug and jupyterhub log tell you?

[pkumar8789]

Okay makes sense :-).

Have you tried the instead of ?

What does the apache debug and jupyterhub log tell you?

Tried the tag earlier but later changed to regex as per few suggestions for fixing the issue.
Apache logs shows no anomalies. JupyterHUB console logs shows wss connection failed as I have attached in my original post.
Let me also check logs on JupyterHUB server.
Thanks for helping me out here. Really bugged by this issue :)

[alelorca]

I have been reviewing all your comments (thanks a lot!) and made some tests in order to have 2 jupyterhub configurations (dev and ops) under the same apache virtual host, which makes the things a bit trickier. Here I go with the config that works better for me (apache 2.4.6, Centos7).

ProxyPreserveHost on
RewriteEngine on

# OPS CONFIG
#RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule ^/ops/(.*) ws://ops.example.com:8000/ops/$1 [P,L]
<Location /ops/>
        ProxyPass http://ops.example.com:8000/ops/
        ProxyPassReverse http://ops.example.com:8000/ops/
        ProxyPassReverseCookiePath / /ops/
</Location>

# DEV CONFIG
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule ^/dev/(.*) ws://dev.example.com:8000/dev/$1 [P,L]
<Location /dev/>
        ProxyPass http://dev.example.com:8000/dev/
        ProxyPassReverse http://dev.example.com:8000/dev/
        ProxyPassReverseCookiePath / /dev/
</Location>

Notes:

1. I prefer deploying the jupyterhub under a non-root path on the backend server. This usually makes cleaner the folder to folder reverse proxy mapping: /dev/ -> /dev/ and /ops/ -> /ops/
2. The RewriteCond %{HTTP:Connection} upgrade [NC] is redundant since all the headers I need to test for are the Upgrade ones (at least tested with Firefox, Chrome and Safari).
3. After the first RewriteRule applied for ops, being of type P (proxy) it has implicit the L (last) which I write nevertheless to remember. It does not apply any further rule. There are improved ways to replicate that for ops, but I find it easier to write both exactly in the same way before the location for /dev/, so that our IT team could replicate that with copy/paste without much hesitation.
4. The ProxyPassReverseCookiePath directive is needed since the
   session cookie jupyterhub-session-id takes "/" as default path (perhaps this can be solved on the hub configuration).

[abratchik]

I have been reviewing all your comments (thanks a lot!) and made some tests in order to have 2 jupyterhub configurations (dev and ops) under the same apache virtual host, which makes the things a bit trickier. Here I go with the config that works better for me (apache 2.4.6, Centos7).

ProxyPreserveHost on
RewriteEngine on

# OPS CONFIG
#RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule ^/ops/(.*) ws://ops.example.com:8000/ops/$1 [P,L]
<Location /ops/>
        ProxyPass http://ops.example.com:8000/ops/
        ProxyPassReverse http://ops.example.com:8000/ops/
        ProxyPassReverseCookiePath / /ops/
</Location>

# DEV CONFIG
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule ^/dev/(.*) ws://dev.example.com:8000/dev/$1 [P,L]
<Location /dev/>
        ProxyPass http://dev.example.com:8000/dev/
        ProxyPassReverse http://dev.example.com:8000/dev/
        ProxyPassReverseCookiePath / /dev/
</Location>

Notes:

1. I prefer deploying the jupyterhub under a non-root path on the backend server. This usually makes cleaner the folder to folder reverse proxy mapping: `/dev/ -> /dev/` and `/ops/ -> /ops/`

2. The `RewriteCond %{HTTP:Connection} upgrade [NC]` is redundant since all the headers I need to test for are the Upgrade ones (at least tested with Firefox, Chrome and Safari).

3. After the first RewriteRule applied for ops, being of type P (proxy) it has implicit the L (last) which I write nevertheless to remember. It does not apply any further rule. There are improved ways to replicate that for ops, but I find it easier to write both exactly in the same way before the location for /dev/, so that our IT team could replicate that with copy/paste without much hesitation.

4. The `ProxyPassReverseCookiePath` directive is needed since the
   session cookie jupyterhub-session-id takes `"/"` as default path (perhaps this can be solved on the hub configuration).

@alelorca tried your config - did not work for me. However I managed to solve the issue by slight modification:

```

OPS CONFIG

RewriteCond %{HTTP:Connection} upgrade [NC]

RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule \/ops\/(.*) ws://ops.example.com:8000/ops/$1 [P,L]

ProxyPass http://ops.example.com:8000/ops/
ProxyPassReverse http://ops.example.com:8000/ops/
ProxyPassReverseCookiePath / /ops/

```

[cmandeiro]

The issue is that there are routes that are http and others that are ws sharing the same bases (honestly ?)

You need to check in your console which ones are failing and setup the correct proxy for them using regexp as you'll have no other choices

Below is the solution that worked for me, the restart issue was a tricky one sharing the base url /api/kernel but being http rather than ws

your also need to add a proxy to have the terminal working properly

TLDR: /api/kernel/X --> ws proxy and /api/kernel/X/restart or interrupt -> http proxy + terminal proxy

        ProxyPreserveHost On
        SSLProxyEngine On

            <Location /api/kernels/>
                ProxyPass        ws://jupyter-lab:8888/api/kernels/
                ProxyPassReverse ws://jupyter-lab:8888/api/kernels/
            </Location>

            <LocationMatch "/api/(kernels/[^/]*)/(restart|interrupt)(.*)">
                ProxyPass        http://jupyter-lab:8888/api/$1/$2$3
                ProxyPassReverse http://jupyter-lab:8888/api/$1/$2$3
            </LocationMatch>


            <Location /terminals/websocket/>
                ProxyPass        ws://jupyter-lab:8888/terminals/websocket/
                ProxyPassReverse ws://jupyter-lab:8888/terminals/websocket/
            </Location>




        ProxyPass         "/" "http://jupyter-lab:8888/"
        ProxyPassReverse  "/" "http://jupyter-lab:8888/"

[creatica-soft]

I think in general if using token authentication in jupyterlab behind a reverse proxy that uses Basic authentication, jupyterlab overrides the request header "Authorization: Basic " with "Authorization: token " and the reverse proxy fails with 401 because it does not find "Authorization: Basic ..." header.

The workaround is to use password authentication with jupyterlab, which will set a cookie upon successful password verification and won't interfere with "Authorization: Basic" request header destined for the reverse proxy.

Though allow a user to change the password at the first login with a token will lead to the same problem: 401 response from the proxy - so users are forced to login with a pre-defined password, which is far from ideal.