Jupyterhub: Jupyterhub connecting to a remote server without https

[matthdan]

Hi all,

I'am currently set up a Jupyterhub server with an interface to SLURM through the batch spawner.

I am using the last stable Jupyterhub :

# jupyterhub  --version
0.8.1

The master is correctly working with SSL.
I set up the ssl crt and key correctly in the configuration :

c.JupyterHub.ssl_cert = u'/etc/pki/tls/certs/' + socket.gethostname() + '.crt'
c.JupyterHub.ssl_key = u'/etc/pki/tls/private/' + socket.gethostname() + '.key'

On the remote server the SSL configuration it quite similar :

# cat /etc/jupyter/jupyter_notebook_config.py 
c.NotebookApp.certfile = u'/etc/pki/tls/certs/' + socket.gethostname() + '.crt'
c.NotebookApp.keyfile = u'/etc/pki/tls/private/' + socket.gethostname() + '.key'

When I launch a notebook server I see all these logs :
On the Hub Master server :

[I 2018-05-29 17:07:28.052 JupyterHub batchspawner:192] Job submitted. cmd: /usr/bin/sudo -E -u toto sbatch --parsable output: 33254141
[D 2018-05-29 17:07:28.053 JupyterHub batchspawner:214] Spawner querying job: /usr/bin/sudo -E -u toto squeue -h -j 33254141 -o '%T %B'
[D 2018-05-29 17:07:28.071 JupyterHub batchspawner:316] Job 33254141 still pending
[D 2018-05-29 17:07:28.574 JupyterHub batchspawner:214] Spawner querying job: /usr/bin/sudo -E -u toto squeue -h -j 33254141 -o '%T %B'
[D 2018-05-29 17:07:28.591 JupyterHub batchspawner:316] Job 33254141 still pending
[D 2018-05-29 17:07:29.093 JupyterHub batchspawner:214] Spawner querying job: /usr/bin/sudo -E -u toto squeue -h -j 33254141 -o '%T %B'
[I 2018-05-29 17:07:29.112 JupyterHub batchspawner:330] Notebook server job 33254141 started at mynode:40483
[D 2018-05-29 17:07:29.116 JupyterHub spawner:727] Polling subprocess every 30s
[I 2018-05-29 17:07:30.891 JupyterHub log:122] 200 GET /hub/api (@2.2.2.36) 0.80ms
[D 2018-05-29 17:07:38.041 JupyterHub batchspawner:214] Spawner querying job: /usr/bin/sudo -E -u toto squeue -h -j 33254141 -o '%T %B'
[W 2018-05-29 17:07:38.062 JupyterHub base:500] User toto is slow to become responsive (timeout=10)
[D 2018-05-29 17:07:38.062 JupyterHub base:502] Expecting server for toto at: http://mynode.mycompany.org:40483/user/toto/

And on the remote node serving the notebook application (SingleUserNotebookApp) :

[I 2018-05-29 17:07:30.886 SingleUserNotebookApp singleuser:365] Starting jupyterhub-singleuser server version 0.8.1
[D 2018-05-29 17:07:30.892 SingleUserNotebookApp _version:46] jupyterhub and jupyterhub-singleuser both on version 0.8.1
[I 2018-05-29 17:07:30.892 SingleUserNotebookApp notebookapp:1619] Serving notebooks from local directory: /mnt/scratch/user/toto/notebooks
[I 2018-05-29 17:07:30.892 SingleUserNotebookApp notebookapp:1619] 0 active kernels
[I 2018-05-29 17:07:30.892 SingleUserNotebookApp notebookapp:1619] The Jupyter Notebook is running at:
[I 2018-05-29 17:07:30.893 SingleUserNotebookApp notebookapp:1619] https://mynode.mycompany.org:40483/user/toto/
[I 2018-05-29 17:07:30.893 SingleUserNotebookApp notebookapp:1620] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2018-05-29 17:07:31.318 SingleUserNotebookApp iostream:1451] SSL Error on 11 ('1.1.1.45', 43880): [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:833)
[W 2018-05-29 17:07:34.948 SingleUserNotebookApp iostream:1451] SSL Error on 11 ('1.1.1.45', 43882): [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:833)
[W 2018-05-29 17:07:38.026 SingleUserNotebookApp iostream:1451] SSL Error on 11 ('1.1.1.45', 43884): [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:833)
[W 2018-05-29 17:07:43.036 SingleUserNotebookApp iostream:1451] SSL Error on 11 ('1.1.1.45', 43894): [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:833)

and finaly the timeout :

[E 2018-05-29 17:09:26.873 JupyterHub gen:940] Exception in Future <Future finished exception=TimeoutError("Server at http://mynode.company.org:40483/user/toto/ didn't respond in 120 seconds",)> after timeout
TimeoutError: Server at http://mynode.company.org:40483/user/toto/ didn't respond in 120 seconds

I'm able connect directly the SingleUserNotebookApp through : https://mynode.mycompany.org:40483/user/toto/

but the master server is trying to connect : Expecting server for toto at: http://mynode.mycompany.org:40483/user/toto/
which is not reachable because of not using the https.

I am looking after a correct configuration to specify to user https ?

Any one could help me ?

Thank you!

[willingc]

Hi @matthdan, Sorry about the delay in responding. The JupyterHub team was at an offsite meeting last week.

This seems similar to https://github.com/jupyterhub/batchspawner/issues/88.

@minrk, There seems to be a few issues with similar error messages. Thoughts?

[rkdarst]

I just commented on jupyterhub/batchspawner#88, I think that is a different issue.

The core problem in this issue seems to be that the single-user notebook server is using SSL, but the JupyterHub proxy defaults/forces non-ssl.

At least in my deployment, the slurm spawners are on a secure network - at least as secure as the Slurm nodes themselves. SSL termination happens at JupyterHub and from then on isn't needed. So, we don't use ssl on the single-user servers, and I haven't seen signs of this being supported - is it?

If you want ssl between hub<->user servers, maybe someone can comment on if that is possible... but is probably more difficult than normal ssl since hostname + port is always changing.

[matthdan]

Hello all,

This issue https://github.com/jupyterhub/batchspawner/issues/88 seems definitively different.
I push a simple patch to the current stable version (0.8.1) to force https connection on my own repo.

https://github.com/matthdan/jupyterhub/commit/ead7fcb173d2c912a661c284a669cf8e91c66c1a#diff-29a823aad1b90aae7bdb34120645954eL381

I works and solve this problem.

It seems patched in the 0.9 version and I think this not relevant and could be closed!

Thank you.

It s a right workaround

[willingc]

Thanks for the follow up @matthdan and @rkdarst. Closing 👍