Notebook: Bind to unix socket instead of IP and port

[DeepHorizons]

Is there any way to have Jupyter bind to a unix socket instead of a TCP port? We are looking into using Jupyter on our cluster and would like to avoid exposing ports, even on localhost.

We used Docker but ran into a security problem so that is out of the question.

[takluyver]

I think it's possible to use Unix sockets for the kernels, though I forget exactly how to set that up. @minrk can probably give more info.

The web server needs to run on a TCP socket, as browsers can't connect to a Unix socket, as far as I know.

[minrk]

Kernels can use unix sockets by setting c.KernelManager.transport = 'ipc'. tornado can bind on unix sockets, but we don't expose that as an option right now. You would have to subclass NotebookApp to do it. I presume if you want to bind the notebook server to a unix socket that will be exposed publicly via a proxy such as nginx?

[dhouck]

I know that’s why I want to use a non-IP socket for the server.

[ibukanov]

If jupyter supported listening on a unix socket, I could use normal unix permissions on the server to restrict who can access it without any passwords. Then, as OpenSSH supports redirecting client port to a unix socket on the server, I could just ssh -L 8888:/path/to/jupyter/socket host. After that I could in my browser typed localhost:8888 to connect to jupyter over secure connection without any passwords.

[takluyver]

Oh neat, I didn't know that was possible. PRs welcome; the relevant code is here:

https://github.com/jupyter/notebook/blob/b76c8d907e3117b91f84236cbe9fff02bc3b665c/notebook/notebookapp.py#L1299

And tornado docs:
http://www.tornadoweb.org/en/stable/httpserver.html
http://www.tornadoweb.org/en/stable/netutil.html#tornado.netutil.bind_unix_socket

[cdyson37]

You can control the permissions of a unix socket. Might be neater than a one time cookie?

[takluyver]

Unfortunately it doesn't help much for the default use case - the server has to be accessible on a TCP port, because browsers can't connect to unix sockets.

[cdyson37]

Didn't know that - boo!

[JackJHarris]

The unix-socket solution would really only be useful as @ibukanov suggested -- on remote systems over an SSH tunnel

However, there is a use-case for this now on HPC environments where users access shared login nodes.
Any remote service running there can only be accessed over SSH port forwarding anyway.
This would allow a user to start a jupyter instance with a unix-socket protected in their home directory and run it indefinitely.

Reconnecting (even from another access terminal) is as simple and a new tunnel to that unix socket

[kwlzn]

I'm interested in picking this one up soon - let me know if anyone else is already working on it.

[JackJHarris]

I implemented the below code as a quick hack for my team -- it is a "hack" to notebookapp.py, if a _negative_ port is passed then create a unix socket in the home directory

notebookapp.py

1399 self.http_server = httpserver.HTTPServer(self.web_app, ssl_options=ssl_options,
1400 xheaders=self.trust_xheaders,
1401 max_body_size=self.max_body_size,
1402 max_buffer_size=self.max_buffer_size)
1403
1404 success = None

        if self.port < 0:
             path = os.path.expandvars('$HOME/.jupyter-unixsockets')
             if not os.path.exists(path):
                 os.mkdir( path, 0o700)
             socket_path = path + "/" + str(self.port)
             from tornado.netutil import bind_unix_socket
             socket = bind_unix_socket(socket_path)
             self.http_server.add_socket(socket)
             tornado.ioloop.IOLoop.instance().start()
             self.log.info(_('Running Jupyter with Unix Sockets at %s') % socket_path)
             success = True

It works, but clearly not the full solution
Hopefully, it helps you out

[kwlzn]

I have an initial PR out for this here: https://github.com/jupyter/notebook/pull/4835

this is working well with local testing, just needs some kind of test coverage. if someone's available to take a look to provide high level guidance, that'd be great.

[kwlzn]

tests are in and #4835 should be good to go now.

cc @takluyver @minrk in case either of you can recommend someone to help review.

[ehdgus8077]

I need the unix socket for the jupyter too much now.
When will the PR be included?
Is there any way to use it right now?

[dhouck]

I don't know what state the PR is in, but if it's working correctly, you can use it by checking out a copy of the repository with git pull origin pull/4835/merge.

[kwlzn]

we've been dogfooding #4835 in production at Twitter for a couple of weeks now with 100% of internal user traffic going over this mechanism - no issues on our side.

would be great to get this landed for the 7.0.0 release tho.

[qguv]

This is currently blocking the upstream VSCode security issue microsoft/vscode-python#8331. It would be lovely to get the upstream issue cleared :slightly_smiling_face: