Voila: Spawning ~20 requests at a time results in a ZMQError

[martinRenou]

Running ab -n 50 -c 20 http://localhost:8866/ eventually results in:

Traceback (most recent call last):
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "</home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/decorator.py:decorator-gen-124>", line 2, in initialize
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 527, in initialize
    self.init_sockets()
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 266, in init_sockets
    self.shell_port = self._bind_socket(self.shell_socket, self.shell_port)
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 213, in _bind_socket
    return self._try_bind_socket(s, port)
  File "/home/martin/miniconda3/envs/voila/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 189, in _try_bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "zmq/backend/cython/socket.pyx", line 550, in zmq.backend.cython.socket.Socket.bind
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Address already in use

[timkpaine]

I see this one a ton, jupyter recovers from it but voila does not

[timkpaine]

@vidartf

[martinRenou]

I'll investigate a bit on this. I'll check where zmq ports are chosen, see if I can see why we chose already used addresses. I wonder if it's not done in nbconvert.

[martinRenou]

Ok. So actually it seems to me that the logic is wrong in jupyter_client for choosing available ports.

Binding to port 0 chooses a currently available port. But then we close the socket, I guess this socket is only used for finding an available port and we create the "real" socket (The one that is actually used) later on. We assume the port will still be available when the socket is actually created. Which is a dangerous asumptions.

[vidartf]

@martinRenou Any suggestions for how to do it in a more robust manner?

[martinRenou]

I don't know the jupyter_client internals enough... So I don't have any suggestion right now.
The ideal solution would be to check for available ports when creating actual sockets used for the communication. But I don't know if it's even possible.

[martinRenou]

@minrk Do you have an idea on this?

Would it be feasible to give responsibility to the kernel for choosing available ports? Instead of choosing those ports at the creation of the connection file?

[martinRenou]

A pseudo-fix would be to put the logic for choosing ports in voila directly, and keep a Python set of those ports in memory.

Voila would prevent two kernels to have the same port, but it would not prevent another application to take one of those ports. It would only reduce the probability to have this issue.

[vidartf]

If this is a jupyter_client issue, then I think we should submit an issue there, and close this one.

[vidartf]

(or rather, this issue can be for tracking us making sure voila handles the failure gracefully)

[martinRenou]

Yes, @JohanMabille is working on writing a proper issue in jupyter_client. But a proper fix would require changes in jupyter_client and kernels implementations.

I am also currently working a the pseudo-fix/workaround for Voila. I guess we can close this issue when the workaround is in.

[martinRenou]

Actually I will do the workaround directly in jupyter_server, closing this issue as it is not relevant to voila

[jtpio]

Thanks @martinRenou for looking into this :+1:

[maartenbreddels]

I see this one a ton, jupyter recovers from it but voila does not

do you mean as a server extension? And which OS?