Notebook: JupyterLab with notebook 5.7.8 and tornado 6

[ian-r-rose]

This issue is to track issues with the recent tornado 6 update. It's corresponding issue in lab is jupyterlab/jupyterlab#6131. Our tests are currently failing on kernel restarts with errors like

    HTTPServerRequest(protocol='http', host='localhost:8889', method='GET', uri='/api/kernels/bb699f4d-dd2a-417c-8883-a0adba0a43da/channels?session_id=081a2738-da46-46db-b556-c8de0d7b0dff&token=5ad5f308acfdba227495ff408b73f8ac98fb64fa99ec9ce8', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/notebook/services/kernels/handlers.py", line 241, in get
        yield super(ZMQChannelsHandler, self).get(kernel_id=kernel_id)
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
        value = future.result()
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/notebook/base/zmqhandlers.py", line 297, in get
        yield gen.maybe_future(res)
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
        value = future.result()
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/websocket.py", line 278, in get
        await self.ws_connection.accept_connection(self)
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/websocket.py", line 881, in accept_connection
        await self._accept_connection(handler)
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/websocket.py", line 964, in _accept_connection
        await self._receive_frame_loop()
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/websocket.py", line 1118, in _receive_frame_loop
        await self._receive_frame()
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/websocket.py", line 1130, in _receive_frame
        data = await self._read_bytes(2)
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/websocket.py", line 1124, in _read_bytes
        data = await self.stream.read_bytes(n)
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/iostream.py", line 436, in read_bytes
        future = self._start_read()
      File "/home/ian/anaconda3/envs/test-tornado/lib/python3.7/site-packages/tornado/iostream.py", line 797, in _start_read
        assert self._read_future is None, "Already reading"

A fix was made in #4513, but we are still seeing these issues. This may also be related to issues that mybinder has been having with lab binders (cc @betatim).

xrefs #4439 #4412

[mathematicalmichael]

subbing to this because I've been experiencing really laggy response times for restarts/reloads.
how far back would I need to go to reverse these problems? or roughly how long until this stuff is sorted and propagates changes to jupyter/docker-stacks?

[fcollonval]

Hey, we are also facing those lags. This gets quickly on user nerves.
@mathematicalmichael rolling back to the 5.7.5 version (before additional security checks were introduced), restore a better user experience.

[mathematicalmichael]

if I run a rollback inside of a container, will the changes take effect or does something else need to be done? seems like I'm still getting lag.

[Zsailer]

I tracked down this issue to a single change from Tornado 5.x to 6.x. I've submitted a PR to tornado that should fix this error.

Look out for that PR to merge and let us know if you're still seeing this issue!

[michaelaye]

Is this related to getting:

Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py:19: VisibleDeprecationWarning: zmq.eventloop.minitornado is deprecated in pyzmq 14.0 and will be removed.
    Install tornado itself to use zmq with the tornado IOLoop.

despite the fact that I have pyzmq 17.1.2 and tornado 6.0.3 where this should have been fixed, IIUC?

[Zsailer]

@michaelaye That's a separate issue with pyzmq, but also something we should address.

pyzmq attempts to import a deleted module in tornado, context_stack. When it cannot find it, it assumes tornado is not installed, falls back to importing a deprecated module inside pyzmq, zmq.eventloop.minitornado.context_stack, and triggers this warning. The warning is not helpful though, because the problem is not tornado, but rather, pyzmq is behind.

Filing an issue there.

[Zsailer]

Opened zeromq/pyzmq#1310 for @michaelaye's issue.

[minrk]

@michaelaye try upgrading pyzmq. 17.1.3 has a fix for that warning, 18.0.2 is current.

[Zsailer]

Thanks @minrk!