Jupyterhub: WebSocketClosedError

[gagan-preet]

Hi,

We ran a piece of code on a Jupyter notebook and it returned an error on the console. I am attaching the error log below. The notebook crashes (loses contact with the kernel) and any further step requires us to restart the kernel and rerun all the cells from the beginning. I saw that there was a timeout warning _WebSocket ping timeout after 90000 ms_. I tried running the same code using the python terminal. It completed the computation within 60 second. Has anyone faced a similar issue before? Is there any way to work around this?

Note: The code is basically 3 nested for loops (320,240,68 respectively) with some basic linear algebra and arithmetic operations. The code itself is not heavy but we saw that it seems to work fine if my loops are of a smaller size (less than 100 for each loop).

_Error log_:

[I 2015-04-27 13:10:41.480 iot kernelmanager:85] Kernel started: 0ebb6d4f-15a0-4dab-ae12-802d3a1f782d
[W 2015-04-27 13:12:11.797 iot zmqhandlers:270] WebSocket ping timeout after 90000 ms.
[E 2015-04-27 13:12:11.849 iot web:1421] Uncaught exception GET /user/iot/api/kernels/0ebb6d4f-15a0-4dab-ae12-802d3a1f782d/channels?session_id=EBB8AEB162D548D898E514A1C3A12FAD (127.0.0.1)
    HTTPServerRequest(protocol='http', host='172.25.1.110:8000', method='GET', uri='/user/iot/api/kernels/0ebb6d4f-15a0-4dab-ae12-802d3a1f782d/channels?session_id=EBB8AEB162D548D898E514A1C3A12FAD', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Cache-Control': 'no-cache', 'Cookie': 'jupyter-hub-token-iot="2|1:0|10:1430119813|21:jupyter-hub-token-iot|44:YmFjNDczNGE1NjBhNDFhY2JjYzY2ZGQ1MjYxNDExZWU=|c59937cc923302b3b9e25a3684614688b065c201f34f504a453edfde77dc7406"', 'Connection': 'Upgrade', 'Sec-Websocket-Extensions': 'permessage-deflate; client_max_window_bits', 'Pragma': 'no-cache', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36', 'X-Forwarded-Port': '8000', 'Upgrade': 'websocket', 'X-Forwarded-Proto': 'ws', 'Origin': 'http://172.25.1.110:8000', 'Sec-Websocket-Version': '13', 'Accept-Encoding': 'gzip, deflate, sdch', 'X-Forwarded-For': '172.25.1.200', 'Accept-Language': 'en-US,en;q=0.8', 'Host': '172.25.1.110:8000', 'Sec-Websocket-Key': 'd8bcNtyyLr7K6IHqHHE2Bw=='})
    Traceback (most recent call last):
      File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 1302, in _stack_context_handle_exception
        raise_exc_info((type, value, traceback))
      File "<string>", line 3, in raise_exc_info
      File "/usr/local/lib/python3.4/dist-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/usr/local/lib/python3.4/dist-packages/zmq/eventloop/zmqstream.py", line 184, in <lambda>
        self.on_recv(lambda msg: callback(self, msg), copy=copy)
      File "/usr/local/lib/python3.4/dist-packages/IPython/html/base/zmqhandlers.py", line 185, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/usr/local/lib/python3.4/dist-packages/tornado/websocket.py", line 200, in write_message
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError

_System info_:
Python:

{'commit_hash': u'128db79',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/usr/local/lib/python2.7/dist-packages/IPython',
 'ipython_version': '3.0.0-dev',
 'os_name': 'posix',
 'platform': 'Linux-3.13.0-32-generic-x86_64-with-Ubuntu-14.04-trusty',
 'sys_executable': '/usr/bin/python',
 'sys_platform': 'linux2',
 'sys_version': '2.7.6 (default, Mar 22 2014, 22:59:56) \n[GCC 4.8.2]'}

Python3:

{'commit_hash': '128db79',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/usr/local/lib/python3.4/dist-packages/IPython',
 'ipython_version': '3.0.0-dev',
 'os_name': 'posix',
 'platform': 'Linux-3.13.0-32-generic-x86_64-with-Ubuntu-14.04-trusty',
 'sys_executable': '/usr/bin/python3',
 'sys_platform': 'linux',
 'sys_version': '3.4.0 (default, Apr 11 2014, 13:05:11) \n[GCC 4.8.2]'}

Numpy version: 1.9.2
Tornado version: I tried to print tornado.__version__ but it ran into an error saying: AttributeError: 'module' object has no attribute '__version__' (underscores around version are not visible due to markdown formatting)

[rgbkrk]

[W 2015-04-27 13:12:11.797 iot zmqhandlers:270] WebSocket ping timeout after 90000 ms.

That line seems somewhat telling.

Is this JupyterHub or a pure IPython/Jupyter notebook server? Where do you have this set up, your local machine or somewhere on the internet? Is there any kind of proxy in front of it?

[gagan-preet]

1. We are using JupyterHub.
2. This setup is on a workstation (Ubuntu 14.04) which I am accessing over the local network.
3. The proxy is whatever JupyterHub provides.

The timeout is for 90 seconds but the code execution on my local without IPython/Jupyter takes only 60 seconds.

Additional information
I am using the following kernel.json:

python { "display_name": "IPython (Python 2)", "language": "python", "codemirror_mode": { "version": 3, "name": "ipython" }, "argv": [ "python2", "-c", "from IPython.kernel.zmq.kernelapp import main; main()", "-f", "{connection_file}" ] }```

[zooman]

Bump - any word on this, we are experiencing as well.

[minrk]

Do you see the same error when connecting from localhost? Are there any firewalls or other corporate meddlers on your LAN?

[gagan-preet]

I tried running the same note book on local. It returned the same error on console.
When I executed the code in IPython terminal, it ran just fine.
The LAN is clean and without any meddlers. The workstation running the server has Ubuntu 14.04 with firewall disabled.
I think the problem in this case is with the Jupyter itself. The timeout is happening even though the code shouldn't be taking so long to execute.

[zooman]

1. No firewalls on the network.
2. It works on IPython terminal (local) but gives the same error on the Jupyter Notebook (local).

[minrk]

The timeout should be unrelated to code execution. It's a websocket ping from the server to the client. What browser and version are you using? And what tornado version (tornado.version)?

[abewley]

Did anyone find a solution to this?
I'm experiencing the same issue on my machine running OS X 10.10.4
The ipython version is: 3.2.0
I was accessing it via localhost in safari.

Below is my error log:

[I 13:46:34.178 NotebookApp] Malformed HTTP message from ::1: Content-Length too long
[I 13:47:09.619 NotebookApp] Malformed HTTP message from ::1: Content-Length too long
[I 13:48:35.746 NotebookApp] Malformed HTTP message from ::1: Content-Length too long
[I 13:49:12.813 NotebookApp] Malformed HTTP message from ::1: Content-Length too long
[W 14:10:18.201 NotebookApp] WebSocket ping timeout after 90002 ms.
[E 14:10:18.236 NotebookApp] Uncaught exception GET /api/kernels/2fca0b76-c0ba-4378-83e3-f7d6b679a87c/channels?session_id=2EF9AD388F5C42FAB1580FE498C972B1 (::1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/kernels/2fca0b76-c0ba-4378-83e3-f7d6b679a87c/channels?session_id=2EF9AD388F5C42FAB1580FE498C972B1', version='HTTP/1.1', remote_ip='::1', headers={'Origin': 'http://localhost:8888', 'Upgrade': 'websocket', 'Sec-Websocket-Extensions': 'x-webkit-deflate-frame', 'Sec-Websocket-Version': '13', 'Connection': 'Upgrade', 'Sec-Websocket-Key': '8IxpBzl4cj+T+SEM+KBHWQ==', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/0600.3.18 (KHTML, like Gecko) Version/8.0.3 Safari/0600.3.18 ADSSO', 'Host': 'localhost:8888', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache'})
    Traceback (most recent call last):
      File "/Users/bewley/anaconda/lib/python2.7/site-packages/tornado/web.py", line 1369, in _stack_context_handle_exception
        raise_exc_info((type, value, traceback))
      File "/Users/bewley/anaconda/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/Users/bewley/anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 184, in <lambda>
        self.on_recv(lambda msg: callback(self, msg), copy=copy)
      File "/Users/bewley/anaconda/lib/python2.7/site-packages/IPython/html/base/zmqhandlers.py", line 185, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/bewley/anaconda/lib/python2.7/site-packages/tornado/websocket.py", line 213, in write_message
        raise WebSocketClosedError()
    WebSocketClosedError

[StephenElston]

I have the same issue on Windows 7 running locally.

[swathimocharla]

Is there any update on this? I am facing the same issue with my notebook.
I am running a code that does a lot of feature engineering and then implements an algorithm on this data.
Running the notebook on my local that is an 8gb RAM.Also, when this happens, the system is getting hung.

[arsturges]

Same issue here, executing long-running code in a long-lived Jupyter notebook running on a remote server.

[minrk]

@arsturges are you seeing it with JupyterHub or a regular notebook server? What browser is everyone using? This seems like a failure to properly implement the websocket protocol by _some_ agent involved. The most likely candidate for this is an intermediate service, such as a proxy or firewall.

Load due to computation in the kernel should really have no effect on this unless it's actually consuming all of the CPU and/or memory of the system, not allowing any other process on the machine to take even basic actions.

[pouya-cognitiv]

I am using chrome on a regular notebook server.

[W 12:34:03.683 NotebookApp] WebSocket ping timeout after 119993 ms.
[E 12:34:03.715 NotebookApp] Uncaught exception GET /api/kernels/89a35957-baf9-4bd8-b6b1-518cca525001/channels?session_id=91ACF4A6C7654FCBB75996A53DC54172 (::1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/kernels/89a35957-baf9-4bd8-b6b1-518cca525001/channels?session_id=91ACF4A6C7654FCBB75996A53DC54172', version='HTTP/1.1', remote_ip='::1', headers={'Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Host': 'localhost:8888', 'Upgrade': 'websocket', 'Sec-Websocket-Extensions': 'permessage-deflate; client_max_window_bits', 'Sec-Websocket-Key': 'FXIwfMJQ2ViS8tAOE91iGQ==', 'Sec-Websocket-Version': '13', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36', 'Accept-Language': 'en-US,en;q=0.8,fr;q=0.6', 'Connection': 'Upgrade', 'Accept-Encoding': 'gzip, deflate, sdch', 'Origin': 'http://localhost:8888'})
    Traceback (most recent call last):
      File "C:\Anaconda3\lib\site-packages\tornado\web.py", line 1401, in _stack_context_handle_exception
        raise_exc_info((type, value, traceback))
      File "<string>", line 3, in raise_exc_info
      File "C:\Anaconda3\lib\site-packages\tornado\stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "C:\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 184, in <lambda>
        self.on_recv(lambda msg: callback(self, msg), copy=copy)
      File "C:\Anaconda3\lib\site-packages\notebook\base\zmqhandlers.py", line 188, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "C:\Anaconda3\lib\site-packages\tornado\websocket.py", line 215, in write_message
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError

[arsturges]

@minrk I am running Jupyter notebook server, not JupyterHub, and using Chrome.

[E 07:09:21.239 NotebookApp] Uncaught exception GET /api/kernels/fd777d21-5d54-4f28-8773-2012cc4339d4/channels?session_id=F79877802E4244E7820F556C049283
53 (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/kernels/fd777d21-5d54-4f28-8773-2012cc4339d4/channels?session_id=F79877802E4244E7820F556C04928353', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Origin': 'http://localhost:8888', 'Upgrade': 'websocket', 'Accept-Language': 'en-US,en;q=0.8', 'Accept-Encoding': 'gzip, deflate, sdch', 'Sec-Websocket-Version': '13', 'Host': 'localhost:8888', 'Sec-Websocket-Key': '$hpBnWK6pCrN2sMskyfWDQ==', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Sa$ari/537.36', 'Connection': 'Upgrade', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache', 'Sec-Websocket-Extensions': 'permessage-deflate; client_max_wi$dow_bits'})
    Traceback (most recent call last):
      File "/home/asturges/sci/trip_tagging/env/local/lib/python2.7/site-packages/tornado/web.py", line 1401, in _stack_context_handle_exception
        raise_exc_info((type, value, traceback))
      File "/home/asturges/sci/trip_tagging/env/local/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/home/asturges/sci/trip_tagging/env/local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 191, in <lambda>
        self.on_recv(lambda msg: callback(self, msg), copy=copy)
      File "/home/asturges/sci/trip_tagging/env/local/lib/python2.7/site-packages/notebook/base/zmqhandlers.py", line 253, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/home/asturges/sci/trip_tagging/env/local/lib/python2.7/site-packages/tornado/websocket.py", line 215, in write_message
        raise WebSocketClosedError()
    WebSocketClosedError

[dmirecki]

I have same issue on long-running code. I use Chrome on Ubuntu 15. The server is running locally.
tornado version: 4.3
ipython version: 4.1.1

[W 20:38:57.040 NotebookApp] WebSocket ping timeout after 102715 ms.
[E 20:38:57.054 NotebookApp] Uncaught exception GET /api/kernels/617d1236-7f15-42f6-9d64-43b16e7e5275/channels?session_id=482659481AFC48348AF5EE123DA677BA (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/kernels/617d1236-7f15-42f6-9d64-43b16e7e5275/channels?session_id=482659481AFC48348AF5EE123DA677BA', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Origin': 'http://localhost:8888', 'Upgrade': 'websocket', 'Accept-Language': 'pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4', 'Accept-Encoding': 'gzip, deflate, sdch', 'Sec-Websocket-Version': '13', 'Host': 'localhost:8888', 'Sec-Websocket-Key': '+Tni1wqi66CofUP8clB0wg==', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36', 'Dnt': '1', 'Connection': 'Upgrade', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache', 'Sec-Websocket-Extensions': 'permessage-deflate; client_max_window_bits'})
    Traceback (most recent call last):
      File "/media/d/venvs/tensorflow/local/lib/python2.7/site-packages/tornado/web.py", line 1401, in _stack_context_handle_exception
        raise_exc_info((type, value, traceback))
      File "/media/d/venvs/tensorflow/local/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/media/d/venvs/tensorflow/local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 191, in <lambda>
        self.on_recv(lambda msg: callback(self, msg), copy=copy)
      File "/media/d/venvs/tensorflow/local/lib/python2.7/site-packages/notebook/base/zmqhandlers.py", line 253, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/media/d/venvs/tensorflow/local/lib/python2.7/site-packages/tornado/websocket.py", line 215, in write_message
        raise WebSocketClosedError()
    WebSocketClosedError

[RGTippens]

Same issue here, running python 2.7 and the latest version of Jupyter notebook on Yosemite. Everything is local. I don't have this problem running the same code on Mavericks. I've tried restoring previous versions of Ipython but those still give the timeout.

[yrevar]

I faced the same issue 3 times with long running kernel. Is there any fix or workaround? Any way to retrieve the old kernel if it's not dead?

I was connected remotely to a jupter running on AWS server. I was training a deep neural network, then probably after 40-50 minutes the connection died. Lost my weights worth 3 hours of training :(

[W 17:52:06.488 NotebookApp] WebSocket ping timeout after 119861 ms.
[W 17:52:12.626 NotebookApp] WebSocket ping timeout after 119978 ms.
[E 17:52:13.947 NotebookApp] Uncaught exception GET /api/kernels/faa9c03f-22f8-4c7e-8851-95de2e0db10f/channels?session_id=CF274D4052EE406993ABCD48764D39DA (ip.ip.ip.ip)
Traceback (most recent call last):
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tornado/web.py", line 1401, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
ret = fn(_args, *_kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 184, in
self.on_recv(lambda msg: callback(self, msg), copy=copy)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/notebook/base/zmqhandlers.py", line 188, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tornado/websocket.py", line 215, in write_message
raise WebSocketClosedError()
WebSocketClosedError

Update:
This happens very frequently, almost all the time!

[efrubin]

I also have this issue.

[SeanBE]

Same issue here. Chrome. Virtualenv. Getting the following traceback in debug mode.
Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/tornado/web.py", line 1401, in _stack_context_handle_exception raise_exc_info((type, value, traceback)) File "/usr/local/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped ret = fn(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 191, in <lambda> self.on_recv(lambda msg: callback(self, msg), copy=copy) File "/usr/local/lib/python2.7/site-packages/notebook/base/zmqhandlers.py", line 253, in _on_zmq_reply self.write_message(msg, binary=isinstance(msg, bytes)) File "/usr/local/lib/python2.7/site-packages/tornado/websocket.py", line 215, in write_message raise WebSocketClosedError() WebSocketClosedError

I'm only getting this when I'm running Keras (neural network library).

[apsey]

Similar issue here, running long jobs overnight on a jupyter notebook running in a container in AWS.

[W 02:59:56.695 NotebookApp] WebSocket ping timeout after 119754 ms.
[W 04:01:38.609 NotebookApp] WebSocket ping timeout after 119773 ms.
[I 04:10:23.012 NotebookApp] KernelRestarter: restarting kernel (1/5)
[W 05:02:18.194 NotebookApp] WebSocket ping timeout after 90001 ms.
[W 06:03:02.363 NotebookApp] WebSocket ping timeout after 90000 ms.
[W 06:35:52.994 NotebookApp] WebSocket ping timeout after 90001 ms.
[W 07:36:37.024 NotebookApp] WebSocket ping timeout after 90000 ms.
[W 08:01:22.660 NotebookApp] WebSocket ping timeout after 90000 ms.
[W 09:02:06.264 NotebookApp] WebSocket ping timeout after 90001 ms.
[W 10:02:50.583 NotebookApp] WebSocket ping timeout after 90000 ms.
[W 11:04:34.876 NotebookApp] WebSocket ping timeout after 119867 ms.
[W 11:10:33.387 NotebookApp] WebSocket ping timeout after 90000 ms.

Kernel restarts and I lose I already calculated.

[silburt]

I have the same issue - I'm running a jupyter notebook on a remote machine, connecting to it via ssh -L and using chrome. After I leave a long simulation running overnight in the jupyter notebook I return to it the next morning to see the kernel has died with the following error message:

[W 18:29:56.824 NotebookApp] WebSocket ping timeout after 119998 ms.
[I 09:30:42.912 NotebookApp] 302 GET / (::1) 0.60ms
[W 09:30:54.551 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20160510171815 (::1) 4.60ms referer=http://localhost:8080/notebooks/ipython_examples/*.ipynb
[I 09:32:39.651 NotebookApp] Kernel shutdown: 3cc40e7b-a66b-496b-b177-954f2565a6c4

[dspoka]

I also have this issue. Anyone have thoughts on how to work around this?

[zxzjim]

same issue here, occurs when the code going through large loops and printing too much. I'm running locally and using OSX El Capitan and Chrome. Python 2.7 Anaconda setup.

[willingc]

@minrk This issue seems related to notebook server vs JupyterHub specific. Should we close this here and add a link to refer to: https://github.com/jupyter/notebook/issues/1164?

[StephenElston]

I encountered this problem on notebook server only.

[willingc]

Closing as a duplicate of jupyter/notebook#1164