Notebook: websocket message size limitation introduced in tornadoweb causes kernel crashes

[AaronWatters]

I have some IPython widgets that stopped working for larger data sets recently and with the help of chatters in https://gitter.im/jupyter-widgets/Lobby in particular Jason Grout @jasongrout it turns out that the failures are due to a new limitation in tornadoweb zmq over websockets

See first entry here: http://www.tornadoweb.org/en/stable/releases/v4.5.0.html

This commit: https://github.com/tornadoweb/tornado/commit/104a302b750131d463f7e3b8e0f71dd334e5f904

This issue: https://github.com/tornadoweb/tornado/issues/1269

This new limitation results in kernel crashes that I didn't see before if (for example) a javascript widget attempts to transfer more than 10 megs of data in a message to a python kernel.

Is it possible to fix/work-around the limitation in a robust way in general within the Jupyter framework?

[takluyver]

It's possible to increase the limit, but you might just hit the higher limit anyway. There are probably good reasons to have some kind of limit on it - it's better to kill that kernel than use up memory until the server crashes.

[AaronWatters]

I agree. However it would be nice to have a way to send an arbitrary amount of data
from javascript to jupyter. One approach is to break large messages into smaller ones and
send those in sequence. I'm probably going to do this at the widget implementation level to allow my widget(s) to send large data to the kernel. I don't know if it makes sense to implement support for large data transfers further down the Jupyter stack (at the comm level or something) but it might.

By the way in the current implementation I have crashed the Jupyter server when attempting to send large enough messages. I believe I can reproduce this by uploading a 200meg file in a javascript widget and then cramming the contents across the websocket as a single message. Boom.

[maartenbreddels]

This is indeed quite a serious showstopper for widgets and binary transfers, did you find a good workaround for this?

[jasongrout]

Of course, you can reset websocket_max_message_size at the tornado level...

[harmon]

I'm also running into this I believe. Got a 12 MB iPyWidget Output field update being pushed from the kernel to the web browser, and it reliably closes the websocket connection with "Message too big (1009)". Can this "websocket_max_message_size " value be exposed so it's configurable?

[AaronWatters]

jp_proxy_widgets implements its own meta protocol to transfer data in chunks.
https://github.com/AaronWatters/jp_proxy_widget/commit/45186aea52a8acbcaca9565fbd873f8ebed15787

[cosmoscalibur]

Of course, you can reset websocket_max_message_size at the tornado level...

How can I reset this value launching notebook? Thia value is not present in notebook app tornado settings.

[cosmoscalibur]

For future reference:

Example to limit size in 100 MB.
Create config file

jupyter notebook --generate-config

Edit (uncomment) the following line in jupyter_notebook_config.py (generated file) and change the value (desired size in MB * 1024 * 1024):

c.NotebookApp.tornado_settings = {"websocket_max_message_size": 100 * 1024 * 1024}

Launch notebook with --config parameter pointing to config file:

jupyter notebook --config="jupyter_notebook_config.py" your_notebook.ipynb

[slw07g]

In addition to @cosmoscalibur's excellent instructions, you could alternatively store the config file in one of the paths jupyter watches, and it will load the config automatically.

To see the paths jupyter is configured to watch, run jupyter --paths