Notebook: SSH remote ipython notebook in detached mode using tmux

[ghost]

After starting ipython notebook on the server, opening it on the local machine works fine. However, when I try to combine tmux with it, the notebook fails to complete the work. The following is my workflow:

1. SSH to the server and start the ipython notebook with tmux session remotely.
2. Open the browser on the local machine and start a long-run task.
3. detach the session, exit SSH and good to go.

The above process is an ideal one, I got kernel shutdown in the end. The following is the message I got from the tmux session:

[W 19:15:52.617 NotebookApp] Notebook simplified_algo.ipynb is not trusted      # The first time I left 
[W 19:27:55.249 NotebookApp] WebSocket ping timeout after 119865 ms.  
[I 22:09:13.696 NotebookApp] Saving file at /simplified_algo.ipynb                        # I then come back 
[W 22:09:13.698 NotebookApp] Saving untrusted notebook simplified_algo.ipynb
[W 22:15:18.390 NotebookApp] Notebook simplified_algo.ipynb is not trusted
[I 22:17:21.426 NotebookApp] Saving file at /simplified_algo.ipynb
[W 22:17:21.428 NotebookApp] Saving untrusted notebook simplified_algo.ipynb
[I 22:19:20.954 NotebookApp] Saving file at /simplified_algo.ipynb
[I 22:21:20.974 NotebookApp] Saving file at /simplified_algo.ipynb                       # The second time I left
[W 22:38:33.102 NotebookApp] WebSocket ping timeout after 119992 ms.
[I 23:01:07.494 NotebookApp] Kernel shutdown: 75b0cf60-610e-4931-99cb-847d821a726b

As you can see, there are WebSocket ping timeout message every time I detach for a while (it is not a instant message after detach), how can it be fixed?

[Carreau]

This should not affect at all the websocket. Do you have any intermediate proxies, or things that would cut connexions it they become iddle ? Does it do the same if you SSH from a 3rd machine ?

[ghost]

Sorry I am fairly unfamiliar with the network configuring. What is going to be idle after detach? How do I check the existence of intermediate proxies? It seems that it happens only if I stop connecting SSH.

[Carreau]

Well if you ssh, the connection between your local machine and your server might idle on port 22, you do not have any packets going through. If you have a Nginx, or whatever in the middle (that might depends on the provider of your server, or sysadmin configuration), it might not recognize websocket as a persisting connexion. So it assume when you disconnect ssh that all connexion between local machine and server can be dropped.

But this has nothing to do with the notebook itself. You just have something in between your notebook and your local machine that drop connections.

[Carreau]

Closing as nothing we can do.

[dimazest]

What happens for me in a similar setup.

I run a notebook, there are no cells that is executing (let say that it takes 10 minutes for a cell to run). I close the tab with the notebook, open it in another window and i'm able to continue.

If I execute a long running cell, close the tab and open it again, the execution of the cell will be stop (I suspect right after I close the tab). Is there a way to tell jupyter to keep execution even if nothing is connected to the notebook?

[Carreau]

If I execute a long running cell, close the tab and open it again, the execution of the cell will be stop (I suspect right after I close the tab). Is there a way to tell jupyter to keep execution even if nothing is connected to the notebook?

the execution of the cell is likely not stopped, just the callback that handle the reply get destroyed with the windows context. We[1] were actually discussing that this morning, but it will take some time to fix/complex to fix correctly.

[1] We in this context, is Me, Fernando, Jason Grout, Sylvain Corlay and a few folks at San Francisco Bloomberg offices.

[dimazest]

Great, thanks for the attention!

[swathimocharla]

Carreau, do you have any update that could help with this situation. I am running a notebook and am running into the same issue. Is there any workaround that you can suggest to keep the ping from timing out.

[Carreau]

Is there any workaround that you can suggest to keep the ping from timing out.

This is not a timeout. What I suggest is to separate the function that do calculation from the function that show results. You can use Futures also.

Run the computation, and disaplay the result only once connected again.

[rhazegh]

If I execute a long running cell, close the tab and open it again, the execution of the cell will be stop (I suspect right after I close the tab). Is there a way to tell jupyter to keep execution even if nothing is connected to the notebook?

the execution of the cell is likely not stopped, just the callback that handle the reply get destroyed with the windows context. We[1] were actually discussing that this morning, but it will take some time to fix/complex to fix correctly.

[1] We in this context, is Me, Fernando, Jason Grout, Sylvain Corlay and a few folks at San Francisco Bloomberg offices.

@Carreau Are you guys still working on this issue or it is resolved?

[Carreau]

It's not resolved. There is still a lot of work to do for this to be functional. It's also not not something we have enough manpower to actively work on. Even the way we want to tackle this is unsure.

[BanuSelinTosun]

I have been seeing the similar issue. I have my jupyter lab running in the tmux session, the cell takes ~ 30-60 min (loading a file, or stuff like that); I close my laptop, go home; open my laptop, reconnect to the jupyter lab and see this websocket time out or it happens when I away from my computer when the computation is happening and but if my laptop goes to sleep, the websocket error kills the notebook. Is there any solutions for this? Is it still being worked on? Or is it actually not something that has any priority?