Vscode-jupyter: Remote jupyter doesn't reconnect after computer sleeps

[8enmann]

Maybe related: microsoft/vscode-jupyter#822

Bug: Notebook Editor, Interactive Window, Editor cells

Steps to Reproduce:

1. Connect to a remote jupyter server
2. Sleep laptop
3. Try to run a cell
   Error: Cannot execute code, session has been disposed. Source: Python (Extension)

The workaround I use is close the notebook editor and reopen it, but this loses all my cells. Remote server still has the session.

Your Jupyter and/or Python environment

Version: 1.47.0
Commit: d5e9aa0227e057a60c82568bf31c04730dc15dcd
Date: 2020-07-09T08:01:54.115Z (2 days ago)
Electron: 7.3.2
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 19.5.0

• Jupyter server running: Remote
• Extension version: v2020.6.91350
• Setting python.languageServer: pylance
• Python and/or Anaconda version: Python 3.7.3
• Virtual environment: conda

Developer Tools Console Output

Error Python Extension: 2020-07-11 21:30:39: [Error: Kernel is dead at f._sendKernelShellControl (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/node_modules/@jupyterlab/services.js:3:510258) at f.sendShellMessage (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/node_modules/@jupyterlab/services.js:3:510027) at f.requestExecute (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/node_modules/@jupyterlab/services.js:3:512579) at _.requestExecute (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:1:535559) at _.requestExecute (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:460661) at f.P.generateRequest (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:436320) at f.P.handleCodeRequest (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:437313) at /Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:450604 at runMicrotasks () at processTicksAndRejections (internal/process/task_queues.js:85:5)]
t.log @ workbench.desktop.main.js:sourcemap:266
workbench.desktop.main.js:sourcemap:2947 Error: Cannot execute code, session has been disposed.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[DavidKutu]

Thanks for the issue @8enmann.

[8enmann]

Any updates?

[DavidKutu]

Hi @8enmann, this issue is parked for now, we'll come back to it if other people run into it.

In the mean time I have a question. You mentioned you loose your cells, does that happen even if you save the notebook?

[8enmann]

The problem is not that the cell goes away, it's that I need to restart the notebook in order to run cells since the remote session is disposed. Saving or not doesn't make a difference.

[DavidKutu]

I see, I'm glad there's a workaround. We're not planning to fix this one on our next release. We'll let the issue open for people to upvote it and will revisit it on triage after the august release.

[aray-openai]

I've got the same problem. Is there any way to get at the debug logs for reconnection -- it feels like I don't have good access to what the jupyter plugin is doing to try to reconnect.

[DavidKutu]

@aray-openai those should be on the webview developer tools:
ctrl + shift + P -> Developer: Open Webview Developer Tools

[shikiponn]

Hi vscode team,

I've run into this porblem too.
Resuming from sleep I see my vscode saying Cannot reconnect. Please reload the window. and I can no more access to python interactive where I was running my code.

[DavidKutu]

Hi @shikiponn, please share with us what your Python output tab on the Panel says when the issue happens.

[clayms]

Issue Type: Bug

Same basic problem. It happens when there is even the slightest interruption to my corporate VPN connection. It is so bad for my overseas colleagues, that despite me trying to get them to use the Python extension in VS Code, they refuse, as it is too frustrating to have the session get "disposed" every few minutes. Instead, they choose to use either Jupyter Labs or Notebooks, as those do not have any issue picking up right where it was before the network disruption.

The reported VS Code Python Extension error is: Error: Cannot execute code, session has been disposed.

Why is it that Jupyter Labs or Notebooks do not have any problems reconnecting to the Jupyter Session running on a remote computer after a slight network disruption and now the Python extension for VS Code cannot handle it?

This was not a problem a year or so ago. With the VS Code Python Extension, I used to be able to:

1. start a Jupyter session on a remote computer in VS Code at the office,
2. put my computer to sleep,
3. go home,
4. wake my computer up,
5. log into my corp VPN,
6. start running my code right where I left off in the VS Code Python Extension, everything in memory exactly as I left it.

Why and how was this broken? Why does this still work in Jupyter Labs or Notebooks?

Might be related to what IanMatthewHuff said in this issue comment:

"Jupyter Notebooks don't connect in the same way that we do so it is possible for them to make the connection and for us to not be able to."

Extension version: 2020.9.114305
VS Code version: Code 1.50.0 (93c2f0fbf16c5a4b10e4d5f89737d9c2c25488a3, 2020-10-07T06:10:52.432Z)
OS version: Windows_NT x64 10.0.18363

System Info

|Item|Value|
|---|---|
|CPUs|Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz (4 x 2496)|
|GPU Status|2d_canvas: enabled
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
opengl: enabled_on
protected_video_decode: unavailable_off
rasterization: enabled
skia_renderer: disabled_off_ok
video_decode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled|
|Load (avg)|undefined|
|Memory (System)|15.88GB (6.92GB free)|
|Process Argv|--crash-reporter-id f652528b-5678-4b12-9ca0-3de6b1e1135f|
|Screen Reader|no|
|VM|0%|

[rchiodo]

Why is it that Jupyter Labs or Notebooks do not have any problems reconnecting to the Jupyter Session running on a remote computer after a slight network disruption and now the Python extension for VS Code cannot handle it?

This is because of how we manage kernels. When our UI is disposed, we close the kernel connection. Whereas with jupyter you start a server and close kernels when you shutdown the server.

Likely what's happening when you put your computer to sleep, VS code is disposing of our Webview UI, causing our dispose method to be called.

In order to accommodate this we'd need some sort of kernel management UI (other than just closing windows). Something like this bug here:
https://github.com/microsoft/vscode-python/issues/14087

[clayms]

@rchiodo that is exactly the issue. If those features were added, I think that would solve this issue.

Concise description of my experience:
https://github.com/microsoft/vscode-python/issues/14087#issuecomment-705055809

I believe this issue could be rolled up into that issue/feature request.

[clayms]

I should note the usual way I run python.

From PowerShell on local computer:

ssh [email protected]

jupyter lab --ip=100.100.100.100

Then copy the displayed link into .code-workspace file:

    "settings":  {
        "python.dataScience.jupyterServerURI": "http://100.100.100.100:8888/?token=5555555555555555555555555"
     }

Then the VS Code Python Extension works great, at least until a slight network interruption or my local computer goes to sleep.

[8enmann]

Any updates on this? Blocking me from using it entirely.

[rchiodo]

Which part is blocking you? Your computer going to sleep and then reconnecting? Does starting the jupyter server yourself (using a remote connection to a localhost) resolve that problem?

(I haven't tried to see what happens with jupyter when a computer goes to sleep, not sure if the server stays running or not)

We're actively changing the way we connect to remote machines. Part of this will be some form of kernel management. We may also have the same kernel management for local kernels (although not sure how we decide when to kill the kernels).

[8enmann]

Yep, the blocker is that when I have a remote jupyter, when I sleep and wake, I lose all my jupyter state. This means I can't have a persistent session and have to rerun all the cells every time my laptop sleeps.

I just tested with a jupyter on localhost and it worked fine. Unfortunately I do a lot of stuff with GPUs, so I have to run on a remote connection.

Really appreciate your work here. Love VSCode!!

[rchiodo]

Remote kernels should be left running though. If you're using notebooks (and not the interactive window), it should reconnect to the original kernel when you reopen the notebook.

Unless you mean remote in the fact that you're using SSH (then the jupyter connection is actually local because it's started on the SSH machine next to where our extension is running).

Using SSH would require us to support not shutting down local kernels as well.

[8enmann]

I'm not sure what the interactive window is. I use Python: Create New Blank Jupyter Notebook, which I assume is a notebook. I never explicitly close the window on my machine, maybe electron is telling it I closed the window when I sleep my mac? When I unsleep, the window's still there with all the state in it.

The kernel does continue to run on my remote machine. I connect to it using HTTPS over VPN. For some reason VSCode can't reconnect to it.

Maybe it'd work better if I used eternal-terminal port forward from remote so VSCode thinks it's actually a local jupyter?

[rchiodo]

The kernel does continue to run on my remote machine. I connect to it using HTTPS over VPN. For some reason VSCode can't reconnect to it.

If you close the notebook and reopen it, does it reconnect? There's no logic for handling sleep (not sure VS code even has an API to tell us about that). However when reopening a notebook we attempt to reconnect to the original kernel. Or at least we should.

[rchiodo]

Maybe it'd work better if I used eternal-terminal port forward from remote so VSCode thinks it's actually a local jupyter?

Not sure what eternal-terminal port forward is. We don't need the server URI to be localhost. It should just work as long as you're connecting through the 'Specify local or remote Jupyter server for connections'

[michaelsanchez-pingidentity]

I am not using jupyter or python extensions in VSCode and I have the exact same issue. All my extensions are Javascript and React related. Steps to reproduce are mostly identical.

1. log on to Macbook Pro.
2. Start VPN client.
3. Launch VSCode
4. Open SSH connection with VSCode remote to dev server.
5. work work work.
6. Walk away for lunch or other and laptop goes to sleep.
7. When I return and wake it up, laptop still connected to VPN and everything else works.
8. VSCode lost connection. Any attempt to "Reconnect Now", "Reload window", "Retry", or even closing the remote connection and restarting all fail at connecting to any SSH host.

I have to kill my VPN connection and reconnect new to get VSCode to connect to any SSH host. I'm mostly just supplying further "evidence" to help out. A solution would be great but I've decided it's just part of my job description now. shrug

Version: 1.51.1
Commit: e5a624b788d92b8d34d1392e4c4d9789406efe8f
Date: 2020-11-11T01:11:34.018Z
Electron: 9.3.3
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 19.6.0

[joyceerhl]

@michaelsanchez-pingidentity I believe you are encountering a separate issue pertaining to the Remote-SSH extension. You might like to check out the Remote-SSH output tab in VSCode when trying to reconnect and see what the logs say, and please do consider filing a bug on their repo so their team can assist you.

[8enmann]

I believe it tries to reconnect after sleep and then gets
workbench.desktop.main.js:sourcemap:2947 Error: Cannot execute code, session has been disposed.