Vscode-jupyter: Kernel won't start (heartbeat). May be related to use of --system-site-packages when defining virtual environment

[mldot]

Bug: Notebook Editor, Interactive Window, Editor cells

Steps to cause the bug to occur

1. Open vscode and select python interpreter from virtual environment
2. Activate virtual env with ipykernel and jupyter library installed from integrated terminal.
3. Command Pallete > Python > Create New Jupyter Notebook
4. In first cell type print('Hello World') and execute

Actual behavior

Notification shows - Connecting to IPython kernel: Connecting to kernel....
After couple of minutes
'Timed out waiting to get a heartbeat from kernel process'

Expected behavior

Cell in notebook should get executed correctly.

Your Jupyter and/or Python environment

_Please provide as much info as you readily know_

• Jupyter server running: Local
• Extension version: 2020.8.103604
• VS Code version: 1.48.2
• Setting python.jediEnabled: false
• Setting python.languageServer: None
• Python and/or Anaconda version: 3.8.5 (No Anaconda)
• OS: Windows | Mac | Linux (distro): OSX 10.14.6 (MacOS Mojave)
• Virtual environment: virtualenv

Python Output

mldot-python-debug.log

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[mldot]

Same issue when trying to open .py file in Python Interactive window. Log attached.
python-logging-debug.log

[IanMatthewHuff]

@mldot Thanks for filing this and sorry about the issue.

I'm digging through the log now. Can you tell me, where is the python that you expect to be using with this virtualenv located? In the log I'm seeing a couple different paths listed:
~/Dev/Projects/python/_ml_master_/bin/python
/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/
/usr/local/opt/[email protected]/bin/python3.8

[mldot]

@IanMatthewHuff : /usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ is the base python home where HomeBrew installed python3.8.5. The other paths point to symlinks to this location. _ml_master_ is the virtualenv created that has all jupyter related packages including ipykernel. The path is activated and VS Code has it set in the status bar. I also saw the other paths in the log but couldn't figure out how it is resolving to those. Maybe the Jupyter setting is looking at base path ignoring the virtualenv paths.

Please see below:
(_ml_master_) [user@host:~/Dev/Projects/python/_ml_master_/bin]$ ls -l python
lrwxr-xr-x 1 user group 39 Aug 20 21:39 python -> /usr/local/opt/[email protected]/bin/python3.8
(_ml_master_) [user@host:~/Dev/Projects/python/_ml_master_/bin]$ cd /usr/local/opt/[email protected]/bin/
(_ml_master_) [user@host:/usr/local/opt/[email protected]/bin]$ ls -l python3.8
lrwxr-xr-x 1 user group 57 Jul 20 08:26 python3.8 -> ../Frameworks/Python.framework/Versions/3.8/bin/python3.8
(_ml_master_) [user@group:/usr/local/opt/[email protected]/bin]$

Jupyter and Ipykernel are installed under _ml_master_ virtualenv:
(_ml_master_) [user@host:/usr/local/opt/[email protected]/bin]$ pip show ipykernel
Name: ipykernel
Version: 5.3.4
Summary: IPython Kernel for Jupyter
Home-page: https://ipython.org
Author: IPython Development Team
Author-email: [email protected]
License: BSD
Location: ~/Dev/Projects/python/_ml_master_/lib/python3.8/site-packages
Requires: traitlets, jupyter-client, ipython, tornado, appnope
Required-by: qtconsole, notebook, jupyter, jupyter-console, ipywidgets

(_ml_master_) [user@host:/usr/local/opt/[email protected]/bin]$ pip show jupyter
Name: jupyter
Version: 1.0.0
Summary: Jupyter metapackage. Install all the Jupyter components in one go.
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: [email protected]
License: BSD
Location: ~/Dev/Projects/python/_ml_master_/lib/python3.8/site-packages
Requires: ipykernel, nbconvert, notebook, ipywidgets, jupyter-console, qtconsole
Required-by:

[mldot]

Some more details: Jupyter and ipykernel seems to work when I installed them under base enviornment. But in the notebook when I switch the interpreter to the _ml_master_ virtualenv it no longer now identifies the libraries installed in virtualenv. Looks like the notebook is ignoring virtualenv settings altogether. This is the reason it could not find the kernel installed in that environment.

import matplotlib.pyplot as plt

import numpy as np

ModuleNotFoundError Traceback (most recent call last)
in
----> 1 import matplotlib.pyplot as plt
2 import numpy as np

ModuleNotFoundError: No module named 'matplotlib

Jupyter_mat.log

[IanMatthewHuff]

@mldot thanks for the info. It does seem that something is up with that python env. Sorry about the questions, but issues like this can always be a bit hard to debug as they are very machine / environment specific and I don't have a repro to debug myself locally. I think that the next step here would be to determine if this is an issue with how VS Code is trying to activate your environment or an issue with just the environment in general.

Is this an environment that you were already using successfully outside of VS Code? If you have a terminal and you activate the ml_master virtualenv can you start up the python prompt and successfully import matplotlib or ipykernel? From your comment above I wasn't quite sure if the ModuleNotFoundErrors were coming from the VS Code scenario, but I assumed that they were.

[mldot]

@IanMatthewHuff Thanks for the follow-up.

Yes this environment works well with a stand alone jupyter notebook install. When I simply have a python .py file running in this environment in. VSCode it can successfully import the libraries. Also the python prompt works. It is only the vscode jupyter notebook itself that is somehow not able to resolve the environment settings.

Just to provide you with an update I was able to get vscode Jupyter notebook working with a new environment I created this morning. The only change was I didnot use the --system-site-packages setting when invoking virtualenv. The _ml_master_ env was created using this parameter which allowed me access to global packages in the parent repository without having to again duplicate installs. But somehow this setting is not working with the vscode python jupyter notebook. If I do need to run Jupyter notebook with vscode I believe for the time being I can use the new environment I created.

[IanMatthewHuff]

@mldot. Glad that you have an environment that you can move forward in for now. The additional info on --system-site-packages is interesting we can try setting up a repro with that parameter to see if we can create this issue ourselves to debug.

[VishalDhayalan]

I'm facing a similar issue except that for me it's "Failed to start a session for the Kernel 'Python 3.7.4 64-bit ('tf-2.3': virtualenv)'.
View Jupyter log for further details." I've also tried all the steps described here: https://stackoverflow.com/questions/58119823/jupyter-notebooks-in-vscode-does-not-use-active-virtual-environment
but to no avail...

the Jupyter Log doesn't have anything useful either:

[rchiodo]

@VishalDhayalan can you create a new issue and fill out the issue template? It might not be related and your output log would be super useful in diagnosing. Thanks.

[VishalDhayalan]

@VishalDhayalan can you create a new issue and fill out the issue template? It might not be related and your output log would be super useful in diagnosing. Thanks.

Done. Created issue microsoft/vscode-jupyter#551 Please check. Thank you