Notebook: Jupyter not recognizing packages

[ericmjl]

Hi!

I've noticed a quirky bug yesterday with conda 4.2.13 and notebook 5.0. The tl;dr version is cannot find package in environment even after conda install.

I had an environment setup and activated, named bayesian, and forgot to install PyMC3. I thus decided to install it after-the-fact. With bayes activated:

$ source activate bayesian

I installed PyMC3.

$ conda install pymc3

It was installed from conda-forge. I did this all with a Jupyter notebook running in the same activated environment.

Previous behaviour was such that I could immediately go back into the Jupyter notebook and without restarting the kernel, find the PyMC3 package installed and immediately start using it.

Yesterday's behaviour was markedly different: PyMC3 could not be found.

It took doing conda env update -f environment.yml, where PyMC3 was added as a required package there, to get it installed.

Any ideas why this is happening? Has the default behaviour of Jupyter changed? Is this related to this comment?

Thanks in advance!

[takluyver]

I don't think anything has changed in Jupyter relating to this.

[ericmjl]

@takluyver that's interesting, conda maintainers have said it doesn't have anything to do with them either. Any suggestions on how I could start debugging this?

[blink1073]

@ericmjl, does !which python in the notebook point to that conda environment?

[ericmjl]

Here's a few diagnostics that I've tried, based on @blink1073's comment. In this case, the environment is called nams, named after my tutorial Network Analysis Made Simple.

!which ipython
/Users/ericmjl/anaconda/envs/nams/bin/ipython

!which jupyter
/Users/ericmjl/anaconda/envs/nams/bin/jupyter

import sys
sys.executable
'/Users/ericmjl/anaconda/bin/python'

!which python
/Users/ericmjl/anaconda/envs/nams/bin/python

[ericmjl]

The result of another thing I tried: with the environment activated, I opened up IPython, and tried to import a package that was failing to import in Jupyter:

In [1]: import nxviz

In [2]: !which python
/Users/ericmjl/anaconda/envs/nams/bin/python

In [3]: !which ipython
/Users/ericmjl/anaconda/envs/nams/bin/ipython

In [4]: !which jupyter
/Users/ericmjl/anaconda/envs/nams/bin/jupyter

In [5]: paste
import sys
sys.executable

## -- End pasted text --
Out[5]: '/Users/ericmjl/anaconda/envs/nams/bin/python'

In this case, IPython succeeds in importing nxviz, while Jupyter Notebook failed in importing.

I'm starting to suspect the kernel is wrong? Previously, in the drop-down menu Kernel > Change Kernel, I would find independent kernels for each conda environment. Now, I'm not seeing independent kernels per environment. Is this correct?

On the other hand, sys.executable is different for Jupyter Notebook and IPython, even though they're launched from the same environment. Is there something I'm missing here?

[ericmjl]

Okay, this is quite interesting!

It had to do with the ability to find different kernels.

Once I installed Jupyter Environment Kernels and enabled it, this problem went away. I was able to see my different conda kernels under Kernel > Change Kernel, and once I switched it to my nams kernel, everything was fine.

Not sure why the default behaviour before (Jupyter auto-recognized my conda environment IPython kernels before) went away, but the sys.executable issue was the key hint that caught my attention.

[blink1073]

You can check for the installed kernels using jupyter kernelspec list. Most likely there wasn't an ipykernel installed in the environment, but installing the environment kernels took care of it.

[ericmjl]

@blink1073 Thanks for the pointer! How do I ensure that an ipykernel is guaranteed to be installed in a new environment? Or is it taken care of when I install Jupyter Notebook?

[blink1073]

It should be taken care of if you install an environment with notebook, it looks like something had gone awry in that env.

[heroxbd]

I am sure this is a conda bug. Where did you discuss this with the conda maintainers?

[takluyver]

I think this was environment related confusion rather than a conda bug. sys.executable shows which Python you're running; if that's not in the env you think you're in, then it won't find packages in that env.

[ericmjl]

For anybody interested in a solution I have more or less permanently adopted, it is to do the following:

1. Have one conda environment per project.
2. Install Jupyter (and/or jupyterlab) into every conda environment I have. (I take advantage of .condarc to ensure this happens.)
3. Run the following commands after creating each environment.

source activate {my-env-no-braces}
python -m ipykernel install --user --name {my-env-no-braces}

[leogodin217]

For anybody interested in a solution I have more or less permanently adopted, it is to do the following:

1. Have one conda environment per project.
2. Install Jupyter (and/or jupyterlab) into every conda environment I have. (I take advantage of .condarc to ensure this happens.)
3. Run the following commands after creating each environment.

source activate {my-env-no-braces}
python -m ipykernel install --user --name {my-env-no-braces}

To add a little bit to this, if you don't install Jupyter in your activated environment, it runs from the system install of python. That is why it is seeing a different environment.