nbconvert kernel_name does know about conda env kernels

[arsenovic]

i am using nbsphinx with conda, and ran into a problem (https://github.com/spatialaudio/nbsphinx/issues/90) . i find that the nbconvert is not finding my conda env kernels correctly, my kernels

(py35) > conda env list 
# conda environments:
#
py27                     /home/alex/anaconda3/envs/py27
py35                  *  /home/alex/anaconda3/envs/py35

but when i try to use nbconvert

jupyter nbconvert --to html --ExecutePreprocessor.kernel_name=py35 \
--execute source/tutorials/Calibration.ipynb

yields

jupyter_client.kernelspec.NoSuchKernel: No such kernel named py35

if i set the kernel to python, python2, or python3, it also gives a similar error. it appears as though the kernels are being found in ~/.local/share/jupyter/kernels/, which is not in-sync with conda envs.
note: the jupyter notebook is using my conda envs correctly.

[takluyver]

Anaconda adds some extension that changes how the notebook finds kernels, but it seemingly doesn't affect nbconvert or other things that start kernels. Their modification is a continuing annoyance to me.

[mpacer]

If you run jupyter kernelspec list what do you see?

Your environment names are not guaranteed to be the same as your kernel names, and their installed name might be different from their display name.

If you want to see their display names you can use jupyter kernelspec list --json | grep "display_name", but those won't appear in the same order as in jupyter kernelspec list.

[arsenovic]

(py35) > jupyter kernelspec list
Available kernels:
  python2    /home/alex/.local/share/jupyter/kernels/python2
  python3    /home/alex/.local/share/jupyter/kernels/python3

and

(py35) > jupyter kernelspec list --json | grep "display_name"
     "display_name": "IPython (Python 2)",
     "display_name": "Python 3",

but when the kernel in the notebook metadata does not align with these. it aligns with the conda envs. for example my notebook has in it's metadata.

 "kernelspec": {
     "display_name": "Python [conda env:py35]",
     "language": "python",
     "name": "conda-env-py35-py"

i could alter nbsphinx to set kernel_name to nbconvert.preprocessors.ExecutePreprocessor, but this is still a bug.

[mpacer]

So jupyter nbconvert --to html --ExecutePreprocessor.kernel_name=python3 --execute source/tutorials/Calibration.ipynb

and

jupyter nbconvert --to html --ExecutePreprocessor.kernel_name="conda-env-py35-py" --execute source/tutorials/Calibration.ipynb

(or jupyter nbconvert --to html --ExecutePreprocessor.kernel_name=conda-env-py35-py --execute source/tutorials/Calibration.ipynb but I think that you might need the quotes because of the dashes)

both don't work?

[arsenovic]

 jupyter nbconvert --to html --ExecutePreprocessor.kernel_name=python3 --execute source/tutorials/Calibration.ipynb

works... but i had to manually edit /home/alex/.local/share/jupyter/kernels/python3/kernel.json, to contain "/home/alex/anaconda3/envs/py35/bin/python",

using conda-env-py35-py does not work with or without quotes

[mpacer]

Did you try to run the conda-env-py35-py command from the root environment as well as from within py35?

I'm assuming you mean that you changed the first element in the "argv" keyed list to /home/alex/.local/share/jupyter/kernels/python3/kernel.json. Do you remember what it was before?

It sounds like this is a conda × kernel issue (I've been wrestling with them myself lately, so I may also be overestimating the likelihood of this).

I'm trying to think through this, since I ran into something I'll have to write up later that relates to this…but check the values of jupyter kernelspec list against the name of each entry in the resulting json along with first values for the "argv" key found in jupyter kernelspec list --json(sorry can't think of a way to grep for this nicely like I could with the display name).

One thing to note, is that there's only one kernel available with a particular name at a time (at least AFAIU this holds from the perspective of nbconvert maybe not for the notebook, qualifications to follow). I think that is exactly what nb_conda_kernels is trying to make possible while hiding the naming differences. But, as @takluyver mentioned, it may only handle this for the notebook and not for other kernel instantiating processes (like nbconvert) because we rely on the standard KernelSpecManager rather than the overwitten one used by the notebook when nb_conda_kernels is installed. There's a nice discussion of some of this at https://github.com/Anaconda-Platform/nb_conda_kernels/issues/42. I don't think we want to switch to using their KernelManager (if only because we don't want to require people to use nb_conda_kernels).

I'm still trying to figure out how all the kernel stuff is being managed, so I apologise for not being able to give you a cleaner answer. I thank you for working through this with me and look forward to reaching a resolution.

[arsenovic]

so i have solved the problem by adding a kernel_name option to nbsphinx, https://github.com/spatialaudio/nbsphinx/pull/91. which is useful without this bug, because a lot of times, my notebooks have random kernels, or someone else will edit a notebook and it will be looking for their kernel on my system.

the value of 'argv' in kernel.json was pointing to an old conda env, which no longer existed. it seems like the jupter kernelspec list output is not reacting to the conda envs .

thanks for all your help,

[takluyver]

it seems like the jupter kernelspec list output is not reacting to the conda envs .

Anaconda modifies the notebook to find its envs as kernels. This isn't how our code works, and their modification doesn't apply to anything other than the notebook server, which results in precisely this kind of confusion.

[mpacer]

@arsenovic Great point, I'm starting to put together best practice suggestions and as you point out, the existence of kernels on others' systems suggests directly setting the kernel_name parameter for that kind of use case is good general best practice to engage in. I'll close this issue once I've put that in some docs somewhere.

[tritemio]

I was also hit this bug when executing notebooks with nbrun. In this case, if the notebook specifies the default kernel displayed as "Python [Default]" then the notebook is executed with the python of the current environment. But if the notebook specifies a kernel from an environment the execution will fail as reported by @arsenovic.

@mpacer, as a workaround, to run notebooks from the command line using the kernel from a specific conda environment, I need to enter the environment and then specify the kernel name as python3 (for python 3.x) or python2 (for python 2.7).

For example, after activating an environment run:

(myconda-env-py35) ~$ jupyter nbconvert --ExecutePreprocessor.kernel_name=python3 notebook.ipynb

This will use whatever python is installed in the current environment (assuming python 3.x) regardless of the notebook metadata.

[scottdraves]

Is there any way to fix this in the nb_conda_kernels extension? could it use some other metadata field?

[takluyver]

We're working on redesigning the way kernel discovery works so that it behaves better with environments (see https://github.com/jupyter/jupyter_client/pull/308 ), but that's definitely more of a long term solution.

[RafalSkolasinski]

Would installing conda kernel explicitly be a good temporarily solution?
Something like

path/to/python -m ipykernel install --user --name conda-env-py35-py

[cyril-p]

For all of those who have the same problem, this is the solution:

$ # source activate # non-windows
$ activate # windows
(envname) $ python -m ipykernel install --name

Then in your notebook, change the kernel to simply "envname".

Now back in the nbconver API, you should be able to run it, whithout specifying
"--ExecutePreprocessor.kernel_name="