Notebook: Linking Jupyter kernelspec to Anaconda Python

[irenatfh]

I've actually asked this question at Stack Overflow, but thought this might be a more appropriate venue. Apologies if this is inappropriate.

So, here's my issue, slightly elaborated for this more specific audience.

I'm using Anaconda to manage both Python and Jupyter. That is:

>> which python
>> /home/.../software/anaconda3/bin/python

and

>> which jupyter
>> /home/.../software/anaconda3/bin/jupyter

But Jupyter's python kernel seems to be pointing to a system version of Python rather than my local version through Anaconda, since the sys.path is different in a Jupyter Python 3 notebook. Also, jupyter kernelspec list gives the following:

 Available kernels:
   ir         /usr/local/share/jupyter/kernels/ir
   matlab     /usr/local/share/jupyter/kernels/matlab
   python3    /usr/local/share/jupyter/kernels/python3

This doesn't seem altogether surprising since the docs say in section 1.5.5:

By default, kernel specs will go in a system-wide location (e.g. /usr/local/share/jupyter/kernels). If doing a --user install, the kernel specs will go in the JUPYTER_DATA_DIR location.

For personal sanity and organization, I want the version of Python that I use in the command line to be the same that is accessed in Jupyter, which as a reminder was installed through Anaconda for my user account on the computer. As a result, I think that what I should do is change my jupyter kernelspec list for python3 so that it points to my desired Anaconda python version, i.e. /home/.../software/anaconda3/bin/python. My questions are: 1) is that indeed the best solution for my stated preferences, 2) is there any chance that the command jupyter kernelspec list is showing kernels for a globally installed version jupyter, rather than my own version, and 3) how do I actually change my jupyter kernelspec entry for python3? Not sure if this will come up, but I want to clarify that I don't want to be using virtual environments--I want the default to be same version of Python across both the command line and Jupyter.

[takluyver]

Short answer: if you delete /usr/local/share/jupyter/kernels/python3, Jupyter should find a default Python kernel using the same Python running Jupyter itself (i.e. Anaconda).

---

If you just want to fix it, that should be all you need. If you're interested in what's going on, keep reading.

The path you see from jupyter kernelspec list is not actually an executable, it's a folder containing a kernel.json which tells Jupyter how to start the kernel. That file will look something like this:

{
 "display_name": "Python 3", 
 "language": "python", 
 "argv": [
  "/usr/local/bin/python3", 
  "-m", 
  "ipykernel_launcher", 
  "-f", 
  "{connection_file}"
 ]
}

There is a search path of different directories which a kernelspec can be in. The priority order is user, environment, system. The first one containing python3/kernel.json defines the python3 kernel. As a special case for Python, if no python3 kernelspec is found and it can import ipykernel, it offers a python3 kernel running on the same Python executable as the notebook server.

This mechanism may be extended in the future: I have a plan to make it easier to add other kernel providers, e.g. conda environments.

[irenatfh]

Thank you so much!!! Deleting /usr/local/share/jupyter/kernels/python3 worked a treat, and your explanation was also very helpful for future reference.

[tdpetrou]

@takluyver Thank you very much for this answer. It prompted me to create this answer to a very popular stack overflow question. It seems like deleting the User 'python3' kernel should be the canonical approach to working in jupyter notebooks in different environments. Creating a new kernel accessible from everywhere (as is the top voted answer in the SO question) doesn't appeal to me.

I also created this issue #4447 to get more guidance on utilizing a canonical approach to using notebooks within a conda environment.

If you had any input on the SO answer or that GH issue, that would be great. Thanks!

[cpatdowling]

@takluyver Also thank you for this answer. The Anaconda documentation on working with notebook kernels is somewhat obfuscated by their GUI instructions, chaff posts on Medium, and unclear questions on SO.

Wanted to keep access to my machine's native kernels while executing precise control over what kernels Jupyter sees.