Notebook: Preprend env's bin dir to PATH when loading a kernel

[tritemio]

This is more of a question of best-practice, not even sure this is the right repo to ask. Feel free to close if OOT.

I am running pydot in a conda environment which has an ipython kernel configured. pydot requires accessing the executable dot that is installed in the bin dir of the environment.
I run the notebook in the base environment, with a kernel for each environment.

If I try to use pydot from the notebook (with the right kernel) I get an error that the dot executable is not found. That's because the kernel does not add the environment's bin folder to the PATH.

I read that I can define a env variable in the kernelspec. But can I modify an existing variable, for example pre-pending a path to the PATH variable? Other ideas on how to modify the PATH for a single ipython kernel?

[tritemio]

This issue is the cause of https://github.com/tensorflow/tensorflow/issues/27304#.

I believe that also #4518 is related.

I rephrase the problem. When you activate an env in the terminal, the PATH is prepended with the environment's bin dir. Instead, if your run the notebook from a base environment and load the kernel of another env, the PATH is not changed.

This causes issues with tools that need to call executables and expect them to be in the PATH. This is the case for pydot (requires dot executable) and the %tensorboard magic (requires the tensorboard executable).

[tritemio]

QUESTION: Is there a way to create a kernel spec which prepends the env's bin dir to the PATH?

If yes, this would fix this issue.

[kevin-bates]

Not sure if this helps, but you could augment the argv stanza of your kernelspec to use launch the kernel from a script and modify PATH prior to its launch...

{
 "env": {
  "MY_ENV_BIN_DIR": "/opt/anaconda/envs/my_special_env/bin"
 },
 "argv": [
  "/usr/local/share/jupyter/kernels/python3_with_env/prepend_and_launch.sh",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 with env",
 "language": "python"
}

then, launch the kernel from...

#!/bin/bash
export PATH=${MY_ENV_BIN_DIR}:${PATH}
$*

with this as the result...

[tritemio]

Yes, it works as a stop-gap, thanks.

However it is cumbersome to repeat for each environment.

Ideally this should be done automatically when creating a kernel with some sort of python -m ipykernel install ...

[kevin-bates]

Agreed. Since this is an issue only when spanning envs, the "target" python env must then be determined.

Btw, here's a repo that, I believe, fits your bill: https://github.com/Cadair/jupyter_environment_kernels. It does this by implementing a version of KernelSpecManager that performs the auto-detection.

In addition, you might get better traction in https://github.com/ipython/ipykernel, perhaps by adding a --env installation option that then does the right thing during launch by capturing the env during install - or some such thing.

[wchargin]

Suggested change: $* should be "$@" (note quotes). See:
https://www.gnu.org/software/bash/manual/bashref.html#Special-Parameters

(It should probably also be exec "$@", unless there’s a good reason.)