Nixpkgs: Jupyter Notebook has no kernels with Python 2

[abbradar]

Issue description

Ditto.

Steps to reproduce

1. Build python2.buildEnv.override { extraLibs = with python2.pkgs; [ notebook ]; }; (you'll need ignoreCollisions; maybe that's the problem but it doesn't seem so);
2. Run jupyter-notebook;
3. Try to create new notebook; see that there are no kernels available.

Technical details

• Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion) c05ac3ea126300ee65345bd0b07ab212aa9b5f6a

[FRidh]

Maybe caused by 1f9bd9246b1ea5af599a94a5104f3f207322be3e?

[abbradar]

Shouldn't be if I understand correctly -- it works with Python 3.

EDIT: I'll try reverting it later.

[FRidh]

Just tested it, and its unrelated.

[abbradar]

I tackled it today to no avail. --debug is fairly useless too. We may need someone knowledgeable about how Jupyter finds its kernels in the first place...

[FRidh]

On 17.03 I get:

>>> from ipykernel.kernelspec import RESOURCES
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/8r6ffh4n5kwvlh4q4cpvsvj97gsqcys7-python2.7-ipykernel-4.5.2/lib/python2.7/site-packages/ipykernel/__init__.py", line 2, in <module>
    from .connect import *
  File "/nix/store/8r6ffh4n5kwvlh4q4cpvsvj97gsqcys7-python2.7-ipykernel-4.5.2/lib/python2.7/site-packages/ipykernel/connect.py", line 13, in <module>
    from IPython.core.profiledir import ProfileDir
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/__init__.py", line 48, in <module>
    from .core.application import Application
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/application.py", line 25, in <module>
    from IPython.core import release, crashhandler
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 28, in <module>
    from IPython.core import ultratb
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/ultratb.py", line 128, in <module>
    from IPython.utils.terminal import get_terminal_size
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/utils/terminal.py", line 22, in <module>
    from backports.shutil_get_terminal_size import get_terminal_size as _get_terminal_size
ImportError: No module named shutil_get_terminal_size

after

$ nix-shell -p python2Packages.notebook -I nixpkgs=.

This test was mentioned at https://github.com/jupyter/notebook/issues/1280#issuecomment-203194205.

[FRidh]

Creating an environment fails because of a collision as you mentioned:

$ nix-shell -p 'python2.withPackages(ps: with ps; [notebook])' -I nixpkgs=.
these derivations will be built:
  /nix/store/caa2w3vx83imyl0jz1azwib444lzp9s9-python-2.7.13-env.drv
building path(s) ‘/nix/store/xvramw3q56h1jgi8m0r9wb20laadx76r-python-2.7.13-env’
collision between `/nix/store/dk6sbn8032p8b6vfxrs1p08ggb6sfgfg-python2.7-backports.ssl_match_hostname-3.5.0.1/lib/python2.7/site-packages/backports/__init__.py' and `/nix/store/as2lqqk8xx28nwx58ab7vsw3h4zhd38f-python2.7-backports.shutil_get_terminal_size-1.0.0/lib/python2.7/site-packages/backports/__init__.py'
builder for ‘/nix/store/caa2w3vx83imyl0jz1azwib444lzp9s9-python-2.7.13-env.drv’ failed with exit code 25
error: build of ‘/nix/store/caa2w3vx83imyl0jz1azwib444lzp9s9-python-2.7.13-env.drv’ failed
/run/current-system/sw/bin/nix-shell: failed to build all dependencies

Ignoring collisions or setting a priority for one of the backport packages doesn't help either.

[abbradar]

Argh, so it's ignoreCollisions after all.

EDIT: I haven't noticed your last post. I meant that it's the conflicting package that is mentioned in the ImportError.

[FRidh]

$(nix-build -A python2Packages.notebook)/bin/jupyter-notebook

fails as well, so the issue also exists when we use the wrapper instead of a buildEnv.

[abbradar]

Hm, I'm not sure how'd that work with sys.path if there are two conflicting __init__.py's. I think we can look at how those two backports_ are packaged elsewhere -- surely conflicts are resolved?...

[FRidh]

Both contain

# This is a Python "namespace package" http://www.python.org/dev/peps/pep-0382/
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

which is what they should.

Possibly related: https://github.com/NixOS/nixpkgs/issues/22319

EDIT: the PEP 382 was rejected.

Anyway

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

is the (older) recommended method.

[abbradar]

I have a very hacky idea -- try to force this file to be a copy, not a symlink, and see what happens with buildEnv. Maybe __path__ is symlink-resolved?

[FRidh]

We could try that for buildEnv, but we should also have a solution for the wrapper.

[FRidh]

$ nix-shell -p python2.pkgs.notebook -I nixpkgs=. --run "ipython kernel install --user"
Traceback (most recent call last):
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/bin/.ipython-wrapped", line 8, in <module>
    from IPython import start_ipython
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/__init__.py", line 48, in <module>
    from .core.application import Application
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/application.py", line 25, in <module>
    from IPython.core import release, crashhandler
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 28, in <module>
    from IPython.core import ultratb
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/ultratb.py", line 128, in <module>
    from IPython.utils.terminal import get_terminal_size
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/utils/terminal.py", line 22, in <module>
    from backports.shutil_get_terminal_size import get_terminal_size as _get_terminal_size
ImportError: No module named shutil_get_terminal_size

In this case the backport is both in PYTHONPATH and added to site in the first lines of the script.
The following does succeed:

$ nix-shell -p python2.pkgs.ipython --run "python -c 'from backports.shutil_get_terminal_size import get_terminal_size'"

[FRidh]

Running ipython works fine. However, if we run ipython kernel

$ nix-shell -p python2.pkgs.ipython --run "ipython kernel"
Traceback (most recent call last):
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/bin/.ipython-wrapped", line 12, in <module>
    sys.exit(start_ipython())
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/__init__.py", line 119, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/nix/store/7gmqr8mh9ldzmcw49b497skq9maqn9mp-python2.7-traitlets-4.3.1/lib/python2.7/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-109>", line 2, in initialize
  File "/nix/store/7gmqr8mh9ldzmcw49b497skq9maqn9mp-python2.7-traitlets-4.3.1/lib/python2.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 300, in initialize
    super(TerminalIPythonApp, self).initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/nix/store/7gmqr8mh9ldzmcw49b497skq9maqn9mp-python2.7-traitlets-4.3.1/lib/python2.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/application.py", line 446, in initialize
    self.parse_command_line(argv)
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 295, in parse_command_line
    return super(TerminalIPythonApp, self).parse_command_line(argv)
  File "<decorator-gen-4>", line 2, in parse_command_line
  File "/nix/store/7gmqr8mh9ldzmcw49b497skq9maqn9mp-python2.7-traitlets-4.3.1/lib/python2.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/nix/store/7gmqr8mh9ldzmcw49b497skq9maqn9mp-python2.7-traitlets-4.3.1/lib/python2.7/site-packages/traitlets/config/application.py", line 514, in parse_command_line
    return self.initialize_subcommand(subc, subargv)
  File "/nix/store/qzjdh37vpaiyrdv9h3iw2cxqbpaiixlr-python2.7-ipython-5.2.1/lib/python2.7/site-packages/IPython/core/application.py", line 236, in initialize_subcommand
    return super(BaseIPythonApplication, self).initialize_subcommand(subc, argv)
  File "<decorator-gen-3>", line 2, in initialize_subcommand
  File "/nix/store/7gmqr8mh9ldzmcw49b497skq9maqn9mp-python2.7-traitlets-4.3.1/lib/python2.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/nix/store/7gmqr8mh9ldzmcw49b497skq9maqn9mp-python2.7-traitlets-4.3.1/lib/python2.7/site-packages/traitlets/config/application.py", line 445, in initialize_subcommand
    subapp = import_item(subapp)
  File "/nix/store/s4r7ml92hg3fd8cx1c4lgqiaqy85vm7h-python2.7-ipython_genutils-0.1.0/lib/python2.7/site-packages/ipython_genutils/importstring.py", line 31, in import_item
    module = __import__(package, fromlist=[obj])
ImportError: No module named ipykernel.kernelapp

This is correct. ipython itself has no dependency on the kernel ipykernel (neither in Nixpkgs nor upstream) and we also cannot add it because it would create a circular dependency.

[abbradar]

This should be OK by itself I think, because ipykernel is dependency of notebook (and it works with Python 3 after all).

[FRidh]

Upgrading to the latest versions doesn't solve the issue either: https://github.com/NixOS/nixpkgs/pull/23885

[abbradar]

I keep thinking that there is some meddling with sys.path or something in ipython kernel; in ipython importing this module or even import IPython works.

[abbradar]

Even funnier, from ipykernel.kernelspec import RESOURCES works in a buildEnv, but Jupyter still sees no kernels.

[FRidh]

So this line does work

$ nix-shell -p python2.pkgs.ipython -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"

but this one doesn't:

$ nix-shell -p python2.pkgs.notebook -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named shutil_get_terminal_size

even though it is there

$ nix-store -qR $(nix-build -A  python2.pkgs.notebook) | grep backports
/nix/store/dk6sbn8032p8b6vfxrs1p08ggb6sfgfg-python2.7-backports.ssl_match_hostname-3.5.0.1
/nix/store/wj4rzlll9faxrp2c7m1m10ak8g2n2cap-python2.7-backports_abc-0.4
/nix/store/as2lqqk8xx28nwx58ab7vsw3h4zhd38f-python2.7-backports.shutil_get_terminal_size-1.0.0

EDIT:

ipython has only one backports package:

$ nix-store -qR $(nix-build -A  python2.pkgs.ipython) | grep backports
/nix/store/as2lqqk8xx28nwx58ab7vsw3h4zhd38f-python2.7-backports.shutil_get_terminal_size-1.0.0

Namespace packages do function:

$ nix-shell -p python2.pkgs.backports_shutil_get_terminal_size python2.pkgs.backports_ssl_match_hostname -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"
$ nix-shell -p python2.pkgs.backports_shutil_get_terminal_size python2.pkgs.backports_ssl_match_hostname -I nixpkgs=. --run "python -c 'import backports.ssl_match_hostname'"

Changing order of the 2 packages also has no (negative) effect.

What is also weird, is that the following happens also with Python 3:

$ nix-shell -p python3.pkgs.notebook -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'backports.shutil_get_terminal_size'

[abbradar]

I think what happens is it takes the first available backports (in this case, ssl_match_hostname) and looks no further; it doesn't actually try to search in other ones. This seems logical but I have absolutely no idea how can we circumvent that.

[FRidh]

It must happen then in the code that is inserted in the script, because

$ nix-shell -p python2.pkgs.backports_shutil_get_terminal_size python2.pkgs.backports_ssl_match_hostname -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"

does function.

EDIT:

No, it doesn't necessarily have to be cause by the code that is inserted, because

$ nix-shell -p python3.pkgs.notebook -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'backports.shutil_get_terminal_size'

fails as well.

[abbradar]

Theory defeated; I thought ordering would matter here but:

$ nix-shell -p python2.pkgs.backports_ssl_match_hostname python2.pkgs.backports_shutil_get_terminal_size -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"

Works too.

EDIT: Oh, or do you think sys.path works differently than PYTHONPATH here? Could be possible...

[abbradar]

For Christ's sake... found it:

$ nix-shell -p python2.pkgs.configparser python2.pkgs.backports_ssl_match_hostname python2.pkgs.backports_shutil_get_terminal_size -I nixpkgs=. --run "python -c 'import backports.shutil_get_terminal_size'"

This offender is used by notebook and is somehow breaking other backports.

[FRidh]

Looks like configparser still uses an older approach to namespace packages:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

try:
    import pkg_resources
    pkg_resources.declare_namespace(__name__)
except ImportError:
    pass

Here it tries to declare, which is because setup.py has

    namespace_packages=['backports'],

Removing those extra lines (try...except) as well as the above fixes the importing for me.

[FRidh]

and the notebook kernel is now working :)

[FRidh]

Fix in https://github.com/NixOS/nixpkgs/pull/23888

[abbradar]

Closed by https://github.com/NixOS/nixpkgs/pull/23885

[yegle]

I'm still able to reliably reproduce this issue with the following default.nix file. If I remove matplotlib then everything would be fine.

I wonder if this is really fixed?

with import <nixpkgs> {};

let
  spark_no_mesos = spark.override { mesosSupport = false; };
in
stdenv.mkDerivation rec {
  name = "pyspark";
  env = buildEnv {
    name = name;
    paths = buildInputs;
  };
  buildInputs = [
    python27Packages.ipython
    python27Packages.notebook
    python27Packages.pandas
    python27Packages.matplotlib
    spark_no_mesos
  ];
  shellHook = ''
    export PYSPARK_DRIVER_PYTHON=$(which jupyter-notebook)
  '';
}

$ nix-shell
[nix-shell:~]$ ipython
Traceback (most recent call last):
  File "/nix/store/dsxxakcw9yzgb37c8k1nb0c8hmwhv0q1-python2.7-ipython-5.3.0/bin/.ipython-wrapped", line 8, in <module>
    from IPython import start_ipython
  File "/nix/store/dsxxakcw9yzgb37c8k1nb0c8hmwhv0q1-python2.7-ipython-5.3.0/lib/python2.7/site-packages/IPython/__init__.py", line 48, in <module>
    from .core.application import Application
  File "/nix/store/dsxxakcw9yzgb37c8k1nb0c8hmwhv0q1-python2.7-ipython-5.3.0/lib/python2.7/site-packages/IPython/core/application.py", line 25, in <module>
    from IPython.core import release, crashhandler
  File "/nix/store/dsxxakcw9yzgb37c8k1nb0c8hmwhv0q1-python2.7-ipython-5.3.0/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 28, in <module>
    from IPython.core import ultratb
  File "/nix/store/dsxxakcw9yzgb37c8k1nb0c8hmwhv0q1-python2.7-ipython-5.3.0/lib/python2.7/site-packages/IPython/core/ultratb.py", line 128, in <module>
    from IPython.utils.terminal import get_terminal_size
  File "/nix/store/dsxxakcw9yzgb37c8k1nb0c8hmwhv0q1-python2.7-ipython-5.3.0/lib/python2.7/site-packages/IPython/utils/terminal.py", line 22, in <module>
    from backports.shutil_get_terminal_size import get_terminal_size as _get_terminal_size
ImportError: No module named shutil_get_terminal_size

[FRidh]

@yegle that's due to how you create your "environment". Where did you read to do it like this? During build-time all packages are there, but buildEnv does not create the necessary wrappers and therefore during runtime Python cannot find the required Python packages.

[yegle]

/facepalm

Never realize I was doing it wrong. I followed https://ariya.io/2016/06/isolated-development-environment-using-nix and wrote the file.

What would be the right way to do it then? (Not trying to hijack this thread but my nix force is weak)

[FRidh]

search for withPackages on https://nixos.org/nixpkgs/manual/#python

[FRidh]

In the example you referenced it works because nix-shell is used which runs a hook that builds up PYTHONPATH from the buildInputs. This is/was a common way of working, but should not be recommended.

[yegle]

Finally figured it out. Thanks @FRidh!

Post here just in case anyone ran into the problem:

with import <nixpkgs> {};

let
  spark_no_mesos = spark.override { mesosSupport = false; };
in
stdenv.mkDerivation rec {
  name = "pyspark";
  env = buildEnv {
    name = name;
    paths = buildInputs;
  };
  buildInputs = [
    spark_no_mesos

    (python.buildEnv.override {
        ignoreCollisions = true;
        extraLibs = with pythonPackages; [
            ipython
            notebook
            pandas
            matplotlib
        ];
    })
  ];
  shellHook = ''
    export PYSPARK_DRIVER_PYTHON=$(which jupyter-notebook)
  '';
}