Nixpkgs: pipenv broken on nixos-19.09

Created on 11 Nov 2019 · 20Comments · Source: NixOS/nixpkgs

Describe the bug

I'm trying to pipenv install requests in a nix shell, but pipenv fails with ModuleNotFoundError: No module named 'pip._internal.main'.

To Reproduce
Steps to reproduce the behavior:

I have the following shell.nix, pointing to the most recent commit in nixos-19.09 branch:

{ pkgs ? import (builtins.fetchTarball {
    url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
    sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";
  }) {}
}:
pkgs.mkShell {
  name = "dev-shell";
  buildInputs = [ pkgs.python37Full pkgs.pipenv];
}

I run nix-shell --pure --run 'pipenv install requests'
I see pipenv failing:

$ nix-shell --pure --run 'pipenv install requests'
Installing requests…
Adding requests to Pipfile's [packages]…
✔ Installation Succeeded 
Installing dependencies from Pipfile.lock (444a6d)…
An error occurred while installing certifi==2019.9.11 --hash=sha256:e4f3620cfea4f83eedc95b24abd9cd56f3c4b146dd0177e83a21b4eb49e21e50 --hash=sha256:fd7c7c74727ddcf00e9acd26bba8da604ffec95bf1c2144e67aff7a8b50e6cef! Will try again.
An error occurred while installing chardet==3.0.4 --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691! Will try again.
An error occurred while installing idna==2.8 --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c! Will try again.
An error occurred while installing requests==2.22.0 --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31! Will try again.
An error occurred while installing urllib3==1.25.6 --hash=sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398 --hash=sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:00
Installing initially failed dependencies…
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:02
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/cli/command.py", line 254, in install
[pipenv.exceptions.InstallError]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 862, in do_install_dependencies
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, False, failed_deps_queue, retry=False)
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: []
[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/Users/zupo/play/.venv/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]

Expected behavior

If, in my shell.nix, I use the latest commit from master branch, then pipenv works as expected:

-     url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
-     sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";
+     url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
+     sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";

$ nix-shell --pure --run 'pipenv install requests'
Installing requests…
Adding requests to Pipfile's [packages]…
✔ Installation Succeeded 
Installing dependencies from Pipfile.lock (444a6d)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Additional context

MacOS Mojave, nix 2.3.1.

I found the following two PRs on master that touch pkgs/development/tools/pipenv/default.nix:

I tried cherry-picking them on top of nixos-19.09 and release-19.09, but pipenv install requests still fails:

bug

Source

zupo

👍1

All 20 comments

cc @jonringer @FRidh

worldofpeace on 12 Nov 2019

I think this was fixed on master, I'll try to find the PR

jonringer on 12 Nov 2019

try doing `nix-shell --pure --run 'unset PYTHONPATH; pipenv install requests'

jonringer on 12 Nov 2019

👍1

im going to defer to @FRidh on this one, as he's more familiar with pipenv + nixpkgs

jonringer on 12 Nov 2019

I think this was fixed on master, I'll try to find the PR

Yes, it is fixed on master, the PRs that fixed it are linked in the issue body above.

zupo on 12 Nov 2019

try doing `nix-shell --pure --run 'unset PYTHONPATH; pipenv install requests'

Fantastic, this does in fact solve the problem!

So the fix is:

take the nixos-19.09 branch
cherry-pick https://github.com/NixOS/nixpkgs/pull/71656
cherry-pick https://github.com/NixOS/nixpkgs/pull/71800
prepend unset PYTHONPATH; to pipenv install

zupo on 12 Nov 2019

I've backported the commits to 19.09 just now.

Typically you should use nix run instead of nix-shell. No need for unsetting PYTHONPATH in that case.

FRidh on 12 Nov 2019

❤1

I've backported the commits to 19.09 just now.

Fantastic, thanks!

Typically you should use nix run instead of nix-shell. No need for unsetting PYTHONPATH in that case.

I used nix-shell --run as a minimal example to showcase the bug. I am using nix-shell with direnv to prepare a local Python development environment on my macbook. Should I use nix run instead of nix-shell for this? What documentation should I read?

zupo on 12 Nov 2019

nix-shell executes setup hooks. You should now for yourself whether for your use case the setup hooks are needed for the packages you load. Given nix-shell resembles a nix-build environment, this is often not what you want when you just want to use those packages for development.

FRidh on 12 Nov 2019

I've confirmed the backports work for me by pointing the URL to the most recent commit in https://github.com/NixOS/nixpkgs/commits/release-19.09:

-     url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
-     sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";
+     url = https://github.com/nixos/nixpkgs/archive/465e405dc9eda8d5f2b88af9e4555fcb4bfc1874.tar.gz;
+     sha256 = "096crqig0srjq8q6z7655yb46jmjj4zvjn0x1brwls7ibg8vv3gc";

Would it make sense to have a test that would try to run pipenv install <something> after the build is completed so we catch a potential future regression like this one? I can give it a stab.

zupo on 12 Nov 2019

How would we run such a test? If you want to run it in a nix-build then you need to prefetch dependencies and install from a folder.

FRidh on 12 Nov 2019

Would it make sense to have a test that would try to run pipenv install after the build is completed so we catch a potential future regression like this one? I can give it a stab.

Unfortunately that can't really be done in a sandbox environment. Would need to prefetch the dependencies like @FRidh states.

You could have a testScript which does the installation steps. But you would need to mention it in the expression, and it would have to be ran manually, not as part of the build. Not particularly elegant, but it's more than nothing.

jonringer on 12 Nov 2019

Ah, of course, and the sandbox does not have network access either, so yes, installing requests wouldn't work.

What could potentially work is installing pyjokes (which does not have dependencies) from a local tarball that was put in place before the checkPhase. I don't know if this is possible.

Yet another option is to ship a dummy python package with the pipenv pkg, that we can install locally with pipenv install -e path/to/dummy/package.

zupo on 12 Nov 2019

I've tried the process from the body of this issue, against latest nixos-19.09 commit, and pipenv install still fails. This is what I did:

$ cd /tmp && mkdir test && cd test
$ cat shell.nix
{ pkgs ? import (builtins.fetchTarball {
    url = https://github.com/nixos/nixpkgs/archive/08e503bac6decf9a3e01b79ab1f9788bf76380b1.tar.gz;
    sha256 = "1xf26fd1ahmz1m3si431xx8q8fs4xkbs6h4cznndgrr3hlafdgvg";
  }) {}
}:
pkgs.mkShell {
  name = "dev-shell";
  buildInputs = [ pkgs.python37Full pkgs.pipenv];
}

$ nix-shell --pure
[nix-shell:/private/tmp/test]$ pipenv install wheel
Creating a Pipfile for this project…
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (fcf4b1)!
Installing dependencies from Pipfile.lock (fcf4b1)…
An error occurred while installing wheel==0.33.6 --hash=sha256:10c9da68765315ed98850f8e048347c3eb06dd81822dc2ab1d4fde9dc9702646 --hash=sha256:f4da1763d3becf2e2cd92a14a7c920f0f00eca30fdde9ea992c836685b9faf28! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: []
[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/Users/zupo/.local/share/virtualenvs/test-6C_DLGjE/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]
ERROR: ERROR: Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

What am I doing wrong?

zupo on 20 Nov 2019

Same as before, doing the same, but taking the last commit on master, works fine:

-     url = https://github.com/nixos/nixpkgs/archive/08e503bac6decf9a3e01b79ab1f9788bf76380b1.tar.gz;
-     sha256 = "1xf26fd1ahmz1m3si431xx8q8fs4xkbs6h4cznndgrr3hlafdgvg";
+     url = https://github.com/nixos/nixpkgs/archive/0af613569c9056b2fa1c95b4a99dcb41219bb6dd.tar.gz;
+     sha256 = "07g3mlh0vfbz24v1dqwa0b72wkqky5zjp5yvbmy5cflirz12izzs";

$ cd /tmp && mkdir test && cd test
$ cat shell.nix
{ pkgs ? import (builtins.fetchTarball {
    url = https://github.com/nixos/nixpkgs/archive/0af613569c9056b2fa1c95b4a99dcb41219bb6dd.tar.gz;
    sha256 = "07g3mlh0vfbz24v1dqwa0b72wkqky5zjp5yvbmy5cflirz12izzs";
  }) {}
}:
pkgs.mkShell {
  name = "dev-shell";
  buildInputs = [ pkgs.python37Full pkgs.pipenv];
}
$ nix-shell --pure
[nix-shell:/private/tmp/test]$ pipenv install wheel
Creating a Pipfile for this project…
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (fcf4b1)!
Installing dependencies from Pipfile.lock (fcf4b1)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

[nix-shell:/private/tmp/test]$

zupo on 20 Nov 2019

[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/Users/zupo/.local/share/virtualenvs/test-6C_DLGjE/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]

unfortuantely this just means that it's picking up pip from the environment:

[nix-shell:~/projects/nixpkgs]$ pipenv install wheel
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded
Installing dependencies from Pipfile.lock (fcf4b1)…
An error occurred while installing wheel==0.33.6 --hash=sha256:10c9da68765315ed98850f8e048347c3eb06dd81822dc2ab1d4fde9dc9702646 --hash=sha256:f4da1763d3becf2e2cd92a14a7c920f0f00eca30fdde9ea992c836685b9faf28! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: []
[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/home/jon/.local/share/virtualenvs/nixpkgs-9itlgSZt/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]
ERROR: ERROR: Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

[nix-shell:~/projects/nixpkgs]$ echo PYTHONPATH
PYTHONPATH

[nix-shell:~/projects/nixpkgs]$ echo $PYTHONPATH
/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages:/nix/store/jvyqrqykg1ln1nwpb5qa9ds12xwbas71-python3.7-certifi-2019.6.16/lib/python3.7/site-packages:/nix/store/4c3z5r6yxsf2cxwwyazhdn92xixn4j5b-python3-3.7.5/lib/python3.7/site-packages:/nix/store/pbsbgky77l7s6sbq32ac2bsh0mqhbw7s-python3.7-setuptools-41.2.0/lib/python3.7/site-packages:/nix/store/mi4khdvzgabzg170an0rjc61cw02sr2f-python3.7-pip-19.2.3/lib/python3.7/site-packages:/nix/store/4rvrlhpfllhfiwczgd0ic0mgfsw86jhf-python3.7-virtualenv-16.4.3/lib/python3.7/site-packages:/nix/store/jyfcmsj081wc1hj6cgwvgpd27h4v6aqn-python3.7-virtualenv-clone-0.5.3/lib/python3.7/site-packages:/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages:/nix/store/jvyqrqykg1ln1nwpb5qa9ds12xwbas71-python3.7-certifi-2019.6.16/lib/python3.7/site-packages:/nix/store/4c3z5r6yxsf2cxwwyazhdn92xixn4j5b-python3-3.7.5/lib/python3.7/site-packages:/nix/store/pbsbgky77l7s6sbq32ac2bsh0mqhbw7s-python3.7-setuptools-41.2.0/lib/python3.7/site-packages:/nix/store/mi4khdvzgabzg170an0rjc61cw02sr2f-python3.7-pip-19.2.3/lib/python3.7/site-packages:/nix/store/4rvrlhpfllhfiwczgd0ic0mgfsw86jhf-python3.7-virtualenv-16.4.3/lib/python3.7/site-packages:/nix/store/jyfcmsj081wc1hj6cgwvgpd27h4v6aqn-python3.7-virtualenv-clone-0.5.3/lib/python3.7/site-packages

[nix-shell:~/projects/nixpkgs]$ unset PYTHONPATH

[nix-shell:~/projects/nixpkgs]$ pipenv install wheel
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded
Installing dependencies from Pipfile.lock (fcf4b1)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Although, this should have been fixed through backports, re-opening as this is still an issue on release-19.09

jonringer on 20 Nov 2019

PYTHONPATH is set because of the setup hook. That's just a consequence of using nix-shell when a buildPython* package is present. People should stop using nix-shell because for most things it is not the tool that should be used.

FRidh on 21 Nov 2019

@FRidh: I'm using https://direnv.net/ with use_nix to build my python project's local development environment. What should I be using instead or what should I be doing differently?

zupo on 21 Nov 2019

@FRidh: I'm using https://direnv.net/ with use_nix to build my python project's local development environment. What should I be using instead or what should I be doing differently?

I think direnv's use_nix probably uses nix-shell. So you might not be able to use direnv.

worldofpeace on 22 Nov 2019

@zimbatm: sorry to drag you into this, but I wonder if you have any advice for how to use direnv without nix-shell, or if that wouldn't make sense at all?

zupo on 23 Nov 2019

Was this page helpful?

0 / 5 - 0 ratings