Nixpkgs: in nix-shell, 'pipenv install' etc. fail with "ModuleNotFoundError: No module named 'pip._internal'"

[das-g]

Issue description

pipenv commands that use pip internally or explicitly fail.

Steps to reproduce

1. make sure we're in a pristine and safely disposable directory
   bash cd $(mktemp -d)
2. run a pipenv command that uses pip explicitly
   bash nix-shell -p pipenv --run 'pipenv run pip --version'

Expected

• Virtualenv is created
• Pipfile is created
• Version of pip in that virtualenv is displayed

Observed

• :heavy_check_mark: Virtualenv is created
  Creating a virtualenv for this project… ⠋Using base prefix '/nix/store/ljhgdba6n8ag6f8clpi4m9zizm7b8mx3-python3-3.6.5' New python executable in /home/das-g/.local/share/virtualenvs/tmp.btSs5gvz0u-HGofKOag/bin/python3.6m Also creating executable in /home/das-g/.local/share/virtualenvs/tmp.btSs5gvz0u-HGofKOag/bin/python Installing setuptools, pip, wheel...done. Virtualenv location: /home/das-g/.local/share/virtualenvs/tmp.btSs5gvz0u-HGofKOag
• :heavy_check_mark: Pipfile is created
  Creating a Pipfile for this project…
• :heavy_multiplication_x: Then, a ModuleNotFoundError error is displayed:
  Traceback (most recent call last): File "/home/das-g/.local/share/virtualenvs/tmp.btSs5gvz0u-HGofKOag/bin/pip", line 7, in <module> from pip._internal import main ModuleNotFoundError: No module named 'pip._internal'

Cleanup after reproducing

In this order:

1. Remove the virtualenv:
   bash nix-shell -p pipenv --run 'pipenv --rm'
2. cd - and remove the temporary directory with its content

Variations

Similar observations for

• pipenv commands using pip internally, e.g. pipenv install requests
• invoking pip in a pipenv shell:
  bash nix-shell -p pipenv --run 'pipenv shell' pip --version
• doing any of the above in an interactive nix-shell with pipenv

Technical details

• system: "x86_64-linux"
• host os: Linux 4.14.35, NixOS, 18.03.132083.06c576b0525 (Impala)
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.0
• channels(root): "nixos-18.03.132083.06c576b0525"
• channels(das-g): ""
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[das-g]

(Dunno whether this is related to #33713, where pipenv complains about not finding /bin/pip rather than not finding some submodule of pip as here.)

[das-g]

I've experimented some more, and it seems like the error only occurs when using nix-shell (whether with --run or interactively, whether with -p pipenv or with a default.nix doesn't seem to matter), but not when having pipenv installed by listing it in /etc/nixos/configuration.nix or by imperatively in the current user's profile with nix-env -iA nixos.pipenv.

[adisbladis]

This can be worked around by setting PYTHONPATH to your virtual env path:
export PYTHONPATH=$(pipenv --venv)/lib/python3.6/site-packages/:$PYTHONPATH

[qrilka]

and this pip from nix-shell is advised in the manual :-\

[qrilka]

but that one is for virtualenv + pip, not pipenv

[das-g]

and this pip from nix-shell is advised in the manual :-\

Is it, @qrilka? Skimming over them, I can see that in neither https://nixos.org/nixos/manual/ nor in https://nixos.org/nixpkgs/manual/

[qrilka]

See https://nixos.org/nixpkgs/manual/#how-to-consume-python-modules-using-pip-in-a-virtualenv-like-i-am-used-to-on-other-operating-systems - that failed with No module named 'pip._internal' for my until I added export PYTHONPATH=venv/lib/python3.6/site-packages/:$PYTHONPATH

[domenkozar]

I've just pushed 7a7463d0a3fe6d39a7fc499a0fd9756a3c671e62 and pipenv works for me now, can you give it a shot?

[das-g]

Yep, seems to work:

cd $(mktemp -d)
nix-shell -p pipenv \
 -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/7a7463d0a3fe6d39a7fc499a0fd9756a3c671e62.tar.gz \
 --run 'pipenv run pip --version'

Creating a virtualenv for this project…
Using /nix/store/yxzy33fw61kj9ljk30i5qyjrv3pax0kc-python3-3.6.5/bin/python3.6m (3.6.5) to create virtualenv…
⠋Already using interpreter /nix/store/yxzy33fw61kj9ljk30i5qyjrv3pax0kc-python3-3.6.5/bin/python3.6m
Using base prefix '/nix/store/yxzy33fw61kj9ljk30i5qyjrv3pax0kc-python3-3.6.5'
New python executable in /home/das-g/.local/share/virtualenvs/tmp.kitAQ9u20x-kvPPKl1d/bin/python3.6m
Also creating executable in /home/das-g/.local/share/virtualenvs/tmp.kitAQ9u20x-kvPPKl1d/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/das-g/.local/share/virtualenvs/tmp.kitAQ9u20x-kvPPKl1d
Creating a Pipfile for this project…
pip 10.0.1 from /nix/store/w18a3cl9g0rppnrvig820ywn8hg6x87z-python3.6-pip-10.0.1/lib/python3.6/site-packages/pip (python 3.6)

:+1:

Will this get backported to release 18.03 Impala?

Should this issue (#39558) be closed?

[das-g]

Oh, and can we add a test for this?

[domenkozar]

As 18.09 is going out in two weeks, I'm closing this as fixed.