Nixpkgs: Python virtualenv not working because of sys.prefix mismatch

[Enteee]

Describe the bug

Virtualenvwrapper in current nixpkgs-unstable exits with the following error when creating a new virtual environment.

ERROR: The executable /tmp/venv/bin/python3.7 is not functioning
ERROR: It thinks sys.prefix is '/nix/store/zjzq4qp81z8qnnnchnkycmsq0impw173-python3-3.7.4' (should be '/tmp/venv')
ERROR: virtualenv is not compatible with this system or executable

This might be intended behavior, maybe because of this PR, but then this section of the wiki probably needs a bit of attention.

To Reproduce

1. write default.nix

with import (builtins.fetchGit {
  # Descriptive name to make the store path easier to identify
  name = "nixpkgs-fixed-rev";
  url = https://github.com/NixOS/nixpkgs-channels/;

  # current unstable:
  rev = "002b853782e939c50da3fa7d424b08346f39eb6f";

  # 19.03 release:
  #ref = "nixos-19.03"; # 56d94c8c69f8cac518027d191e2f8de678b56088
}) {};

let
  python-with-base-packages = python3.withPackages(ps: with ps; [
    ipython
    virtualenvwrapper
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
  ]);

in stdenv.mkDerivation {
  name = "virtualenvwarpper-test";

  src = null;

  buildInputs = [
    python-with-base-packages
  ];

  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)

    set -x

    if [ ! -e venv/bin/activate ]; then
      virtualenv --system-site-packages --clear venv
    fi
    source venv/bin/activate
    export PATH="''${PATH}:venv/bin"

    if [ -e requirements.txt ]; then
      pip install -r requirements.txt
    fi

    set +x

  '';
}

1. nix-shell

Expected behavior
The behavior virtualenv shows when switching in the default.nix posted above to the 19.03 release.

Metadata

• system: "x86_64-linux"
• host os: Linux 4.14.136, NixOS, 19.09pre188002.e275a6c2f38 (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2.2
• channels(root): "nixos-19.09pre188002.e275a6c2f38, nixos-hardware, nixpkgs-19.09pre187222.002b853782e"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[tadfisher]

I was able to do a cheap workaround by copying sitecustomize.py from the system python lib dir into the virtualenv.

[mdedetrich]

I am also getting this issue, @tadfisher can you explain your workaround in more details? I tried doing this but I couldn't get it to work.

[FRidh]

This is indeed a regression.

At some point NIX_PYTHONPATH was introduced to replace PYTHONHOME in python.buildEnv/python.withPackages (https://github.com/NixOS/nixpkgs/pull/64634). This was done so users could set PYTHONHOME themselves. Programs often use sys.executable to determine the location of the interpreter, so NIX_PYTHONEXECUTABLE was introduced that would set sys.executable to the location of the interpreter in the python.buildEnv (https://github.com/NixOS/nixpkgs/pull/65454).

When creating a virtual environment, virtualenv will copy the Python executable which, when using python.buildEnv, will be a wrapper script.

What can be done about this? Well, the main reason for introducing NIX_PYTHONPATH was that users could still use PYTHONHOME and add the same time unset it so it would not leak.

Actually, we can still do that without PYTHONHOME. If we remove NIX_PYTHONPATH and NIX_PYTHONEXECUTABLE and use instead PYTHONHOME and say NIX_PYTHONHOME_UNSET where the latter would be used to control whether PYTHONHOME should be unset (by default on), then we achieve the same, however, without regressions. Users that want to use virtualenv, would then create a Nix env first using

python3.buildEnv {
  allowPythonHome = true;
}

When true, in the wrapper it would check if PYTHONHOME was set and use that, and otherwise use the Nix-generated value. In sitecustomize.py it would unset PYTHONHOME in case allowPythonHome = false;.

cc @abbradar

[FRidh]

Even better, in case of Python 3 we should just use pyvenv.cfg. Before I never considered it an option because we had to support 2 and 3, but now, with Python 2 going EOL, I think we should just take that route and leave a "best-effort" alternative for Python 2.

[FaustXVI]

Hi, I am encountering this issue and don't understand the workaround you gave. In particular, I don't understand where to put the python3.buildEnv section you gave.
Could you help me ?
Thanks

[sondr3]

I just hit this as well, makes using Nix with Python and virtualenv basically useless. Any updates?

[jonringer]

@sondr3 this works on master. are you using stable (19.09)?

[sondr3]

I gave up on writing the project in Python/Nix/virtualenv and converted it to Rust so I won't be able to check it, but yes, at the time I was using nixos-19.09.

[andersk]

On master, python3.withPackages (ps: [ ps.virtualenv ]) still fails as described, but python3Packages.virtualenv works.

[myme]

To circumvent this issue in one of our projects at $WORK, we do the following:

shellHook = ''
  # Python wheels
  export SOURCE_DATE_EPOCH="$(date +%s)"

  # https://github.com/NixOS/nixpkgs/issues/66366
  export PYTHONEXECUTABLE=${venv}/bin/python
  export PYTHONPATH=${python}/lib/python3.7/site-packages
'';

This is sufficient to resolve the packages included with python37.withPackages, as well as satisfying virtualenv. Please note that it's not a solution or fix, but a temporary workaround.

Edit: python is alias for python37.withPackages (...) and venv is just the location of the virtualenv on disk.

[jonringer]

now that #77644 has been merged; if you're on unstable, you should be able to just do:

shell.nix:

with import ./. {};

mkShell rec {
  name = "tmp";
  venvDir = "./venv";
  buildInputs = [
    python3Packages.venvShellHook
  ];
}

[jonringer]

Also of note, python3.3+ has virtualenv as part of the standard library, you don't actually need the virtualenv package, and that package might be introducing the regressions.

$ nix-shell -p python3

[nix-shell:/home/jon/projects/nixpkgs]$ python -m venv testenv

[nix-shell:/home/jon/projects/nixpkgs]$ source ./testenv/bin/activate
bash: hash: hashing disabled
bash: hash: hashing disabled
(testenv)
[nix-shell:/home/jon/projects/nixpkgs]$ pip install requests
Collecting requests
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2019.11.28 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.7
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(testenv)
[nix-shell:/home/jon/projects/nixpkgs]$ python
Python 3.7.6 (default, Dec 18 2019, 19:23:55)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>>

[andersk]

The venv standard library module also works with python3 but not with python3.withPackages.

$ nix-shell --pure -p 'python3.withPackages (ps: [])'  

[nix-shell:/tmp]$ python -m venv testenv
Error: Command '['/tmp/testenv/bin/python3.7', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

[nix-shell:/tmp]$ /tmp/testenv/bin/python3.7 -Im ensurepip --upgrade --default-pip
Looking in links: /run/user/1000/tmpdx_j6cmh
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
ERROR: Could not install packages due to an EnvironmentError: [Errno 30] Read-only file system: '/nix/store/bzvpdbzdy3a1qba8f9ynrzyz811c10pp-python3-3.7.6/lib/python3.7/site-packages/easy_install.py'

[jonringer]

ah, interesting. I don't think this is an intended use case of withPackages function. However, i could be wrong

cc @FRidh

[mudrii]

I do not think the issue is related to withPackages or python3Packages.virtualenv
I tried both and got the same error.

virtualenv my-new-python-venv
Using base prefix '/nix/store/aw85aijllnc5spk2fqbivygrhx6x7411-python3-3.7.6'
New python executable in /home/mudrii/my-new-python-venv/bin/python3.7
Not overwriting existing python script /home/mudrii/my-new-python-venv/bin/python (you must use /home/mudrii/my-new-python-venv/bin/python3.7)
ERROR: The executable /home/mudrii/my-new-python-venv/bin/python3.7 is not functioning
ERROR: It thinks sys.prefix is '/nix/store/aw85aijllnc5spk2fqbivygrhx6x7411-python3-3.7.6' (should be '/home/mudrii/my-new-python-venv')
ERROR: virtualenv is not compatible with this system or executable

Trying to replicate @jonringer

> nix-shell -p python3
The program ‘clear’ is currently not installed. It is provided by
several packages. You can install it by typing one of the following:
  nix-env -iA nixos.busybox
  nix-env -iA nixos.ncurses
  nix-env -iA nixos.ncurses5
  nix-env -iA nixos.toybox

>  whereis clear                                                                                                                                                                                      278ms
clear: /nix/store/zzigxi14myf0lvb8k738i2hc1436zxfj-user-environment/bin/clear /nix/store/mvfv6ja85k9mc3ynfmk8yjx1m11x87xz-system-path/bin/clear

> python -m venv testenv
Error: [Errno 13] Permission denied: '/home/mudrii/testenv/bin/activate.csh'

> ls -la /home/mudrii/testenv/bin/activate.csh                                                                                                                                                         
-r--r--r-- 1 mudrii users 1258 Feb  6 23:19 /home/mudrii/testenv/bin/activate.csh

any other workaround available?

[jonringer]

> ls -la /home/mudrii/testenv/bin/activate.csh                                                                                                                                                         
-r--r--r-- 1 mudrii users 1258 Feb  6 23:19 /home/mudrii/testenv/bin/activate.csh

it's probably trying to execute the file, but there's only read permissions.

are you able to do source /home/mudrii/testenv/bin/activate.csh?

EDIT: if it is execute permissions, you could do chmod +x /home/mudrii/testenv/bin/activate.csh and that should fix it

[mudrii]

@jonringer

I got below results

>  python -m venv testenv    
                                                                                                  286ms
> ls -la /home/mudrii/testenv/bin/activate.csh                                                                                     1678ms
-r--r--r-- 1 mudrii users 1258 Feb  7 09:44 /home/mudrii/testenv/bin/activate.csh

>  source /home/mudrii/testenv/bin/activate.csh
~/testenv/bin/activate.csh (line 23): Missing end to balance this if statement
        if (`basename "VIRTUAL_ENV"` == "__") then
        ^
from sourcing file ~/testenv/bin/activate.csh
        called on standard input

source: Error while reading file “/home/mudrii/testenv/bin/activate.csh”

[jonringer]

not familiar with csh, but that doesn't look to be related to nixpkgs.

[jonringer]

Either there's a mismatch in shell interpreters, or theirs a legitimate error in the generated pip script

[mudrii]

Usually, I use fish as a shell but I tried in bash and get a similar errors

mudrii@p53-nixos ~ $ python --version
Python 3.7.5

mudrii@p53-nixos ~ $  python -m venv testenv
Error: [Errno 13] Permission denied: '/home/mudrii/testenv/bin/activate.csh'

mudrii@p53-nixos ~ $ echo $0
bash

mudrii@p53-nixos ~ $ source /home/mudrii/testenv/bin/activate.csh
bash: alias: deactivate: not found
bash: alias: `test $?_OLD_VIRTUAL_PATH !': invalid alias name
deactivate: command not found
setenv: command not found
setenv: command not found
bash: /home/mudrii/testenv/bin/activate.csh: line 38: syntax error: unexpected end of file

is it only me getting this errors?

[mudrii]

@jonringer I think you right

The problem is related to the shell.
Somehow python -m venv testenv tries to use activate.csh as this file is for csh shell.
In the folder ~/testenv/bin/ I find few files for activision

ls -la ~/testenv/bin/
total 40
drwxr-xr-x    2 mudrii   users         4096 Feb  8 12:12 .
drwxr-xr-x    5 mudrii   users         4096 Feb  8 12:12 ..
-r--r--r--    1 mudrii   users         2206 Feb  8 12:12 activate
-r--r--r--    1 mudrii   users         1258 Feb  8 12:12 activate.csh
-r--r--r--    1 mudrii   users         2410 Feb  8 12:12 activate.fish
-rwxr-xr-x    1 mudrii   users          248 Feb  8 12:12 easy_install
-rwxr-xr-x    1 mudrii   users          248 Feb  8 12:12 easy_install-3.7
-rwxr-xr-x    1 mudrii   users          230 Feb  8 12:12 pip
-rwxr-xr-x    1 mudrii   users          230 Feb  8 12:12 pip3
-rwxr-xr-x    1 mudrii   users          230 Feb  8 12:12 pip3.7
lrwxrwxrwx    1 mudrii   users           40 Feb  8 12:12 python -> /etc/profiles/per-user/mudrii/bin/python
lrwxrwxrwx    1 mudrii   users            6 Feb  8 12:12 python3 -> python

Now correct way to activate env is to specify exact source file
For bash > source /home/mudrii/testenv/bin/activate
For Fish > source /home/mudrii/testenv/bin/activate.fish

NOTE: But the problem still persists with _python3Packages.virtualenv_

[jonringer]

I use bash, and haven't encountered an issue, but i also just use python3 -m venv instead of the package.

[FRidh]

ERROR: The executable /tmp/venv/bin/python3.7 is not functioning
ERROR: It thinks sys.prefix is '/nix/store/zjzq4qp81z8qnnnchnkycmsq0impw173-python3-3.7.4' (should be '/tmp/venv')
ERROR: virtualenv is not compatible with this system or executable

See https://github.com/NixOS/nixpkgs/issues/79702#issuecomment-598296978 for the explanation.

If we cannot wrap python, then we maybe can have a file like pyvenv.cfg but with out wrapper arguments. Unfortunately that won't work either because of shebangs and wrappers created already in the builds of the packages that were added to the withPackages env.

[FRidh]

When possible, I recommend not using virtualenv but venv, which is part of the standard library in Python 3. No need to create a python.withPackages then. This won't work well if you want to use Nixpkgs Python packages as input, but that's anyway not recommended.

Edit: note there are actually good reasons to use virtualenv.

[andersk]

@FRidh Did you miss my comment that venv also fails with python3.withPackages?

(Also, when we’re trying to get some complicated third-party script to work on NixOS, sometimes we don’t have the luxury of choosing which modules it tries to use, no matter how Wrong its choices are.)

[jonringer]

@andersk if you want to use venv, please take a look at https://github.com/NixOS/nixpkgs/issues/85791#issuecomment-617961269 . This will likely solve a lot of your issues. I think I'll make a video on how to setup a venv environment in a nix environment.

[andersk]

@jonringer Sometimes the goal is not to run venv or virtualenv, but to run a third-party script that itself invokes venv or virtualenv in a way that’s difficult to disentangle from the script. We ought to be able to make that work without manual hacking.

[jonringer]

oh, I see.

That usecase should still work if you call the script from a venv environment :) (at least it does for one of ours that we use at work, but it uses python -m venv )

[iwinux]

This is really annoying.

Using base prefix '/nix/store/vs4vj1yzqj1bkcqkf3b6sxm6jfy1gb4j-python3-3.7.7'
New python executable in /tmp/xxx/bin/python3.7
Also creating executable in /tmp/xxx/bin/python
ERROR: The executable /tmp/xxx/bin/python3.7 is not functioning
ERROR: It thinks sys.prefix is '/nix/store/hdy9m0sfn8fcww97lhfrkirj1ba947zm-python3-3.7.7-env' (should be '/tmp/xxx')
ERROR: virtualenv is not compatible with this system or executable

[iwinux]

NIX_PYTHONPREFIX seems to be the culprit:

$ readlink -f $(which virtualenv)
/nix/store/hdy9m0sfn8fcww97lhfrkirj1ba947zm-python3-3.7.7-env/bin/virtualenv

$ cat $(which virtualenv)
#! /nix/store/ffli6m23501dkiznwlkf6n4xvrj02snr-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/hdy9m0sfn8fcww97lhfrkirj1ba947zm-python3-3.7.7-env'
export NIX_PYTHONEXECUTABLE='/nix/store/hdy9m0sfn8fcww97lhfrkirj1ba947zm-python3-3.7.7-env/bin/python3.7'
export NIX_PYTHONPATH='/nix/store/hdy9m0sfn8fcww97lhfrkirj1ba947zm-python3-3.7.7-env/lib/python3.7/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/aldxb9bzrdc8dsvab5snxpw76w0w18l4-python3.7-virtualenv-16.7.9/bin/virtualenv"  "$@"

/nix/store/aldxb9bzrdc8dsvab5snxpw76w0w18l4-python3.7-virtualenv-16.7.9/bin/virtualenv without these NIX_whatever env vars works just fine 😑.

[eonpatapon]

I also hit this bug using pre-commit. For python hooks pre-commit creates virtualenvs to install them.

This is currently broken when using pre-commit from nix:

 nix-env -iA nixpkgs.pre-commit
installing 'pre-commit-1.21.0'
[...]
 cat .pre-commit-config.yaml
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v3.0.1
  hooks:
  - id: trailing-whitespace
 pre-commit run -a
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/nix/store/vs4vj1yzqj1bkcqkf3b6sxm6jfy1gb4j-python3-3.7.7/bin/python3.7', '-mvirtualenv', '/home/eon/.cache/pre-commit/repoacxaqktt/py_env-python3.7', '-p', '/nix/store/vs4vj1yzqj1bkcqkf3b6sxm6jfy1gb4j-python3-3.7.7/bin/python3.7')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    /nix/store/vs4vj1yzqj1bkcqkf3b6sxm6jfy1gb4j-python3-3.7.7/bin/python3.7: No module named virtualenv

Check the log at /home/eon/.cache/pre-commit/pre-commit.log

Actually the initial problem is that the python used to run pre-commit has not access to the virtualenv module. But if I use a python w/ virtualenv to run pre-commit I'm hitting the issue discussed here.

By patching pre-commit to use -mvenv instead of -mvirtualenv it works fine.

[calbrecht]

Ok, i noticed this as well while trying to mkShell an environment for building servo....

There seems to be great confusion about virtualenv and venv, therefore this SO What is the difference between venv, pyvenv, pyenv, virtualenv, virtualenvwrapper, pipenv, etc? shall shed some light.

Also, while digging deeper, i noticed the virtualenv pkg in nixpkgs is four major versions behind upstream.

Updating virtualenv with

{ pkgs ? import <nixpkgs> {} }:

let
  virtualenv = pkgs.python3Packages.virtualenv.overridePythonAttrs (old: rec {
    pname = "virtualenv";
    version = "20.0.21";

    src = pkgs.python3Packages.fetchPypi {
      inherit pname version;
      sha256 = "1kxnxxwa25ghlkpyrxa8pi49v87b7ps2gyla7d1h6kbz9sfn45m1";
    };

    propagatedBuildInputs = with pkgs.python3Packages; [
      appdirs distlib filelock setuptools_scm six contextlib2 importlib-metadata
      importlib-resources pathlib2
    ];

    patches = [];
  });

  python3WithPackages = pkgs.python3.withPackages (python3Pkgs: with python3Pkgs; [
    virtualenv
    # other python packages you want
  ]);
in
pkgs.mkShell {
  nativeBuildInputs = [ python3WithPackages ];
}

makes this error ERROR: It thinks sys.prefix is ... go away

[mdedetrich]

@calbrecht Someone should set up a PR to update this then

[calbrecht]

Ok, so i will do this. ^^

[FRidh]

Please check whether dc80e0724c2d5407dbe3cff2972500065951f888 functions for you as expected.

[andersk]

Works for me in all Python 3.x versions, but fails in Python 2.7:

$ rm -rf ve
$ nix-shell -p 'python27.withPackages (ps: [ps.virtualenv])' --run 'virtualenv ve'
ValueError: source and destination is the same /nix/store/38kz3j1a87cq5y59k5w7k9yk4cqgc5b2-python-2.7.18/lib/python2.7/os.py

[andersk]

I filed #88711 for the venv issue, since I guess the fix for that will be separate.

[FRidh]

Yes, feel free to find a solution for 2.7, I won't spend time on that.

edit: (ok, I did...)

[calbrecht]

@andersk, i have accomplished to make virtualenv work with python2.7, also for https://github.com/NixOS/nixpkgs/issues/84568, where i will paste the patch, but unsure if it's correct to do it this way.

[djmaze]

By patching pre-commit to use -mvenv instead of -mvirtualenv it works fine.

pre-commit still does not work, and patching it to use -mvenv is probably not the right solution. Any other solutions?