Nixpkgs: `python3 -m venv` creates virtualenv with read-only files that cannot be replaced

[jherland]

Describe the bug
Running python3 -m venv to _re_-initialize a virtualenv (previously created with the same python3 -m venv command) fails because the activate scripts in the virtualenv are created _read-only_.

To Reproduce
Steps to reproduce the behavior:

$ python3 -m venv foo
$ ls -l foo/bin/
total 32
-r--r--r-- 1 jherland users 2206 Sep 30 11:22 activate
-r--r--r-- 1 jherland users 1262 Sep 30 11:22 activate.csh
-r--r--r-- 1 jherland users 2414 Sep 30 11:22 activate.fish
-rwxr-xr-x 1 jherland users  261 Sep 30 11:22 easy_install
-rwxr-xr-x 1 jherland users  261 Sep 30 11:22 easy_install-3.7
-rwxr-xr-x 1 jherland users  243 Sep 30 11:22 pip
-rwxr-xr-x 1 jherland users  243 Sep 30 11:22 pip3
-rwxr-xr-x 1 jherland users  243 Sep 30 11:22 pip3.7
lrwxrwxrwx 1 jherland users    7 Sep 30 11:22 python -> python3
lrwxrwxrwx 1 jherland users   39 Sep 30 11:22 python3 -> /home/jherland/.nix-profile/bin/python3
$ python3 -m venv foo
Error: [Errno 13] Permission denied: '/path/to/cwd/foo/bin/activate.fish'

Expected behavior
Re-running python3 -m venv foo should not fail to re-initialize the given virtualenv. Current workaround is to run $ python3 -m venv --clear foo which wipes the virtualenv and forces you to pip-install everything again.

Notify maintainers

@FRidh

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.4.67, NixOS, 20.03.3021.5659cb448e9 (Markhor)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.6
• channels(jherland): "home-manager-20.03, unstable-20.09pre234399.0260f04fe25"
• channels(root): "nixos-20.03.2520.add5529b3ee"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
  - python3
  - python36
  - python37
  - python38
  - python39
# a list of nixos modules affected by the problem
module:

[newkozlukov]

Side note: at first I misinterpreted the issue and thought it's about symlinks to /nix/store which are expected to be created and to be read-only, but python -m venv --copies venv suffers too

[jherland]

Yeah, this is totally independent of /nix/store. Even without --copies there are no symlinks (directly) into /nix/store:

$ find foo -type l | xargs ls -al
lrwxrwxrwx 1 jherland users 38 Sep 30 12:54 foo/bin/python -> /home/jherland/.nix-profile/bin/python
lrwxrwxrwx 1 jherland users  6 Sep 30 12:54 foo/bin/python3 -> python
lrwxrwxrwx 1 jherland users  3 Sep 30 12:54 foo/lib64 -> lib

Everything else is written/copied by the venv module. That said, I speculate (but haven't confirmed) that it might be copying the permission bits from some original/template file under /nix/store.

[jherland]

Confirmed my suspicion: venv.EnvBuilder.setup_scripts() calls self.install_scripts(context, path) with path=/nix/store/$python/lib/python$version/venv/scripts, and install_scripts() then copies files from under that dir, including a shutil.copymode(srcfile, dstfile) at the end.

[jonringer]

Not denying this is an issue, but if you want to use venv with nix, you should look at https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems

What's the purpose of re-initializing a venv after it's been created?

[FRidh]

I've created the upstream issue https://bugs.python.org/issue41913.

[jherland]

Not denying this is an issue, but if you want to use venv with nix, you should look at https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems

Thanks for the link. I've seen similar versions before, but that is definitely better documented than I remember it. I'd agree that there are better ways to manage this from the Nix POV.

What's the purpose of re-initializing a venv after it's been created?

In this case I'm working with a legacy in-house build system for embedded devices, and trying to clean up and isolate the environment in which this build system is run. I'm using Nix to take control of the build environment, but Nix is not (yet) an acceptable requirement for the build system. (In other words, I can use Nix around this build system, but I cannot tell everybody else to use Nix). The build system in question manages and refreshes its own venv, and I can probably tweak the build system to work around this issue. That said, I wanted to raise it here, as it didn't seem right in general.

[jonringer]

In this case I'm working with a legacy in-house build system for embedded devices, and trying to clean up and isolate the environment in which this build system is run. I'm using Nix to take control of the build environment, but Nix is not (yet) an acceptable requirement for the build system. (In other words, I can use Nix around this build system, but I cannot tell everybody else to use Nix). The build system in question manages and refreshes its own venv, and I can probably tweak the build system to work around this issue. That said, I wanted to raise it here, as it didn't seem right in general.

ah this makes sense