Pip: Local installs now require write permissions

[rpkilby]

Environment

• pip version: 20.1
• Python version: 3.6.8
• OS: CentOS 8

Description
The new build-in-place behavior (ref #7882) for local project installs requires that a user have write permissions in the project directory. This normally makes sense, since a user is typically installing their own local work, but this breaks when users are installing a project owned by another user.

Without write permissions, other users will get a permissions error. However, giving them write permissions isn't ideal, and the build artifacts left behind are owned by them instead of the author.

Related to #8168

Expected behavior
Users should not require write permissions to install a local project. Possibly, pip tests for write access, and falls back to building in a temp dir? Possibly, pip could have an option for not building in place, or for providing a directory for where the build should occur (note that the --build option does not fix this).

How to Reproduce

At a high level:

1. Create a local project
2. Ensure other users don't have write permissions chmod 755 -R <project>
3. Switch to another user
4. pip install -e <project> (note that non-editable installs also fail similarly)

Example error:

$ pip install -e /work/test/test/
Obtaining file:///work/test/test
Could not build wheels for test, since package 'wheel' is not installed.
Installing collected packages: test
  Running setup.py develop for test
    ERROR: Command errored out with exit status 1:
     command: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /work/test/test/
    Complete output (4 lines):
    running develop
    running egg_info
    creating test.egg-info
    error: could not create 'test.egg-info': Permission denied
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

Detailed steps:

Vagrantfile contents:

Vagrant.configure("2") do |config|
    config.vm.box = "bento/centos-8.0"
    config.vm.provision "shell", inline: "yum install -y python36"
end

Create primary/secondary users and test project owned by primary user:

[vagrant@localhost ~]$ sudo su
[root@localhost vagrant]# useradd primary
[root@localhost vagrant]# useradd secondary
[root@localhost vagrant]# mkdir -p -m 755 /work/test
[root@localhost vagrant]# chown primary /work/test/
[root@localhost vagrant]# exit
exit
[vagrant@localhost ~]$ sudo -iu primary
[primary@localhost ~]$ mkdir /work/test/test
[primary@localhost ~]$ echo "from setuptools import setup; setup(name='test', version='1', py_modules=['foo'])" > /work/test/test/setup.py
[primary@localhost ~]$ touch /work/test/test/foo.py
[primary@localhost ~]$ exit
logout

Verify project is reachable and attempt to install as secondary user:

[vagrant@localhost ~]$ sudo -iu secondary
[secondary@localhost ~]$ ls /work/test/test/
foo.py  setup.py
[secondary@localhost ~]$ python3 -m venv venv
[secondary@localhost ~]$ source venv/bin/activate
(venv) [secondary@localhost ~]$ pip install -U pip setuptools
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 1.1MB/s 
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl (582kB)
    100% |████████████████████████████████| 583kB 1.6MB/s 
Installing collected packages: pip, setuptools
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
  Found existing installation: setuptools 39.2.0
    Uninstalling setuptools-39.2.0:
      Successfully uninstalled setuptools-39.2.0
Successfully installed pip-20.1 setuptools-46.1.3
(venv) [secondary@localhost ~]$ pip install -e /work/test/test/
Obtaining file:///work/test/test
Could not build wheels for test, since package 'wheel' is not installed.
Installing collected packages: test
  Running setup.py develop for test
    ERROR: Command errored out with exit status 1:
     command: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /work/test/test/
    Complete output (4 lines):
    running develop
    running egg_info
    creating test.egg-info
    error: could not create 'test.egg-info': Permission denied
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/secondary/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/work/test/test/setup.py'"'"'; __file__='"'"'/work/test/test/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
(venv) [secondary@localhost ~]$ exit
logout

Make project writable and reattempt install as secondary user:

[vagrant@localhost ~]$ sudo su
[root@localhost vagrant]# chmod -R 777 /work/test/
[root@localhost vagrant]# exit
exit
[vagrant@localhost ~]$ sudo -iu secondary
[secondary@localhost ~]$ source venv/bin/activate
(venv) [secondary@localhost ~]$ pip install -e /work/test/test/
Obtaining file:///work/test/test
Could not build wheels for test, since package 'wheel' is not installed.
Installing collected packages: test
  Running setup.py develop for test
Successfully installed test
(venv) [secondary@localhost ~]$ pip list
Package    Version Location
---------- ------- ---------------
pip        20.1
setuptools 46.1.3
test       1       /work/test/test

Build artifacts are owned by secondary user:

(venv) [secondary@localhost ~]$ ls -la /work/test/test/
total 4
drwxrwxrwx. 3 primary   primary   57 May  5 20:03 .
drwxrwxrwx. 3 primary   root      18 May  5 20:01 ..
-rwxrwxrwx. 1 primary   primary    0 May  5 20:01 foo.py
-rwxrwxrwx. 1 primary   primary   82 May  5 20:01 setup.py
drwxrwxr-x. 2 secondary secondary 90 May  5 20:03 test.egg-info

[uranusjr]

I’m starting to think maybe we should revert this change. PEP 517 supporting out-of-tree builds is the only way this can be improved, but that will take a significant amount of time to roll out. In the meantime, the previous method is slow, but at least works for everyone.

[sbidoul]

at least works for everyone

Not everyone. There were also correctness issues. Let's discuss in #7555.