Pip: Virtualenv with --system-site-packages breaks pip's build isolation

[pganssle]

It seems that if you create a virtualenv with --system-site-packages, the system packages (but not the user packages) will be on the PYTHONPATH in the PEP 517 isolated build environment that pip creates (it does not effect python -m pep517.build), and it seems they will be on there with higher precedence than the requirements installed in the build environment. It only affects pip >= 19.0.0.

The most common way I've seen this cause problems is with the new setuptools.build_meta:__legacy__ backend. Since the most recent version of pip requires a recent version of setuptools, if you have an older version of setuptools installed on the system, pip install will fail due to the missing setuptools.build_meta:__legacy__ backend. It is possible to reproduce this by crafting a deliberately bad package and installing it, then creating a wheel for a newer version of the package (which allowed me to test that this failure was actually introduced with pip==19.0.0), but for the MWE we can stick with setuptools.

To reproduce, create a package like this:

cd /tmp
mkdir demo
cd demo
mkdir badpkg

touch badpkg/pyproject.toml
echo 'from setuptools import setup; setup(name="foo", version="0.1.0")' \
    > badpkg/setup.py

Then install an older version of setuptools on your system (can't be --user or in the virtualenv), pip install 'setuptools < 40.8.0' (I did this in a pyenv environment). If your system already has an older version of setuptools on it, you're already good.

Next create a virtualenv with --system-site-packages and activate it:

virtualenv venv --system-site-packages
source venv/bin/activate

Finally try to install badpkg (pip wheel or pip install -t tmp also works to demonstrate the problem):

pip install ./badpkg

You should get a traceback the ends like this:

AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

At first I thought this was because the affected packages had too loose bounds on build-system.requires (e.g. requires=["setuptools"]) and that pip was failing to install a more recent version in the isolated environment, but this bug still occurs even if you specify "setuptools>=40.8.0", so I believe it's not a problem at install-time, it's a problem with the path resolution at build time.

At the moment it's unclear if this is a problem with pip or virtualenv, but since it does not affect python -m pep517.build, I'm reporting it here. It could be a combination of both.

CC: @gaborbernat

[pganssle]

Per my Bloomberg colleague @lkollar, it seems that the problem is that pip creates a custom sitecustomize.py file in order to inject itself into the build environment.

Looks to me like it's just blacklisting the system packages with distutils.get_python_lib, which doesn't include the system path added by virtualenv. Seems like pep517.envbuild.BuildEnvironment doesn't have this problem - maybe switch over to using that? Or was there a reason to do it this way?

[boegel]

I'm also seeing this problem, when installing cryptography 2.5 using Python 3.7.2, even though I'm using the latest setuptools (40.8.0).

What I don't understand is why I'm seeing this even though I'm using pip install --no-build-isolation .... How is that possible if the problem is related to pip's build isolation feature?

[boegel]

I should also mention that I'm not using virtualenv at all (although maybe somehow the cryptography installation procedure does that somehow?).

[pganssle]

@boegel Can you create a minimal-working example that reproduces this, and include your pip versions? I've never known anyone to be able to trigger this without virtualenv --system-site-packages.

[boegel]

@pganssle Does this suffice?

Note that this is a self-built Python 3.7.2 from source, in a custom location, not sure if that's relevant to the problem...

$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug

$ curl -OL https://files.pythonhosted.org/packages/69/ed/5e97b7f54237a9e4e6291b6e52173372b7fa45ca730d36ea90b790c0059a/cryptography-2.5.tar.gz

$ tar xfz cryptography-2.5.tar.gz

$ cd cryptography-2.5

$ pip install --prefix=$PWD --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/user/pip_debug/cryptography-2.5
    Preparing wheel metadata ... error
    Complete output from command /prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e:
    Traceback (most recent call last):
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e" failed with error code 1 in /tmp/pip-req-build-q9knj58j

This is on CentOS 7.6.1810, with only python 2.x installed on the system. We do have an old setuptools in the OS as well, but I think that one is irrelevant since that's only for Python 2.

$ rpm -qa | grep ^python[0-9]*-[0-9]
python-2.7.5-76.el7.x86_64
$ rpm -qa | grep setuptools
python-setuptools-0.9.8-7.el7.noarch

[boegel]

I should also mention that --no-use-pep517 fixes the problem for me.

[pfmoore]

pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

Looks like your pip (and your setuptools) have been installed as eggs, presumably using easy_install. This is obsolete, can you try with a copy of pip installed using pip?

[boegel]

@pfmoore Same problem when pip and setuptools were installed with pip, when using latest cryptography from PyPI. Note that there's no other Python 3 installation on the system (and hence no other setuptools either).

$ which python
/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python
$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug
$ curl -OL https://files.pythonhosted.org/packages/07/ca/bc827c5e55918ad223d59d299fff92f3563476c3b00d0a9157d9c0217449/cryptography-2.6.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  480k  100  480k    0     0   755k      0 --:--:-- --:--:-- --:--:--  754k

$ tar xfz cryptography-2.6.1.tar.gz
$ cd cryptography-2.6.1

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
    Preparing wheel metadata ... error
    Complete output from command /prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_:
    Traceback (most recent call last):
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_" failed with error code 1 in /tmp/pip-req-build-s_w63g1g

Works fine when using --no-use-pep517:

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  . --no-use-pep517
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
Building wheels for collected packages: cryptography
  Building wheel for cryptography (setup.py) ...
...

[pfmoore]

module 'setuptools.build_meta' has no attribute '__legacy__'

That indicates you're not picking up setuptools 40.8.0 - see here

Try running

import setuptools.build_meta
print(setuptools.build_meta.__legacy__)

You should get something like <setuptools.build_meta._BuildMetaLegacyBackend object at 0x000000000401B2E8>. If you're not, then there's something wrong with your system setuptools. If you are, your build environment is not picking up the same packages as your system Python - in spite of your assertion that there's only the one setuptools installed on your system :-(

Also, the fact that you're using --prefix may be relevant here - I don't know how that interacts (if at all) with build environments. Can you reproduce this issue without --prefix?

[gaborbernat]

@pfmoore if virtualenv inherits the system site package (and that has a setuptoools before 40.8.0) the pip install of the setuptools build dependency is ignored (as pip does not check the version of what is installed unless -U is passed, and says any setuptools satisfies the install requirement); therefore once pip tries to get the build backend the above failure is thrown. I don't think there needs to be anything wrong with system python to fall into this issue. pip should make sure to never-ever create an isolated build environment that inherits from the system site package, which is not the case at the moment.

[pfmoore]

if virtualenv inherits the system site package

Presumably when --no-build-isolation is specified? An isolated build ignores the existing installation.

(and that has a setuptoools before 40.8.0)

Well, the point of --no-build-isolation is that the user is agreeing to set up the correct build environment, so that seems like it's a simple case of user error.

However, in this case @boegel is claiming that he has setuptools 40.8.0 installed, and no other setuptools is present. If that's the case, and his setuptools doesn't contain build_meta.__legacy__, then that setuptools installation is somehow broken, because 40.8.0 does contain that backend.

My suspicion is that @boegel is actually mistaken, and there's another copy of setuptools lingering around somewhere, which is getting picked up and is confusing things. The other possibility is that there's a bug somewhere - but if that is the case, then we're currently struggling to reduce the example to something that can easily reproduce/demonstrate that bug.

pip should make sure to never-ever create an isolated build environment that inherits from the system site package

Well, yes, that's sort of the definition of "isolated" :-) But I'm not sure how relevant that is in this case, where --no-build-isolation is set.

which is not the case at the moment

That's what @pganssle is saying in the original post in this thread, yes. But I'm responding to @boegel, who is claiming that he can reproduce the problem without a virtualenv, using --no-build-isolation.

If @boegel's problem turns out to be real, then I don't think it's related to this one (there's too many differences in the steps to reproduce it). So it should probably be a separate issue. But I'd be interested to understand why he thinks it is the same issue...

[boegel]

Dug a little further, it looks it's indeed a case of a broken setuptools installation, in some sense...

First, I should clarify my statement w.r.t. not having another setuptools installed. I should clarify that I mean that there's no other setuptools installed for Python 3.x. There is one for Python 2 (which is available in the OS), I'm assuming that one will never be picked up by Python 3.

That said, I'm pretty sure the right setuptools is being picked up:

$ python -c "import setuptools; print(setuptools.__file__); print(setuptools.version.__version__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/__init__.py
40.8.0

Yet:

$ python -c "import setuptools.build_meta; print(setuptools.build_meta.__file__); print(setuptools.build_meta.__legacy__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/build_meta.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

So I do have setuptools 40.8.0, but no __legacy__ in setuptools.build_meta?!

This sheds some light on it:

ls -ld /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools*
drwxr-xr-x 6 myuser myuser 4096 Mar  7 18:48 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:56 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.6.2.dist-info
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:57 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg-info
drwxr-xr-x 3 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm-3.2.0.dist-info

The setuptools-40.6.2.dist-info looks very suspicious, it seems like I somehow ended up with a frankenbuild of setuptools-40.6.2 (which doesn't have __legacy__ in setuptools.build_meta yet) and setuptools-40.8.0?! Ugh...

I got redirect to this issue via https://github.com/pypa/setuptools/issues/1694, which shows the exact same error I'm running into, so I naively assumed it's the same problem.

It seems in my case it's indeed because of a corrupted setuptools though, my apologies for the noise...

[pfmoore]

No problem, glad we got to the bottom of it!

[Lawouach]

Hey,

I'm running into this issue as well and I'm thoroughly confused. As a mere user, it looks totally broken with no clear path of understanding how I'm doing anything wrong.

[Lawouach]

Well, having said that. I ended up removing the system-wide setuptools package and now local pip doesn't complain.

[tgs]

I'm encountering the same issue, I think. My organization defaults virtualenvs to --system-site-packages, and the globally installed setuptools is 40.4.3. I cloned the pip git repo, and then:

In a system-site-packages virtualenv, pip install ./pip/ gives the AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'.

In a virtualenv with no-site-packages, pip install ./pip/ works correctly.

I think I agree with @gaborbernat that it would be better for "build isolation" to include ignoring --system-site-packages. Or maybe there's at least some way to give a better error message?

[webknjaz]

FTR I'm hitting this with tox + system-site-packages with setuptools>=40.8.0 in pyproject.toml under Python 2.7.15. I was actually hitting pypa/setuptools#1136 originally (which I knew was fixed a while back) and only after some time of playing around with removing pyproject.toml and then just playing with --no-build-isolation I've found this one...

Couldn't figure it out for a while because it was only reproducible in Travis CI which drops you into a virtualenv from the beginning automatically. Had to request debug access to their VMs to nail it down...

[webknjaz]

Okay, now I'm hitting this in https://github.com/rtfd/readthedocs.org/issues/5572 somehow.

[webknjaz]

They use explicit --no-site-packages: https://readthedocs.org/projects/octomachinery/builds/8869252/

[webknjaz]

@pganssle it looks like it's time to remove --system-site-packages from the title. I've already hit two instances of the bug with --no-site-packages.

[pganssle]

@webknjaz How does it manifest with --no-site-packages? That may even be a distinct bug.

[webknjaz]

@pganssle it looks the same to me: setuptools fails to use a newer feature because it's not the version of setuptools I specified in pyproject.toml

[webknjaz]

@pganssle one is https://readthedocs.org/projects/octomachinery/builds/8869252/ (pure virtualenv) another is https://travis-ci.com/sanitizers/octomachinery/jobs/190175604#L401-L407 (tox). Both don't have system site packages.

[webknjaz]

Yea, it's another missing feature hit but the root of the issue is still the same.

[pganssle]

I just meant we already know the cause of the problem with --system-site-packages, can you verify if it's the same mechanism here?

[webknjaz]

@pganssle can you point me to the comment with verification plan, please? I've scanned through and it's not obvious which one to follow...

[pganssle]

@webknjaz Just this: https://github.com/pypa/pip/issues/6264#issuecomment-463229977

TBH I remember verifying this by setting a breakpoint or printing something, but I guess at the time it seemed obvious what to do once you knew the cause 😅.

[webknjaz]

I'd need more precise instructions because debugging this with all subprocesses and stuff is PITA and unfortunately I don't have time for this right now... Maybe later.

[rkaplan]

Hitting this issue when installing scikit-learn inside a --system-site-packages virtualenv that uses Python 3.6.8 on Ubuntu 18.04. My setuptools version is 41.0.1.

[naisy]

I also hit this problem.
In my case, downgrade pip. it works.

pip install --upgrade pip==19.0
pip install numpy

[dmccreary]

I also hit this problem when installing the DonkeyCar system that uses many python machine learning libraries. I downgraded to pip 19.0 and everything seems to be working fine. Thank you Naisy.

[Jovons]

Same problem. Setuptools: 47.1.1, python: 3.5.2, pip: 20.1.1 virtualenv: 16.3.0. Only solution works so far is downgrade pip to 19.

[austinjp]

Just another data point, I hit this when installing Huggingface transformers on a Jetson Nano running Ubuntu 18.

$ uname -a
Linux my-hostname 4.9.140-tegra #1 SMP PREEMPT Tue Jul 16 17:04:49 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux

Failed with pip version 20, succeeded with pip version 19 as suggested by @naisy above. Other software: virtualenv 15.1.0, python 3.6.8. Failure message from pip v20 below:

pip install transformers
Collecting transformers
  Using cached transformers-3.0.2-py3-none-any.whl (769 kB)
Requirement already satisfied: tqdm>=4.27 in ./venv/lib/python3.6/site-packages (from transformers) (4.48.0)
Collecting tokenizers==0.8.1.rc1
  Downloading tokenizers-0.8.1rc1.tar.gz (97 kB)
     |████████████████████████████████| 97 kB 1.1 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /whatever/venv/bin/python3 /whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpm6zfgdjl
       cwd: /tmp/pip-install-r8hkyld0/tokenizers
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 99, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /whatever/venv/bin/python3 /whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpm6zfgdjl Check the logs for full command output.

[godlygeek]

I believe this is the root cause of #8823 and https://github.com/pypa/setuptools/issues/2353 - a PEP 517 build of a package is performed, its build requirements are installed into a directory that is added to sys.path for the subprocess that runs the build backend, but they are put _later_ in sys.path than an older version of setuptools. There's a .pth file in the latest setuptools that tries to do some magic, and the magic that it does is busted because the version of setuptools that's earlier on the module search path is an earlier version, incompatible with the .pth file.

A simpler reproducer of this than what I see above is:

python3.8 -m venv --system-site-packages venv38
./venv/bin/pip install --no-binary :all: --ignore-installed python-dateutil

which fails with:

Collecting python-dateutil
  Using cached python-dateutil-2.8.1.tar.gz (331 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 2:
   command: /home/matt/venv/bin/python /home/matt/venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7fsbsn8s/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm
       cwd: None
  Complete output (60 lines):
  Ignoring setuptools: markers 'python_version == "3.3"' don't match your environment
  Collecting setuptools
    Using cached setuptools-50.0.2.zip (2.2 MB)
  Collecting wheel
    Using cached wheel-0.35.1.tar.gz (59 kB)
  Collecting setuptools_scm
    Using cached setuptools_scm-4.1.2.tar.gz (48 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
  ERROR: Exception:
  Traceback (most recent call last):
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
      status = self.run(options, args)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 325, in run
      reqs, check_supported_wheels=not options.target_dir
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 483, in prepare_linked_requirement
      req, self.req_tracker, self.finder, self.build_isolation,
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution
      abstract_dist.prepare_distribution_metadata(finder, build_isolation)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
      self._setup_isolation(finder)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
      reqs = backend.get_requires_for_build_wheel()
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 161, in get_requires_for_build_wheel
      'config_settings': config_settings
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
      raise BackendUnavailable(data.get('traceback', ''))
  pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
      obj = import_module(mod_path)
    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 994, in _gcd_import
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load
    File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 994, in _gcd_import
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load
    File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 678, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 5, in <module>
      import distutils.core
    File "/tmp/pip-build-env-zel77eqn/overlay/lib/python3.6/site-packages/_distutils_hack/__init__.py", line 83, in create_module
      return importlib.import_module('setuptools._distutils')
    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ModuleNotFoundError: No module named 'setuptools._distutils'

  ----------------------------------------
ERROR: Command errored out with exit status 2: /home/matt/venv/bin/python /home/matt/venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7fsbsn8s/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm Check the logs for full command output.

This failure is happening because there are two versions of setuptools on the search path, an older one (from the site-packages directory of the interpreter that created the virtualenv) and a newer one (installed as part of the build requirements for the PEP 517 build). The newer one is shipped along with a .pth file that tries to import setuptools._distutils. The .pth file runs even though another version of setuptools is before it in the module search path, but the import it performs fails because the old version, earlier on the search path, doesn't contain setuptools._distutils.

[godlygeek]

For completeness, when running from a virtualenv with --system-site-packages, the sys.path that I see in the subprocess when the get_requires_for_build_wheel hook is called is something like:

['/home/matt/some_venv/lib/python3.8/site-packages/pip/_vendor/pep517',
 '/tmp/pip-build-env-5v90m1w9/site',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/usr/local/lib/python3.8/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/tmp/pip-build-env-5v90m1w9/overlay/lib/python3.8/site-packages',
 '/tmp/pip-build-env-5v90m1w9/normal/lib/python3.8/site-packages']

When the virtualenv wasn't created with --system-site-packages, I see:

['/home/matt/venv2/lib/python3.8/site-packages/pip/_vendor/pep517',
 '/tmp/pip-build-env-lc6d9yps/site',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/tmp/pip-build-env-lc6d9yps/overlay/lib/python3.8/site-packages',
 '/tmp/pip-build-env-lc6d9yps/normal/lib/python3.8/site-packages']

[jaraco]

In https://github.com/pypa/setuptools/issues/2353#issuecomment-686844357, I've confirmed the issue does not require a virtualenv and can be replicated by simply ensuring that the package being installed has a dependency on a version of setuptools that's incompatible with the installed setuptools.

[uranusjr]

I wonder if python-build’s isolated environment implementation would have this problem. That project would be the best place for this functionality to live in IMO. cc @gaborbernat @FFY00

[gaborbernat]

@uranusjr unless pip plans to use python-build as the wheel builder tool I don't see how python-build will help this issue 🤔

[uranusjr]

That’s exactly my plan 😛 pip already uses pep517 under the hood, and the main reason it does not also use its isolated environment implementation is because pep517 maintainers don’t want anything outside of the core PEP 517 interface to be depended on. It would be best IMO if the isolated environment implementation is maintained outside of pip (so other projects can reuse it), and python-build would be in the perfect position since it does the exactly thing pip needs, and already plans to join PyPA.

[pradyunsg]

pep517 maintainers don’t want anything outside of the core PEP 517 interface to be depended on

Oh, yes -- we recently removed all of that code: https://github.com/pypa/pep517/pull/83, in preparation for python-build. :)

[FFY00]

python-build is ready in my opinion, just waiting on some feedback and then we can make a new release.

[gaborbernat]

Oh my independent how ready python build is we should offer a six month deprecation notice before we remove a core functionality... If we don't want to angry people using it.

[uranusjr]

Can anyone confirm whether python-build have this particular problem though? All discussion would be moot (for this issue specifically) if the implementation is not better than pip’s.

[FFY00]

It's a bit late and I am not on the computer right now, but it seems to me that python-build would not have this issue, since it does not put the site directories in PYTHONPATH, instead it symlinks them to the environment and sets PYTHONHOME. But please test this, I am not in a state to judge the issue properly right now, so I may have misinterpreted something 😝 .

[dHannasch]

We've been running into possibly-the-same-issue. There's a complete minimal repro repo at https://github.com/dHannasch/PyPy-CFFI, trying to pip-install on Travis and failing, if anyone's still lacking for that.

There's a lot of diagnostic output in the Travis log for anyone who wants it, but the key point is:

$ python -c "import setuptools; print(setuptools.__version__)"
41.0.1
The command "python -c "import setuptools; import setuptools.build_meta"" exited with 0.
The command "env-with-site-packages/bin/pip install . --no-build-isolation" exited with 0.
$ env-with-site-packages/bin/pip install .
Processing /home/travis/build/dHannasch/PyPy-CFFI
  Installing build dependencies ... |\/-done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
    reqs = backend.get_requires_for_build_wheel()
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_vendor/pep517/wrappers.py", line 161, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/lib-python/3/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools.build_meta'

I'm not 100% sure whether this should actually be a separate issue. It looks like this might be just the same issue, due to the Travis automatic virtualenv.