Setuptools: Tests failing to run with BackendUnavailable since pip 19

[jaraco]

It seems that since the release of pip 19, Setuptools' own tests fail to run thus:

setuptools fix_889_and_non-ascii_in_setup.cfg_take_2 $ tox -e py27                                                                                                                       
py27 create: /Users/jaraco/code/main/setuptools/.tox/py27
py27 installdeps: -rtests/requirements.txt
py27 develop-inst: /Users/jaraco/code/main/setuptools
ERROR: invocation failed (exit code 2), logfile: /Users/jaraco/code/main/setuptools/.tox/py27/log/py27-2.log
ERROR: actionid: py27
msg: developpkg
cmdargs: '/Users/jaraco/code/main/setuptools/.tox/py27/bin/pip install --exists-action w -e /Users/jaraco/code/main/setuptools'

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 149, in prep_for_dist
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
BackendUnavailable

py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,apipkg==1.5,atomicwrites==1.2.1,attrs==18.2.0,configparser==3.7.1,contextlib2==0.5.5,coverage==4.5.2,enum34==1.1.6,execnet==1.5.0,flake8==3.6.0,funcsigs==1.0.2,futures==3.2.0,importlib-metadata==0.8,mccabe==0.6.1,mock==2.0.0,more-itertools==5.0.0,path.py==11.5.0,pathlib2==2.3.3,pbr==5.1.1,pluggy==0.8.1,py==1.7.0,pycodestyle==2.4.0,pyflakes==2.0.0,pytest==3.10.1,pytest-cov==2.6.1,pytest-fixture-config==1.4.0,pytest-flake8==1.0.3,pytest-shutil==1.4.0,pytest-virtualenv==1.4.0,scandir==1.9.0,six==1.12.0,termcolor==1.1.0,virtualenv==16.3.0,zipp==0.3.3
________________________________________________________________________________________ summary ________________________________________________________________________________________
ERROR:   py27: InvocationError for command /Users/jaraco/code/main/setuptools/.tox/py27/bin/pip install --exists-action w -e /Users/jaraco/code/main/setuptools (see /Users/jaraco/code/main/setuptools/.tox/py27/log/py27-2.log) (exited with code 2)

I suspect this is related to #1642 if not just another manifestation of the same issue.

[jaraco]

In order to unblock this project, I was going to configure tox to downgrade pip after creating the environment, but tox only allows one install command, and this project has already used up that configuration option to work around another issue, and it's currently not possible to specify a pip version.

[jaraco]

After getting a test failure from tox, I'm able to replicate the failure outside of tox by invoking the same command tox does (or similar):

setuptools master $ .tox/py27/bin/pip install -e .                                                                                                                                       
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 149, in prep_for_dist
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
BackendUnavailable

The problem, of course, is that the virtualenv doesn't have setuptools installed. Setuptools historically has relied on itself to build/install itself... but pep517 is breaking from the convention/expectation that '' is on sys.path.

[jaraco]

the virtualenv doesn't have setuptools installed

I'm wrong about that. The virtualenv _does_ have setuptools installed... it just doesn't show up with pip freeze (probably due to special handling).

[jaraco]

If I update pep517._in_process to inject '' to sys.path in main(), the command no longer fails. That might also explain why python -m pep517.build . succeeds also - because in that mode of invocation, '' will be on sys.path.

[jaraco]

This is particularly difficult to debug because of the subprocess invocation. If I put a breakpoint in the _in_process script to troubleshoot, it exits immediately because there's no pipes for stdin/stdout.

[jaraco]

With lots of edits in the pip codebase, I managed to get a PDB prompt inside the _in_process, and here's what I see:

setuptools master $ .tox/py27/bin/pip install -e .                                                                                                                                       
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies ... done
> /Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py(186)main()
-> if len(sys.argv) < 3:
(Pdb) import sys
(Pdb) sys.path
['/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/site', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python27.zip', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-darwin', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-mac', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-tk', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-old', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-dynload', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/overlay/lib/python2.7/site-packages', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages']
(Pdb) import os
(Pdb) os.getcwd()
'/Users/jaraco/code/main/setuptools'
(Pdb) import setuptools
*** ImportError: No module named setuptools

Why is setuptools not available? I suspect it has something to do with the 'overlay' and the fact that the site-packages of the virtualenv isn't in sys.path.

[jaraco]

~ $ ls /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages                                                                  
ls: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages: No such file or directory
~ $ ls /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/overlay/lib/python2.7/site-packages                                                                 
wheel                  wheel-0.32.3.dist-info

So it seems something is creating a clean environment of dependencies in which the build can happen, containing only the dependencies declared in build-system.requires. That makes sense. As I was tracing the code, I didn't see where that happens.

[benoit-pierre]

I've used pdb-clone for debugging such cases.

The build environment code is in pip._internal.build_env.

So setuptools is a particular case, since it provides its own backend, which mean that for supporting such a use-case, the package source directory must be added to sys path, and this must be done in pip.

[benoit-pierre]

I can only reproduce the issue with Python 3 when tox-venv is not installed, which is also weird.

[jaraco]

I can only reproduce the issue with Python 3 when tox-venv is not installed, which is also weird.

That's probably a pip bug in the build isolation code on a venv.

[benoit-pierre]

No, it's because the versions of pip being used are different...

[jaraco]

Following the acceptance of #1634, setuptools' own test suite now fails.

[daa]

As you have mentioned BackendUnavailable is raised because of ImportError: No module named setuptools and this happens because setuptools is not available in build environment. Speaking about details let's consider how pip builds wheel with pep517:

1. Prepare build environment
2. Install build requirements into that path
3. From prepared build environment invoke build backend hook to get requirements to build wheel - get_requires_for_build_wheel
4. Invoke build_wheel() hook from build environment

To find why setuptools is not available in the build environment inspite of it being in current working directory let's see how isolated build environment is prepared:

1. Create temporary directory with structure resembling python directory layout
2. Create sitecustomize.py (pip/_internal/build_env.py at line 79) which removes original paths from sys.path, adds system site directory and adds build environment site directory.

So step 2 is making setuptools from current directory unavailable for code run inside of build environment.

[pganssle]

Presumably even though @daa's solution works, it means that we can't actually bootstrap our PEP 517, right? We will always be relying on the previously built versions of setuptools.

That will probably work for now, but this seems like it must be a bigger issue, and I think it's one we can't fix in this project, right? The issue is that we supply our own PEP 517 backend, and there's no way to declare that you depend on a PEP 517 backend that exists only in your local directory.

I see two ways forward with this:

1. Fix this problem with pip, adjusting PEP 517 to allow this kind of situation if necessary.
2. Create a new "PEP 517 bootstrapper" backend, that simply delegates all backend tasks to a backend that exists in your local directory, then start using that.

Of course with option 2 the PEP 517 bootstrapper still needs some way to build itself, and all PEP 517 build systems have this problem, so if I understand the problem correctly, the most robust solution is to pursue option 1.

Is there already a ticket tracking this in pip?

[benoit-pierre]

It's another use case that has already been pointed out in https://github.com/pypa/pip/issues/6163, yes. Note that can't just add setuptools if we want setuptools to pip install --no-binary :all: to work.

[pganssle]

It's another use case that has already been pointed out in pypa/pip#6163, yes. Note that can't just add setuptools if we want setuptools to pip install --no-binary :all: to work.

Despite the ocean of ink spilled discussing this, I guess I didn't think to try pip install --no-binary :all:, seems to actually work just fine. Am I doing this wrong, or does the --no-binary :all: flag not apply to the build requirements?

$ pip install --no-binary :all: .
Processing /path/to/setuptools
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Skipping bdist_wheel for setuptools, due to binaries being disabled for it.
Installing collected packages: setuptools
  Found existing installation: setuptools 40.7.2
    Uninstalling setuptools-40.7.2:
      Successfully uninstalled setuptools-40.7.2
  Running setup.py install for setuptools ... done
Successfully installed setuptools-40.7.2.post20190201

It also works if I do export PIP_NO_BINARY=":all:".

[benoit-pierre]

That's with the current pyproject.toml, no?

[pganssle]

@benoit-pierre No, I added setuptools to the pyproject.toml. Without that, the install fails with or without --no-binary.

[benoit-pierre]

That's not how to show the issue:

$ git diff
 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
modified: pyproject.toml
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ pyproject.toml:2 @
[build-system]
requires = ["wheel"]
requires = ["wheel", "setuptools"]
build-backend = "setuptools.build_meta"

[tool.towncrier]
$ python setup.py sdist
$ pip download --no-binary :all: wheel -d dist
$ pip install --no-binary :all: --no-index -f dist -t tmp -v setuptools

[benoit-pierre]

Trying to install from patched source with the index available will work because there's currently a setuptools sdist available on PyPI that itself does not depend on another version of setuptools for building.

[pganssle]

@benoit-pierre Ah, so as soon as we upload this version of setuptools it will start breaking. Good point.

[benoit-pierre]

Even if we discount the --no-binary :all: case, how would you build your first version of a PEP 517 backend? Do you need to have a separate code path to create the wheel?

[pganssle]

Even if we discount the --no-binary :all: case, how would you build your first version of a PEP 517 backend? Do you need to have a separate code path to create the wheel?

That's the idea, yes. In practice what it means is that all PEP 517 backends will use a different, existing PEP 517 backend to bootstrap their first build. It's not terribly different from the situation with compilers, most compilers are built either by another compiler or by an older version of themselves (clang builds itself, for example), and they usually start out being built from other compilers.

It's possible for a project to bootstrap the process by manually creating a wheel, though I imagine no one would do that because wheels for setuptools and flit already exist.

Personally, I'm arguing in the discourse thread for adding a mechanism in PEP 517 to allow us to bootstrap ourselves, but if PEP 517 simply changes to mandate that front-ends must use wheels if available to satisfy the build requirements, that would neatly solve our problem.

Another possibility is that PEP 517 could change to mandate cycle detection and wheels would be required for the first package in a cycle. That seems much more complicated to implement than a bootstrapping mechanism, though.

[pganssle]

At this point the discussion in Discourse does not seem to be going anywhere. I suggest we simply add setuptools to the build-system.requires. This will have the unfortunate side-effect of breaking --no-binary :all: for pretty much everyone using PEP 517 (and possibly everyone?), but I think pretty much everyone using PEP 517 will be broken anyway.

pip can decide whether or not to support --no-binary :all:. My understanding is that as PEP 517 stands right now, it's not intended to support bootstrapping, and I think that even if they revert the "pyproject.toml opts you in to PEP 517" change with pypa/pip#6229, we'll still be affected because we have a [build-system] in our pyproject.toml, so building from source will be broken anyway. Might as well just get into compliance with the spec and let pip decide what to do about --no-binary :all:.

[ncoghlan]

I think the discussionin Discourse is still going somewhere, but the direction it's now going is the direction you describe here: projects that need bootstrapping should add themselves to their own build-system.requires, and then the onus is on frontends to detect that and halt the infinite regress.

[pganssle]

Thinking about this, even if pip --no-binary :all: switches to resolving self-referential source distributions with wheels, if they don't handle long-range cycles, we're still going to break --no-binary :all: if we depend on wheel as part of the PEP 517 build.

That basically means that we cannot have any dependencies in build-system.requires, and we'll need to vendor wheel, so I think we have three options:

1. Specify 'setuptools' and 'wheel' in our build-system.requires and let pip fix --no-binary :all: or not, as desired.
2. Opt out of PEP 517 forever, meaning that we will never be installable by a PEP 517-only frontend except as a wheel.
3. Vendor wheel in our source tree and wait for PEP 517 to be extended to allow for self-bootstrapping builds.

I don't think number 3 is going to happen, so we're really choosing between options 1 and 2. I'm in favor of option 1, but we may want to give pip some fair warning that we'll be doing it, since it will likely cause at least some breakages.

[ncoghlan]

Reading https://github.com/pypa/pip/issues/6222 (and based on the other threads), I think the pip folks already agree that the current interaction between --no-binary :all: and PEP 517 needs work, so Option 1 seems like a reasonable choice at the setuptools level to me.

[pganssle]

So after the absurd amount of discussion on this, the final decision (around 9 months ago) was to modify PEP 517 to add backend-path for bootstrapping, but no front-ends are actually spec compliant as far as I can tell. pip doesn't seem to support it at all.

This makes it very difficult to install our own library from git. I suggest that we just go ahead with this configuration for now:

[build-system]
requires = ["wheel", "setuptools>=40.0"]
build-backend = "setuptools.build_meta"

We can continue to exclude pyproject.toml from the source distribution so that pip install setuptools --no-binary :all: will continue to work for now. Once pip implements this we can probably switch over to using backend-path.