Pip: Pip > 19.x does not fall back to legacy setup.py installation mode if wheel package missing

[mhaas]

Environment

• pip version: pip 18.1 (working), 19.2.3 (not working) and 20.1.1
• Python version: Python 3.7.5
• OS: Darwin host 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64

Description

First, some words about my environment before we go into the details:

I am building a Cloudfoundry (CF) app using the CF Python Buildpack. As I have some private dependencies, I am vendoring my dependencies: all dependencies are downloaded using pip download into a vendor/ directory. The app is pushed to CF and the buildpack then basically installs these dependencies inside a container.

With the dependencies vendored along the app, the Python buildpack to use --no-build-isolation (Buildpack implementation). I assume this is done to avoid downloading dependencies.

Now, my basic issue that this worked fine with Pip 18.x (using the CF provided Python 3.6) and no longer works with Pip 19.x (CF provided Python 3.7). Details to reproduce the general issue locally follow below:

Using pip 18.x, I am able to install alembic==1.4.2 perfectly fine when using --no-build-isolation in an environment where the wheel package is missing:

(venv) ➜  tmp pip install --no-build-isolation alembic==1.4.2
Looking in indexes: --company internal --
Collecting alembic==1.4.2
Requirement already satisfied: SQLAlchemy>=1.1.0 in ./venv/lib/python3.7/site-packages (from alembic==1.4.2) (1.3.17)
--- snip ---
Installing collected packages: alembic
Successfully installed alembic-1.4.2
You are using pip version 18.1, however version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Starting with pip 19.x, the presence of pyproject.toml triggers the PEP-517 backend and the build fails because the wheel package is missing.

(venv) ➜  tmp pip install --no-cache-dir --no-build-isolation alembic==1.4.2
Looking in indexes: --company internal --
Collecting alembic==1.4.2
  Downloading http://nexus.wdf.sap.corp:8081/nexus/content/groups/build.milestones.pypi/packages/alembic/1.4.2/alembic-1.4.2.tar.gz (1.1MB)
     |████████████████████████████████| 1.1MB 1.7MB/s
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/d073668/tmp/venv/bin/python3.7 /Users/d073668/tmp/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/tmpbsl6ggmx
         cwd: /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic
    Complete output (18 lines):
    running dist_info
    creating /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info
    writing /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/dependency_links.txt
    writing entry points to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/entry_points.txt
    writing requirements to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/requires.txt
    writing top-level names to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/top_level.txt
    writing manifest file '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/SOURCES.txt'
    reading manifest file '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.jpg' under directory 'docs'
    warning: no files found matching '*.sty' under directory 'docs'
    warning: no files found matching '*.dat' under directory 'tests'
    warning: no files found matching 'run_tests.py'
    no previously-included directories found matching 'docs/build/output'
    writing manifest file '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.egg-info/SOURCES.txt'
    creating '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-zygplwh9/alembic/pip-wheel-metadata/alembic.dist-info'
    error: invalid command 'bdist_wheel'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/d073668/tmp/venv/bin/python3.7 /Users/d073668/tmp/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/tmpbsl6ggmx Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The same happens with the latest pip, 20.1.1:

(venv) ➜  tmp pip install --no-cache-dir --no-build-isolation alembic==1.4.2
Looking in indexes: --company internal --
Collecting alembic==1.4.2
  Downloading http://nexus.wdf.sap.corp:8081/nexus/content/groups/build.milestones.pypi/packages/alembic/1.4.2/alembic-1.4.2.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 1.8 MB/s
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/d073668/tmp/venv/bin/python3.7 /Users/d073668/tmp/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/tmp1se8pobp
         cwd: /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-install-vn6tna9b/alembic
    Complete output (18 lines):
    running dist_info
    creating /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info
    writing /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/dependency_links.txt
    writing entry points to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/entry_points.txt
    writing requirements to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/requires.txt
    writing top-level names to /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/top_level.txt
    writing manifest file '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/SOURCES.txt'
    reading manifest file '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.jpg' under directory 'docs'
    warning: no files found matching '*.sty' under directory 'docs'
    warning: no files found matching '*.dat' under directory 'tests'
    warning: no files found matching 'run_tests.py'
    no previously-included directories found matching 'docs/build/output'
    writing manifest file '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.egg-info/SOURCES.txt'
    creating '/private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-modern-metadata-8gkkse4l/alembic.dist-info'
    error: invalid command 'bdist_wheel'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/d073668/tmp/venv/bin/python3.7 /Users/d073668/tmp/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/tmp1se8pobp Check the logs for full command output.

Expected behavior

I would appreciate if pip would fall back to the legacy installation mode if PEP-517 fails. This was always the case in pip before 19.x: it would check if wheel was installed and if wheel could not be imported, pip would execute setup.py install.

With this hypothetical fallback in place, my use case would continue working as before.

I spent some time looking into this issue, looking for different workarounds and options. I also appreciate that the pip project and the wider ecosystem is currently moving to a better and brighter packaging future. I am very grateful for this work. I also understand that the new behavior is desirable in general, but just so happens to break my existing use case. I will raise an issue separately with the CF Python Buildpack because they should either stop using --no-build-isolation or pre-install wheel (or some variation thereof).

My notes:

• https://github.com/pypa/pip/issues/7874 - basically this issue without my "buildpack" twist, opened by the alembic maintainer himself. This issue seems to have been closed as being related to a Debian packaging bug. Given my experience on macOS, I believe this is not a Debian problem
  
  
  
  • As a side node: alembic is now removing pyproject.toml for the time being: https://gerrit.sqlalchemy.org/c/sqlalchemy/alembic/+/1808
  
  
• Use of --no-use-pep517 is not possible for me. This breaks installation of other source dists which rely on PEP517 (see https://github.com/pypa/pip/issues/6230)
  
  
  
  • Since the CF Python buildpack basically does pip install -r requirements.txt, it is not possible for me to disable PEP-517 only for affected packages
  
  
  • Additionally, whitelisting individual packages seems to be a lot of work
  
  
• It would certainly be an option to also vendor all build dependencies. However, this is currently not supported natively by pip: https://github.com/pypa/pip/issues/7863
• --no-build-isolation is documented as requiring wheels and setuptools pre-installed. Given this, I will also file a bug with the CF python buildback.

How to Reproduce

With pip 18.1:

1. pip uninstall wheel || true
2. pip uninstall alembic || true
3. pip install alembic==4.2.2

With pip 19.2.3:

1. pip uninstall wheel || true
2. pip uninstall alembic || true
3. pip install alembic==4.2.2

Output

See intro above.

[sbidoul]

As you noted,

--no-build-isolation is documented as requiring wheels and setuptools pre-installed. Given this, I will also file a bug with the CF python buildback.

so that is indeed the best thing to do.

In upcoming versions, pip will warn about the setup.py install fallback as discussed in #8102, with the goal of ultimately deprecating it. So I propose to close this issue and track the topic in #8102.

[pradyunsg]

I propose to close this issue and track the topic in #8102

I'm in agreement. Let's wait for @mhaas's response (as you've indicated via the S: awaiting response label)

[mhaas]

Hi @sbidoul, @pradyunsg,

thanks for the quick response. I read through the linked #8102 - it was a bit hard to follow what was in scope at first, but the discussion eventually converged.

Just to ensure that we are all talking about the same thing in this issue here, let me use the nice flow explanation posted by @xavfernandez here. Enumeration added by me:

Currently:

1 if pyproject.toml exists, use PEP-517/518
2. otherwise, expect a setup.py:
  2.1 if wheel is installed, run setup.py bdist_wheel and install the wheel
  2.2 otherwise fallback to setup.py install

The linked issue #8102 is mainly concerned with path 2.. The issue I opened here is that path 1 fails completely if wheel is not available - and that the same setup was working on 18.x and broken in 19.x due to the use of the PEP-517/518 backend in 19.x

Basically, a fix could be:

1. if pyproject.toml exists
  1.1 if wheel is installed or installable, use PEP-517/518 backend
  1.2 if wheel is not avaibale, fallback to setup.py install
2. otherwise, expect a setup.py:
  2.1 if wheel is installed, run setup.py bdist_wheel and install the wheel
  2.2 otherwise fallback to setup.py install

Now, I still understand that the legacy code is deprecated or will be deprecated soon - but just making sure we are on the same page here.

Thanks

Michael

[pfmoore]

@mhaas That's useful to clarify. But the whole point of PEP 518 is that it makes 100% explicit what a project requires in order to build. In the "build isolation" case, pip will manage automatically the process of installing those requirements. But in the case of --no-build-isolation, the user is explicitly assuring pip "I have taken into account the requirements as stated in pyproject.toml and made them available". If that assurance turns out to be false, pip should report that fact, but (IMO) definitely should not try to work around it. Yes, we have a legacy code path that avoids the need for wheel, but why would we special-case that package? What if the project was based on flit and flit wasn't installed?

Maybe pip could provide a better error message, by checking that the build requirements are present and if they are not, failing with an error

"--no-build-isolation specified, but foo > 2.0 not available (from package-x)"

indicating what was missing and what package required it? But that's as far as I'd want to go. This is a user error, as --no-build-isolation means you've agreed to manage the build environment, but we can help you better understand what went wrong.

[mhaas]

@pfmoore I agree with you. It would be great to have a better error message for this particular case.

You are also entirely correct that simply checking for wheel would not be enough to enable fallback to the legacy code setup.py install code path.
I think my argument should have been more "if PEP-517/518 mode fails for any reason, try setup.py install". Anyways, I understand that setup.py install might go away and the PEP517/518 is the future - this includes the hard requirement that setuptools.build_meta:__legacy__ absolutely wants to build a wheel. And this also works out of the box with pip, as you pointed out correctly!

On to more useful points:

We are definitely on the same page about --no-build-isolation. After I created this issue, I also created the corresponding issue in the CF buildpack and while writing that down, I actually understood how broken their current use of --no-build-isolation actually is.

While I was writing, I was also thinking about their use case, which is probably "install all packages from a pre-built package cache", aka "vendoring your dependencies". With the advent of PEP-518, it is now much harder to actually do that.

Traditionally, you would do a pip download -d vendor/ -r requirements.txt and then upload your app to CF. The CF buildpack will then do something like pip install -r requirements.txt --no-build-isolation --ignore-installed --exists-action=w --no-index --find-links=$(pwd)/vendor.

There is currently no nice way of implementing this use case with the current PEP-517/-518 based backend. The main missing piece here is #7863, I believe.

As a side note: I am not trying to argue here or have a discussion for the sake of having a discussion - I understand that pip is moving forward and I am trying to understand how I can preserve my (and possibly others') use of pip. So, please take this as constructive feedback, and not as me demanding things be fixed :)