Alembic: v1.4.2 missing wheel when pip download and then pip install from .tar.gz

[LGro]

When I pip download alembic==1.4.2 and then pip install alembic from the downloaded .tar.gz file, then I get the following error:

  ERROR: Command errored out with exit status 1:                                                                                                                                                            
   command: /home/myself/anaconda3/envs/tmp3/bin/python /home/myself/anaconda3/envs/tmp3/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c_bv0n7g/overlay
 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /home/myself/workspace/my_project/./ -- 'setuptools>=40.8.0' wheel                                          
       cwd: None                                                                                                                                                                                            
  Complete output (4 lines):                                                                                                                                                                                
  Looking in links: /home/myself/workspace/my_project/./                                                                                                                                               
  Processing ./setuptools-46.0.0-py3-none-any.whl                                                                                                                                                           
  ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)                                                                                                                
  ERROR: No matching distribution found for wheel                                                                                                                                                           
  ----------------------------------------                                                                                                                                                                  
ERROR: Command errored out with exit status 1: /home/myself/anaconda3/envs/tmp3/bin/python /home/myself/anaconda3/envs/tmp3/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix
/tmp/pip-build-env-c_bv0n7g/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /home/myself/workspace/my_project/./ -- 'setuptools>=40.8.0' wheel Check
the logs for full command output.

However, the same procedure works with the versions before, like 1.4.1.

[CaselIT]

Could you share with os are you using and the environment configuration? Does the install happen on an offline machine?
I could not reproduce it on windows.

In the meantime could you try using pip install --no-use-pep517 alembic to check if the install works? This may be connected to https://github.com/sqlalchemy/sqlalchemy/issues/5207

[zzzeek]

anaconda, that's another crazy installer

[LGro]

Thanks for getting back so quickly, much appreciated.

I tried to reproduce it on another Linux machine and couldn't. Let me figure out what is different about the environment where I get the error and if its origin lies with alembic at all.

[LGro]

@CaselIT, good hint about the offline machine. It seems like the pip download & offline install combination only works up to 1.4.1 but not with the most recent version 1.4.2.
I was hoping to get a fully offline installable version including all dependencies with pip download. Do we expect version 1.4.2 to need internet connectivity for a pip install from tar?

[CaselIT]

My hint comes from the error on install of setuptools.
My guess is:

• 1.4.2 added pyproject.toml
• pyproject.toml makes pip first try to install using pep517
• pep517 requires setuptools
• the machine is offline so it does fail the install if setuptools is not already installed.

Could you try using --no-use-pep517 to see if that is indeed the case?

[LGro]

pip install --no-use-pep517 alembic-1.4.2.tar.gz works, so given @zzzeek's comment in sqlalchemy/sqlalchemy#5207 about conforming to best practices it's up to me now to install with the --no-use-pep517 flag whenever I'm offline?

[zzzeek]

no it's up to the tooling vendors to fix their tools to not have this problem. make sure you've upgraded pip and then please report your issue at https://github.com/pypa/pip/issues/7874

[CaselIT]

So my guess above seems correct.
I guess that for now you can use the flag or ensure that setuptools is installed in your offline environment.
Ideally pip should check what's in pyptoject.toml before tring pep517, since alembic is using it only for the black config, so it should not trigger pep517 at all in this case

[zzzeek]

are you using debian ?

[pganssle]

This is almost certainly unrelated to sqlalchemy/sqlalchemy#5207 and I doubt it has anything to do with debian.

Based on this:

ERROR: Command errored out with exit status 1:                                                                                                                                                            
   command: /home/myself/anaconda3/envs/tmp3/bin/python /home/myself/anaconda3/envs/tmp3/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c_bv0n7g/overlay
 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /home/myself/workspace/my_project/./ -- 'setuptools>=40.8.0' wheel                                          
       cwd: None                                                                                                                                                                                            
  Complete output (4 lines):                                                                                                                                                                                
  Looking in links: /home/myself/workspace/my_project/./                                                                                                                                               
  Processing ./setuptools-46.0.0-py3-none-any.whl                                                                                                                                                           
  ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)                                                                                                                
  ERROR: No matching distribution found for wheel                     

It seems to me that the user is trying to install without hitting PyPI at all, but their "wheelhouse" doesn't include all the packages needed to do the installation (namely setuptools and wheel). Not sure why that information was left out of the ticket, because it's critical information.

@LGro Presumably you just need to populate alembic's build dependencies like this (you can replace pip wheel with pip download):

pip wheel setuptools wheel -w /home/myself/workspace/my_project

I'm not sure of a way to do this automatically, though replacing pip download alembic with pip wheel alembic will also solve your problem, since it will build the wheel as part of the "download from PyPI" phase, since it can solve the build-time dependencies at that point.

I don't know of a way to download all the build and run-time dependencies as tarballs ahead of time, that might be a useful feature request for pip.

[CaselIT]

@pganssle wheel is not installed with pip wheel alembic. I've just tried it.
These are the filed produced by the command

-rwxrwxrwx 1 federico federico  159547 Mar 20 20:10 alembic-1.4.2-py2.py3-none-any.whl
-rwxrwxrwx 1 federico federico   75521 Mar 20 20:09 Mako-1.1.2-py2.py3-none-any.whl
-rwxrwxrwx 1 federico federico   16800 Mar 20 20:09 MarkupSafe-1.1.1-cp38-cp38-win_amd64.whl
-rwxrwxrwx 1 federico federico  227183 Mar 20 20:09 python_dateutil-2.8.1-py2.py3-none-any.whl
-rwxrwxrwx 1 federico federico    4877 Mar 20 20:09 python_editor-1.0.4-py3-none-any.whl
-rwxrwxrwx 1 federico federico   10938 Mar 20 20:09 six-1.14.0-py2.py3-none-any.whl
-rwxrwxrwx 1 federico federico 1183131 Mar 20 20:10 SQLAlchemy-1.3.15-cp38-cp38-win_amd64.whl

this is pip list

Package      Version
------------ -------------------
certifi      2019.11.28
pip          20.0.2
setuptools   46.0.0.post20200309
wincertstore 0.2

[CaselIT]

also using the same environment, but selecting the flag --no-pep-517 makes the build work, so wheel is not required by alembic, but by pip. Maybe pip should install wheel since it requires it?

[zzzeek]

@pganssle thanks for your help and attention to these issues!

[CaselIT]

The mystery thicken.

I'm trying in docker alpine and without wheel installed pip wheel fails because MarkupSafe seems to require it.
In a virtualenv in my pc if worked, event though it was not installed.

[pganssle]

@CaselIT The issue is not that pip requires it but that alembic requires it to build a wheel, and pip is moving to a system where the workflow always goes through a wheel.

The idea with PEP 517 is that when building from source distributions, the steps go like this:

1. Front-end (pip) queries a project for its list of build-time dependencies. In this case you are using the default, setuptools and wheel, but you may have things like Cython or flit in there instead.
2. pip creates an isolated environment (like a virtual environment) and installs all the build-time dependencies into it (these do not persist to runtime).
3. pip induces the backend (setuptools, in this case) to build a wheel.
4. pip installs the wheel.

Note that when you do it this way, if you get a wheel from any other source, pip's installation works the exact same way.

What pip wheel does is that it creates a "wheelhouse" which is a store of wheels that you can use to install whatever it is you need (including all its dependencies). It will try to download those wheels from PyPI first before building them.

So I would not expect pip wheel to install wheel in your environment, it will just install it in the isolated build environment when building a wheel for alembic, and then pip install -t MY_WHEELHOUSE alembic will just work (because the dependency on wheel only exists when you are building the wheel, not at install time).

So, like I said, the original poster can either include the appropriate build-time dependencies in their local mini-index or they can use pip wheel to build their mini-index instead of pip download, in which case they won't need the build-time dependencies.

[pganssle]

I'm trying in docker alpine and without wheel installed pip wheel fails because MarkupSafe seems to require it.
In a virtualenv in my pc if worked, event though it was not installed.

This is because you don't have wheel and setuptools installed, and markupsafe has not declared that it has build-time dependencies on setuptools and wheel, because it has not opted in to PEP 517 builds by adding a pyproject.toml. That is one of the problems that PEP 517 and 518 were designed to solve.

I'm guessing it's failing on alpine but not on other platforms because it has platform-specific wheels and doesn't ship one for musl targets (I don't even know if such a thing is even possible, TBH), so it's trying to build everything from source. On any other platform, it would just download the wheel and not need to satisfy any build-time dependencies.

[CaselIT]

@pganssle Thanks for the explanation. It's starting to make sense.

wheels and doesn't ship one for musl targets (I don't even know if such a thing is even possible, TBH)

It actually can happen, since I've experienced the same behavior with other packages, that install with a wheel on other linux distribution, but from source in alpine

[pganssle]

It actually can happen, since I've experienced the same behavior with other packages, that install with a wheel on other linux distribution, but from source in alpine

Sorry, to be clear I meant that I don't know if there's a platform tag for musl targets. I think it's possible that on alpine you can only install universal wheels, locally-built wheels or install from source.

[CaselIT]

Well ok, I guess it does not change the outcome.

Btw, on debian:latest MarkupSafe uses a wheel: Skipping MarkupSafe, due to already being wheel.

[pradyunsg]

Yea, because debian is based on glibc and has wheels covered by the manylinux standard for wheel compatibility.

[brettcannon]

@pganssle there is no musl platform tag to my knowledge because I don't think distutils supports it (see https://github.com/pypa/packaging/issues/259).

[uranusjr]

Tangentially related: pypa/pip#7863. It seems that it is not uncommon to populate a “wheelhouse” with pip wheel or pip download to avoid hitting PyPi, but the current behaviour does not blend very well with PEP 517 build dependencies.

[CaselIT]

I'm trying to add explicit support for pep517 in
https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/1804
and
https://gerrit.sqlalchemy.org/c/sqlalchemy/alembic/+/1805

[LGro]

I'm following all of this attentively. Thanks a lot for looking into this with such rigour!

[sqla-tester]

Mike Bayer has proposed a fix for this issue in the master branch:

Remove pyproject.toml from distribution https://gerrit.sqlalchemy.org/c/sqlalchemy/alembic/+/1808