pip fails to install projects that have a setup.py and pyproject.toml and there is no documentation on best practices I can locate

[zzzeek]

Environment

• pip version:19.0
• Python version: 3.7
• OS:any

Description

multiple users are reporting that they cannot install SQLAlchemy because it has added a pyproject.toml file. The reason this file was added is becasue it is documented as a best practice for black configuration and for no other purpose.

I do not know why the builds are failing on various environments (Debian, Poetry) or who is doing what wrong. However I am not going to remove the pyproject.toml file until I can get some clarity on what the best practice is for having setup.py and pyproject.toml at the same time is, or if it should never be done, or alterantively if SQLAlhcemy can indeed upgrade to "pep517" in some way such that my users are not complaining.

Related issues:

https://github.com/sqlalchemy/sqlalchemy/issues/5209
https://github.com/sqlalchemy/sqlalchemy/issues/5207
https://github.com/python-poetry/poetry/issues/2197

[pfmoore]

The arrangement you have in sqlalchemy looks fine, so I'd expect the current version of pip to work fine with it. However, pip version 19.0 is out of date, and I believe there was a bug which is what you are seeing.

Please can you check with the current version of pip?

[rvanlaar]

It's weird.

I'm using the python:3.5.2 docker container.
Upgraded pip to 20.0.2 and setuptools to 46.0.0. wheel is already 0.34.2

Removed the poetry.lock file, sqlalchemy and re-added it.
Now it's working.

For bitbucket-pipelines I also had to clear my cache.
It's working now.

[zzzeek]

more problems reported, again, users all use --no-use-pep517 and that resolves.....can this problem be fixed in pip? when pip looks in pyproject.toml and sees nothing, that should mean, hey we aren't pep517...no ? https://github.com/sqlalchemy/alembic/issues/672

[pfmoore]

@zzzeek As noted above, I believe this has been fixed in current versions of pip. Can you provide a reproducible case with the current version of pip and we'll definitely look into it, because this should not be happening.

But without a way of reproducing it, it's hard to know what's (still) unfixed here :-(

[pfmoore]

Update: Apologies, I may be misremembering the sequence of events here. This code looks like we do use PEP 517 if pyproject.toml is present.

But I guess the question is then why does using PEP 517 break your project? We default to the setuptools backend, which should be equivalent to running setup.py in "legacy" mode, so there's clearly still something odd here.

Getting a reproducible test case is still likely to be the best way forward.

https://github.com/pypa/pip/issues/6453 (and related issues around that time) is what I was recalling - it appears that the main issue at the time was editable mode, which isn't relevant (as far as I can see) in this case.

[CaselIT]

Hi @pfmoore. Regarding

But I guess the question is then why does using PEP 517 break your project?

• https://github.com/sqlalchemy/sqlalchemy/issues/5207 the problem is from a broken pip (actually pep517) installed from apt in debian:unstable docker image. So not really pip's fault, but detecting if the pyproject actually contains pep517 instructions before running it would not trigger this
• sqlalchemy/alembic#672 seems that in an offline install wheel is not installed and/or pip tries to install it and fails. I'm not sure what is happening. Again this is triggered by pep517 logic before falling back to using setup.py

I guess that the problem here is that trying pep517 then falling back to setup.py still has side effects, event if ideally it should not.
Maybe a better approach may by avoiding pep517 if no related instructions are found?

[pfmoore]

Maybe a better approach may by avoiding pep517 if no related instructions are found?

The problem here is that in the long term, we will be removing the "directly invoke setup.py" code path. If we do as you suggest, then we'll simply be deferring this problem until then - at which point, there won't be the fallback option of --no-use-pep517 that we have at the moment.

So what I'd much rather do is to identify the issue here now, and ensure that PEP 517 works for you, while still having the "escape hatch" of --no-use-pep517 to allow you to continue working while we understand and fix the underlying issue.

Of the two cases you mention, you say that sqlalchemy/sqlalchemy#5207 is a broken pip, so I assume that's not our issue (let me know if that's incorrect!). And sqlalchemy/alembic#672 seems to be someone trying to build and install but not having wheel and setuptools available in their offline wheel store, so they can't set up the build environment (which needs setuptools and wheel installed). That's nothing to do with PEP 517 - it's build isolation that's the issue here - we set up an isolated build environment and install the needed tools to do the build in there. So the solution there is simply to add wheel to the offline wheel store, surely?

[CaselIT]

Thanks for clarifying. I did not know that the aim was to make pep517 mandatory.

Right now I cannot identify a specific issue, other that pep517 does things differently so new assumptions are required in the install environment.

The only suggestion I have at the moment it that pip could suggest trying --no-use-pep517 if an error happens in pep517 where the correct approach is the fallback to setup.py (and maybe warn that this flag is going way in a future version closer to the removal of it?).
This could hopefully reduce the issues reported to package maintainers because of it?

Edit: note that the better errors are also be useful without suggesting the --no-use-pep517. Like in the second case, if pep517 requires wheel and it not installed surely a clearer error stating it would be useful, even when only pep517 is available?

[zzzeek]

IIUC we are saying that pip has made a silent behavioral change such that new practices such as "add wheel to the offline wheel store" are now required for users environments where this was not required before, and that this new, silent requirement crops up as a result of the pyproject.toml file being present in our packages, and can only be bypassed to have pip behave the old way by using a flag called --no-use-pep517 that users will not generally be familiar with.

This sounds like it is a fully backwards incompatible change that was added to pip, AFAICT, assuming SQLAlchemy has broken no best practice by adding a pyproject.toml file.

Basically, if running pip==<old version that doesnt know about pyproject.toml> works, and then running pip==<new version with this new behaivior> fails for these users, I can't see how that's not a regression in pip.

[pganssle]

This is almost certainly a bug in whatever debian does to pip when it packages it up, not a bug in pip itself.

I can reproduce this by running debian:unstable in a docker container and doing:

apt upgrade && apt install python3 python3-dev python3-pip
python3 -m pip install sqlalchemy

This fails because of a broken pep517 library - a library which is normally vendored into pip, but evidently Debian pulls it out for some reason.

I can then fix the problem by installing pip from PyPI:

root@cb02e61f75c9:/# python3 -m pip install sqlalchemy
Collecting sqlalchemy
  Using cached SQLAlchemy-1.3.15.tar.gz (6.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/share/python-wheels/pep517-0.7.0-py2.py3-none-any.whl/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpnrq41zeb
       cwd: /tmp/pip-install-l9coyky_/sqlalchemy
  Complete output (1 lines):
  /usr/bin/python3: can't find '__main__' module in '/usr/share/python-wheels/pep517-0.7.0-py2.py3-none-any.whl/pep517/_in_process.py'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/share/python-wheels/pep517-0.7.0-py2.py3-none-any.whl/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpnrq41zeb Check the logs for full command output.
root@cb02e61f75c9:/# python3 -m pip install --user --ignore-installed pip
Collecting pip
  Using cached pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Installing collected packages: pip
  WARNING: The scripts pip, pip3 and pip3.8 are installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-20.0.2
root@cb02e61f75c9:/# python3 -m pip install sqlalchemy                   
Collecting sqlalchemy
  Using cached SQLAlchemy-1.3.15.tar.gz (6.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: sqlalchemy
  Building wheel for sqlalchemy (PEP 517) ... done
  Created wheel for sqlalchemy: filename=SQLAlchemy-1.3.15-cp38-cp38-linux_x86_64.whl size=1237080 sha256=52c1dd7ded99149874a5e5dc0a0b6466ce9a0f53d294b69e158bc6e9f2ffa473
  Stored in directory: /root/.cache/pip/wheels/d1/0c/78/33448c81fd8e458d60897744f30462ca39e682637ec9591c0d
Successfully built sqlalchemy
Installing collected packages: sqlalchemy
Successfully installed sqlalchemy-1.3.15
root@cb02e61f75c9:/# 

So I would say that this is not a regression in pip, nor is it really related to PEP 517 - Debian could have broken pip in any number of ways, it just happens to be that in this case they broke the PEP 517 code path.

I think this issue can be closed and hopefully someone can report it to Debian?

[pganssle]

As a workaround, I suggest people who run into this issue use pip install -U --user --ignore-installed pip to get the most recent version of pip from PyPI.

I believe that using a virtual environment will also solve this problem, though I believe Debian messes around with the virtualenv packaging as well :man_shrugging: Using the system python: almost always a bad idea!

[zzzeek]

we have not identified that https://github.com/sqlalchemy/alembic/issues/672 is debian specific. the user has not provided the OS in use.

[zzzeek]

also https://github.com/sqlalchemy/alembic/issues/672 is using a virtual environment already.

[pganssle]

@zzzeek That looks like a very different problem. I have commented in the issue, but again I see no reason to believe that's a problem with pip.

[davidag]

I've also come across this problem in Debian testing/bullseye trying to install Pelican _inside a virtual environment_.

$ git clone https://github.com/getpelican/pelican.git
$ cd pelican ; python3 -m venv venv ; source venv/bin/activate
(venv) $ pip install -e .
Obtaining file:///home/david/projects/pelican
Installing build dependencies ... done
Getting requirements to build wheel ... error
ERROR: Command errored out with exit status 1:
 command: /home/david/projects/pelican/venv/bin/python3 /home/david/projects/pelican/venv/share/python-wheels/pep517-0.7.0-py2.py3-none-any.whl/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpsrpqz9i6
   cwd: /home/david/projects/pelican
Complete output (1 lines):
/home/david/projects/pelican/venv/bin/python3: can't find '__main__' module in '/home/david/projects/pelican/venv/share/python-wheels/pep517-0.7.0-py2.py3-none-any.whl/pep517/_in_process.py'
----------------------------------------
ERROR: Command errored out with exit status 1: 
/home/david/projects/pelican/venv/bin/python3 
/home/david/projects/pelican/venv/share/python-wheels/pep517-0.7.0-py2.py3-none-any.whl/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpsrpqz9i6 Check the logs for full command output.

In this case, a solution has been to force reinstalling pip:

(venv) $ pip install --force-reinstall pip
# Now this works:
(venv) $ pip install -e .

I hope this helps to shed some light on the issue.

[pradyunsg]

Looks like this is resolved! Closing!

[uranusjr]

For additional information, this has been filed as a Debian bug since it is present on Debian-supplied pip only (as mentioned in #9746). To follow up this bug, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954256