Setuptools: tag_build breaks prepare_metadata_for_build_wheel

[uranusjr]

Previously reported in pypa/pip#9446.

This is similar to #1462, but for PEP 517 hooks, so I’m guessing a similar treatment is needed?

[egg_info]
tag_build = dev

[metadata]
name = pkg
version = 0.1

egg_info works as expected:

$ py -c 'from setuptools import *; setup()' egg_info >/dev/null
$ rg '^Version:' < pkg.egg-info/PKG-INFO 
Version: 0.1.dev0

But not the PEP 517 hook:

$ py -c 'import os; from setuptools.build_meta import *; prepare_metadata_for_build_wheel(os.getcwd())' >/dev/null
$ rg '^Version:' < pkg.dist-info/METADATA 
Version: 0.1.dev0dev

[jaraco]

Due to #2536, the fix had to be rolled back and will need to be revisited in light of this new information.

[zzzeek]

im so impressed you understand how setuptools works thanks for the efforts! i still remember ten years ago when nobody could change anything about it.

[zzzeek]

Just checking that I also get this problem if I install straight from pip but also try to add an extras_require like this:

.venv/bin/pip install git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy[postgresql] git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy
Collecting sqlalchemy[postgresql]

# ...

ERROR: Cannot install sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy) and sqlalchemy[postgresql]==1.4.0b2.dev0dev because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy)
    sqlalchemy[postgresql] 1.4.0b2.dev0dev depends on sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy[postgresql])

same thing right? I guess I have to start considering what tag_build=dev is getting me these days as this "dev" tag is now automatically appended in most cases installing from SCM ?

[jaraco]

I'm unsure why you would specify sqlalchemy twice to pip install. Why not do something like this:

~ $ pip-run --use-pep517 'sqlalchemy[postgresql]@git+https://github.com/sqlalchemy/sqlalchemy@master' -- -c pass
Collecting sqlalchemy[postgresql]@ git+https://github.com/sqlalchemy/sqlalchemy@master
  Cloning https://github.com/sqlalchemy/sqlalchemy (to revision master) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-e3mtb3dg/sqlalchemy_946f25bccb604571a8aa49e2d76bac6e
  Running command git clone -q https://github.com/sqlalchemy/sqlalchemy /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-e3mtb3dg/sqlalchemy_946f25bccb604571a8aa49e2d76bac6e
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting greenlet!=0.4.17
  Using cached greenlet-1.0.0-cp39-cp39-macosx_10_14_x86_64.whl (86 kB)
Collecting psycopg2>=2.7
  Using cached psycopg2-2.8.6-cp39-cp39-macosx_10_9_x86_64.whl
Building wheels for collected packages: sqlalchemy
  Building wheel for sqlalchemy (PEP 517) ... done
  Created wheel for sqlalchemy: filename=SQLAlchemy-1.4.0b2.dev0-cp39-cp39-macosx_10_9_x86_64.whl size=1414363 sha256=fdb1f57b673d1d270cd7004e566849fe21d2437003a10c71abc0feaae238b5b4
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-pnjsfy6s/wheels/a5/aa/f0/c81b331ed799c5bb42262fbe93241571ac8c2110330a3d45f1
Successfully built sqlalchemy
Installing collected packages: greenlet, sqlalchemy, psycopg2
Successfully installed greenlet-1.0.0 psycopg2-2.8.6 sqlalchemy-1.4.0b2.dev0

This command also seems to work fine:

pip-run --use-pep517 'sqlalchemy[postgresql]@git+https://github.com/sqlalchemy/sqlalchemy@master' sqlalchemy@git+https://github.com/sqlalchemy/sqlalchemy@master -- -c pass

Note that I use pip-run in place of pip install to save the setup/teardown of a venv.

Perhaps using the more modern specifiers avoids whatever codepaths cause the duplicated tags. Does that help?

[zzzeek]

it's for a tox file. I dont really know how to do what I want:

[tox]

SQLA_REPO = {env:SQLA_REPO:git+https://github.com/sqlalchemy/sqlalchemy.git}

[testenv]

deps=pytest>4.6
     pytest-xdist
     mock
     sqla13: {[tox]SQLA_REPO}@rel_1_3#egg=sqlalchemy
     sqlamaster: {[tox]SQLA_REPO}@master#egg=sqlalchemy
     sqlalchemy: sqlalchemy>=1.3.0

     cov: pytest-cov
     mako
     python-editor>=0.3
     python-dateutil


     postgresql: sqlalchemy[postgresql]
     mysql: sqlalchemy[mysql]
     mysql: sqlalchemy[pymysql]
     oracle: sqlalchemy[oracle]
     mssql: sqlalchemy[mssql]

[jaraco]

How about this:

[tox]

SQLA_REPO = {env:SQLA_REPO:git+https://github.com/sqlalchemy/sqlalchemy.git}

[testenv]
skip_install = True
deps =
     pytest>4.6
     pytest-xdist
     mock
     sqla13: sqlalchemy@{[tox]SQLA_REPO}@rel_1_3
     sqlamaster: sqlalchemy@{[tox]SQLA_REPO}@master
     sqlalchemy: sqlalchemy>=1.3.0

     cov: pytest-cov
     mako
     python-editor>=0.3
     python-dateutil


     postgresql: sqlalchemy[postgresql]
     mysql: sqlalchemy[mysql]
     mysql: sqlalchemy[pymysql]
     oracle: sqlalchemy[oracle]
     mssql: sqlalchemy[mssql]

I was able to run tox -e sqlalchemy,sqla13,sqlamaster,postgresql with that tox file.

[zzzeek]

that looks great! I will try it in a bit but what's this syntax?

sqlalchemy@{[tox]SQLA_REPO}@rel_1_3

@ sign on both sides? I think I saw maybe a random stackoverflow post that referred to that syntax. how does that relate to #egg=<packagename> ?

[zzzeek]

it works, thanks a ton! [edit: sorry, it's not working]

[uranusjr]

You can think of the name @ URL format as the replacement of URL#egg=name. Easier to parse, and does not depend on the URL part to get the package name (which is an issue when comparing URL identity). The format is specified in PEP 508.

[zzzeek]

is "egg=name" deprecated / unnecessary ? (which would be awesome?)

[uranusjr]

I believe it’s not yet formally deprecated (there are still a couple of edge cases), but yeah, basically.

[zzzeek]

I was wrong. it doesn't work, it actually ignores the other packages. we run the tests such as:

tox -e py3-sqlamaster-sqlite-postgresql-mysql

it should be installing all the extras for postgresql and mysql. it's actually just ignoring them:

$ tox -e py3-sqlamaster-sqlite-postgresql --notest
GLOB sdist-make: /home/classic/dev/alembic/setup.py
py3-sqlamaster-sqlite-postgresql create: /home/classic/dev/alembic/.tox/py3-sqlamaster-sqlite-postgresql
py3-sqlamaster-sqlite-postgresql installdeps: pytest>4.6, pytest-xdist, mock, sqlalchemy@git+https://github.com/sqlalchemy/sqlalchemy.git@master, sqlalchemy[postgresql], mako, python-editor>=0.3, python-dateutil
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py3-sqlamaster-sqlite-postgresql inst: /home/classic/dev/alembic/.tox/.tmp/package/2/alembic-1.5.4.dev0.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py3-sqlamaster-sqlite-postgresql installed: alembic @ file:///home/classic/dev/alembic/.tox/.tmp/package/2/alembic-1.5.4.dev0.zip,apipkg==1.5,attrs==20.3.0,execnet==1.8.0,greenlet==1.0.0,iniconfig==1.1.1,Mako==1.1.4,MarkupSafe==1.1.1,mock==4.0.3,packaging==20.9,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.2,pytest-forked==1.3.0,pytest-xdist==2.2.0,python-dateutil==2.8.1,python-editor==1.0.4,six==1.15.0,SQLAlchemy @ git+https://github.com/sqlalchemy/sqlalchemy.git@532026f97f402d6673cd9746f1a7daee99327a68,toml==0.10.2
_________________________________________________________________ summary _________________________________________________________________
  py3-sqlamaster-sqlite-postgresql: skipped tests
  congratulations :)
[classic@photon3 alembic:use_new_pip_format]$ .tox/py3-sqlamaster-sqlite-postgresql/bin/python
Python 3.8.3 (default, May 23 2020, 16:34:37) 
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'psycopg2'

[jaraco]

I think what you've found is an issue with pip. From that tox.ini and with tox -e sqlamaster-postgresql, tox constructs this pip install command:

python -m pip install 'pytest>4.6' pytest-xdist mock sqlalchemy@git+https://github.com/sqlalchemy/sqlalchemy.git@master mako 'python-editor>=0.3' python-dateutil 'sqlalchemy[postgresql]'

Running that command in clean environment goes into deep recursion on projects due to an issue in the new resolver (pypa/pip#8785). I gave up running the command after 10 minutes.

Disabling the new resolver, pip fails to install the postgresql extra:

draft $ env PIP_USE_DEPRECATED=legacy-resolver pip-run 'pytest>4.6' pytest-xdist mock sqlalchemy@git+https://github.com/sqlalchemy/sqlalc
hemy.git@master mako 'python-editor>=0.3' python-dateutil 'sqlalchemy[postgresql]'
Collecting pytest>4.6
  Using cached pytest-6.2.2-py3-none-any.whl (280 kB)
Collecting pytest-xdist
  Using cached pytest_xdist-2.2.0-py3-none-any.whl (36 kB)
Collecting mock
  Using cached mock-4.0.3-py3-none-any.whl (28 kB)
Collecting sqlalchemy@ git+https://github.com/sqlalchemy/sqlalchemy.git@master
  Cloning https://github.com/sqlalchemy/sqlalchemy.git (to revision master) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-lmxkw_v2/sqlalchemy
  Running command git clone -q https://github.com/sqlalchemy/sqlalchemy.git /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-lmxkw_v2/sqlalchemy
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Processing /Users/jaraco/Library/Caches/pip/wheels/cd/56/62/f7e8f1528d26bbdd5d1ea7693b945b4cbef4bbaa0db0bea2b0/Mako-1.1.4-py2.py3-none-any.whl
...
Collecting apipkg>=1.4
  Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Building wheels for collected packages: sqlalchemy
  Building wheel for sqlalchemy (PEP 517) ... done
  Created wheel for sqlalchemy: filename=SQLAlchemy-1.4.0b2.dev0-cp39-cp39-macosx_10_9_x86_64.whl size=1415486 sha256=657f025c579781e86261d89aa200119d575f4b359dfaa316f417f5cd6352ff99
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-e_ib4n0j/wheels/74/f8/a3/5cdbf2d34871d0a2545a526befcf91053137b290f7bcda75c4
Successfully built sqlalchemy
Installing collected packages: pluggy, pyparsing, packaging, attrs, toml, iniconfig, py, pytest, apipkg, execnet, pytest-forked, pytest-xdist, mock, greenlet, sqlalchemy, MarkupSafe, mako, python-editor, six, python-dateutil
Successfully installed MarkupSafe-1.1.1 apipkg-1.5 attrs-20.3.0 execnet-1.8.0 greenlet-1.0.0 iniconfig-1.1.1 mako-1.1.4 mock-4.0.3 packaging-20.9 pluggy-0.13.1 py-1.10.0 pyparsing-2.4.7 pytest-6.2.2 pytest-forked-1.3.0 pytest-xdist-2.2.0 python-dateutil-2.8.1 python-editor-1.0.4 six-1.15.0 sqlalchemy-1.4.0b2.dev0 toml-0.10.2

Or more simply:

$ env PIP_USE_DEPRECATED=legacy-resolver pip-run sqlalchemy 'sqlalchemy[postgresql]' -- -c pass
Collecting sqlalchemy
  Using cached SQLAlchemy-1.3.22-cp39-cp39-macosx_10_14_x86_64.whl (1.2 MB)
Installing collected packages: sqlalchemy
Successfully installed sqlalchemy-1.3.22

My guess is there's already a bug for this unmet use-case.

[jaraco]

Looks like pypa/pip#4957 captures the weakness with the legacy resolver and the fix is "use 2020 resolver".

In the simple case, the 2020 resolver works for me:

$ pip-run sqlalchemy 'sqlalchemy[postgresql]' -- -c pass
Collecting sqlalchemy
  Using cached SQLAlchemy-1.3.22-cp39-cp39-macosx_10_14_x86_64.whl (1.2 MB)
Collecting psycopg2
  Using cached psycopg2-2.8.6-cp39-cp39-macosx_10_9_x86_64.whl
Installing collected packages: sqlalchemy, psycopg2
Successfully installed psycopg2-2.8.6 sqlalchemy-1.3.22

[jaraco]

Attempting to use sqlalchemy@master, however, pip is unable to reconcile the pre-release from master with any public release. Apparently it thinks that sqlalchemy[postgresql] depends on sqlalchemy of the same version, e.g.

sqlalchemy[postgresql] 1.3.22 depends on sqlalchemy 1.3.22

So it's never satisfied.

[zzzeek]

yes, that's the thing where it seems to go down a whole list of SQLAlchemy versions and doesn't find anything, wasn't sure if I should have reported that too as I wasnt sure which circumstances caused it, but if I add --use-feature-resolver-2020 that allows the non-SCM case to work, but the SCM case looks like:

Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.22-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.21-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.20-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.19-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]

... things go for a long time

WARNING: sqlalchemy 0.9.1 does not provide the extra 'postgresql'
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-0.9.0.tar.gz (4.3 MB)
WARNING: sqlalchemy 0.9.0 does not provide the extra 'postgresql'
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-0.8.7.tar.gz (3.5 MB)
ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

thanks for the help so far!