Setuptools: Please consider undeprecating setup_requires

[etc0de]

I have run into an issue with the deprecation of setup_requires: the recommended alternative appears to be PEP 517/build-system.requires, but that comes with its own issues:

• apparently it breaks wheels unless you use special options like --no-binary. This breaks very common tools like tox which no longer can install a project if this package is a dependency - also, why would you want to not have wheels? this is wrong, my apologies

• setup_requires isn't installed when invoking hooks.get_requires_for_build_wheel() as called by pip/pep517. However the new build-system.requires is. In practice this is an issue when putting a Cython project as a build dependency for .pxd imports: a .pxd dep is only actually required when building the wheel (where setup_requires will provide it) but not for basic package analysis (where build-system.requires per design provides it too, setup_requires won't). In practice since "providing it" means building the whole package, as a result obtaining any package metadata is greatly slowed down

• pyproject.tomls build-system.requires is declarative but not exhaustive. the problem with that is that the dependencies are then spread over multiple places, e.g. no longer just requirements.txt parsed by setup.py - which can't be parsed similarly in the declarative pyproject.toml file with its static build_system.requires

Edit: a lot of this was initially quite misinformed so I corrected the reasons

[benoit-pierre]

The alternative to setup_requires is PEP 518, not necessarily PEP 517. What's the issue with pip's PEP 518/518 support and wheels?

[benoit-pierre]

It would help if you provided a reproducible example of the issue you are running into.

[benoit-pierre]

Assuming the issue is with your nettools package, using https://github.com/JonasT/nettools/archive/a8b9236f985d468e49953f818dad3dcb1bbe2b77.zip with an added minimal pyproject.toml works fine:

[build-system]
requires = ["setuptools", "wheel", "cython"]

That is pip wheel . from a clean virtualenv (with only pip available) successfully build a wheel.

[benoit-pierre]

Note that setuptools won't include pyproject.toml automatically, so you'll need a manifest entry if you want the source distribution to be complete, in MANIFEST.in:

include pyproject.toml

[etc0de]

Yes that message appears if you use it in a dep that itself is a build requires dep of your main project in tox.

Edit from way later: the message appeared because my package was broken :smile: apologies from future me

[benoit-pierre]

Again, can you point to a reproducible example?

[etc0de]

I'll try to. This involves 3 projects with interdependencies and tox, so I'm not sure how quickly I can find out what constellation breaks it in detail. But it seems to be related to how tox builds the deps, and possibly --only-binary and other options it uses to build the wheel

Edit: also, the other two disadvantages I listed still remain. Declarative can be both good but also a huge disadvantage, and having dependencies somewhat arbitrarily split (there is a criteria of course but it's not very transparent) between two files and wildly different formats just is a little ugly

Edit 2: (I just had a broken package. Realized why later, see below!)

[etc0de]

The exact error I believe was this @ tox/wheel: Could not build wheels for wobblui which use PEP 517 and cannot be installed directly.

[etc0de]

@benoit-pierre Okay, so my MANIFEST.in was wrong and this lead to problems! Now I fixed that, and it almost works, but I get a really strange error on a dependent package and basically it doesn't work for some reason:

https://github.com/pypa/packaging-problems/issues/252
(this issue now breaks it both inside and outside of tox)

But I suppose my ticket here wasn't that constructive given the wheel thing was just wrong, and the other points are admittedly a bit personal taste (I just don't think the multiple files or forcibly declarative approach just for half the things mixes well with how setuptools works right now) so I'll close the ticket for now even though it doesn't actually work, but see the above issue link for that

Edit: unintentional comment spam from my side (sorry again :grimacing: ) ends with this comment of my past self, next comment explains true issue more in-depth

[etc0de]

For what it's worth, I discovered using PEP518 (build-system.requires) instead of setup_requires has the following problem:

• setup_requires is added to returned value of hooks.get_requires_for_build_wheel() as called by pip/pep517, but in practice not installed if one just needs to run setup.py and obtain hooks.get_requires_for_build_wheel() without actually building the wheel - which is necessary e.g. to obtain package metadata like dependencies.
• and build-project.requires always is, per design, installed first, even when just hooks.get_requires_for_build_wheel() is run and the wheel is never built, affecting obtaining package metadata

This is a problem because it slows down obtaining basic package metadata greatly if you put in a build dependency that builds slowly. And many .pxd cimport targets tend to be larger Cython libs with native code compilation that is very slow. Some of my projects went from a few seconds metadata extraction to more than five minutes! And in some environments like https://github.com/kivy/python-for-android this metadata analysis is necessary to automatically map to cross-compile patches and other tasks which turns abhorrently slow due to this.

I'm therefore reopening, and suggesting setup_requires is undeprecated (is it even still deprecated?) or something else is found to provide that behavior, also see discussion here: https://discuss.python.org/t/support-for-build-and-run-time-dependencies/1513/61?u=jtt

[pradyunsg]

It would help if you provided a reproducible example of the issue you are running into.

@JonasT Could you provide a minimal example case that can be used to reproduce the issue you're reporting here?

[pradyunsg]

@JonasT it does not help anyone if you comment at multiple locations for reporting the same issue - especially multiple times. You are reaching (mostly) the same folks on all those channels. To keep track of everything that's been discussed, I need to look at 2 Discourse threads and at least 3 issues spread across 3 repositories.

Could you please keep the discussion in one place, so that your issue and all possible options are all discussed in one place? That would make life easier for the project maintainers.

I'm gonna take the liberty to say that this issue is fine to have further discussion -- I'm happy to defer to setuptools maintainers to tell us a better place.

[etc0de]

@pradyunsg I know this is off-topic, but since I did get multiple remarks like this over the last few days I would like to say something about it:

It can be super complicated to find the right place when not understanding yet what even the problem is. I try to make the best guess but from the outside the packaging ecosystem is really difficult to get a grasp on, so I will be almost inevitably wrong. When someone points out the correct place or I realize a discussion is actually indicating a bug that needs a ticket, I try my best to figure out where this actually should be and correct things. I also try to always refer to the next place, but I may simply mess things up because I usually report for much simpler projects involving no such "moves".

I am therefore hoping you won't hold any grudges and that you'll be forgiving also to people coming after me, because it is more difficult than it may look like :woman_shrugging: my apologies and please have some mercy, we're not all as intimately familiar with things as you are! That would certainly help sometimes :smile:

---

Sorry, I did actually forget about the example. This is a common way my pyproject.toml's look like right now:

[build-system]
requires = ["setuptools", "wheel", "Cython", "wobblui @ https://github.com/wobblui/wobblui/archive/master.zip#egg=wobblui"]
build-backend = "setuptools.build_meta"

wobblui is a UI library I am working on from which I often want to do .pxd cimports for performance reasons. Sadly, it is also quite large so it may take a while to build.

Put the above pyproject.toml into a folder and add a trivial setup.py (e.g. just put from setuptools import setup; setup(name="bla")) and try to examine the metadata like this:

$ python3
Python 3.7.3 (default, May 11 2019, 00:45:16) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pep517.envbuild import BuildEnvironment
>>> from pep517.wrappers import Pep517HookCaller
>>> import pytoml
>>> with open("pyproject.toml") as f:
...     build_sys = pytoml.load(f)['build-system']
... 
>>> env = BuildEnvironment()
>>> with env:
...     hooks = Pep517HookCaller(".", build_sys["build-backend"])
...     env.pip_install(build_sys["requires"])
...     reqs = hooks.get_requires_for_build_wheel({})

This is also the approach https://github.com/kivy/python-for-android uses, and due to the build-system.requires entry it now takes a really long time to run. If you take that exact wobblui ref out of the pyproject.toml and move it into the setup.py as setup_requires=["wobblui ... "] then on the other hand doing that same analysis will only take a few seconds

[pradyunsg]

It can be super complicated to find the right place when not understanding yet what even the problem is. I try to make the best guess but from the outside the packaging ecosystem is really difficult to get a grasp on, so I will be almost inevitably wrong.

Yep, I empathize with you here. This stuff can be pretty complicated so it is perfectly okay to be wrong. :)

I am therefore hoping you won't hold any grudges and that you'll be forgiving also to people coming after me, because it is more difficult than it may look like 🤷‍♀ my apologies and please have some mercy, we're not all as intimately familiar with things as you are! That would certainly help sometimes 😄

I didn't mean to come across as harsh (apologies if I came across as that) and I'm definitely not holding grudges. 🙈

As someone who's been trying to understand what exactly your issue is, I was mostly just suggesting this since it was getting a little difficult to keep track up with how this conversation was evolving.

I also try to always refer to the next place, but I may simply mess things up because I usually report for much simpler projects involving no such "moves".

Yep yep, I noticed. It'll be easier to point folks from other channels to chime in at a single "broad" place (like I've tried to do with this issue now). That way, it's easier for everyone to keep track of the discussion and the conversation can branch/flow as needed on a single platform and it's easier for everyone who has interest in the discussion can see how things evolved.

---

We definitely need to improve the situation with our communication channels to make them easier for new contributors. (btw, I do want to take inputs from you on how we could improve things for this, but let's do that elsewhere)

---

Thanks for the example! That should help us here. :)

[astrojuanlu]

it is declarative. before you say "well that's intentional", the problem with that is that I like to have my dependencies in one place, currently requirements.txt

Comment from the peanut gallery: requirements.txt was never meant to list library requirements https://caremad.io/posts/2013/07/setup-vs-requirement/ and also, I guess you're kind of parsing it inside of your setup.py, which can lead to strange issues because some syntax supported in requirements.txt breaks setup().

[etc0de]

@Juanlu001 first of all yeah that's really a nitpick, the main problem is the exploding package analysis time as illustrated in the example above. (which normally doesn't matter much, but is a huge problem with python-for-android because it causes 5-15 minutes additional build time which is massive) All the others are just nitpicks, but I felt it made sense to list them all for a full picture

As for the unsupported syntax, for me the problem is really the reverse: I'd like to use PEP 508 URLs and requirements.txt doesn't support them so I have quite some parsing to get them back in install_requires from a different format. So install_requires has better support for what I want to do than requirements.txt does - but I suppose this really differs for every use case. I wouldn't need a requirements.txt I just have it as additional convenience for some people who like to use it to set up their dev environment

[etc0de]

I just decided to check for some more real world numbers, and in my current build for checking the deps of 7 packages, I just measured and I lose 13 minutes doing this. I'm pretty sure if there was a way to avoid compiling the cython deps just to examine a package I'd be going through that part in under a minute. So the time loss really is quite massive... :cry: This is on a first-gen AMD zen hexacore, so it's not the slowest machine ever

[McSinyx]

Where was it stated that setup_requires is deprecated if I may ask? I cannot seem to find that in PEP 518.

[etc0de]

@McSinyx https://github.com/pypa/setuptools/issues/1320#issuecomment-380093653 (also, some remarks about it here: https://github.com/pypa/setuptools/issues/1784 )

[McSinyx]

Thanks, that should be officially documented elsewhere, but that's out of the scope of this issue I guess.