Setuptools: Please allow "file:" for setup.cfg install_requires

[nedbat]

The new declarative setup.cfg syntax is fabulous. It's wonderful to have two-line setup.py files!

I tried to use "install_requires = file: requirements/base.in" and found that "file:" wasn't supported there. Our pip-compile workflow works really well, and it would be great to be able to start with base.in as our source of truth, and easily use it in our setup.cfg.

As a design point, I'm not sure it makes sense to special-case which fields can use which kinds of info-gatherers. Why can't I use "file:" for any field? I can understand if the implementation requires it, but the more the developer can choose how to structure their own world, the better.

[pganssle]

There are two things I'd like to address here:

I tried to use "install_requires = file: requirements/base.in" and found that "file:" wasn't supported there. Our pip-compile workflow works really well, and it would be great to be able to start with base.in as our source of truth, and easily use it in our setup.cfg.

My intuition is that I think that supporting file: in this context would mainly serve to encourage people to use their (version-locked) requirements.txt for their install_requires, which I consider to be an anti-pattern, since it is almost certainly the case that this should go the other direction. If there were a linter for build configurations, I'd want "imports a requirements.txt" to be in there. If 99% of people are going to use this feature to enable an anti-pattern, and 1% have a legitimate use like @nedbat's, I am -1 on it. For me, setup.py exists to enable "I know what I'm doing" use cases like this. (Though #1805 throws a bit of a monkey wrench in this, since static requirements in setup.py would actually be treated differently from requirements read in from a file).

Regardless of how this turns out in setuptools, it would be really nice if pip-compile could be used with arbitrary projects rather than just requirements.txt files, since that would make it a lot easier for install_requires to be your source of truth from which a lock file is generated.

As a design point, I'm not sure it makes sense to special-case which fields can use which kinds of info-gatherers. Why can't I use "file:" for any field? I can understand if the implementation requires it, but the more the developer can choose how to structure their own world, the better.

This is probably not a big deal for file: itself, but I am mildly worried about the idea of inventing a whole generic DSL for our "declarative" build files. The more complexity we add, the harder those files will be to actually parse or statically analyze, and eventually we'll get back into the situation we're in now, where we desperately want to move to more constrained build specs.

I would say I am -0 on both of these propositions. I'm instinctively conservative here and don't see a super compelling use case, though I will admit that for people like @nedbat who have many different contextual dependencies, it makes sense to provide them some ability to refactor and avoid having a huge monolithic configuration file.

One possible compromise here could be supporting file:, but throwing an error if == is specified in the included file. Surely this would be frustrating for people who want to use their locked requirements.txt file, but if the main reason we're not supporting file: at all is to frustrate people who want to reuse their locked requirements.txt file, this might be a lower-impact way to do that that solves the "anti-pattern" portion of the issue.

[layday]

My intuition is that I think that supporting file: in this context would mainly serve to encourage people to use their (version-locked) requirements.txt for their install_requires, which I consider to be an anti-pattern ...

This is something you should expect to see more of now with pipx simplifying the installation of Python CLIs from PyPI. It _is_ an anti-pattern in the context of libraries but the reality is setuptools and PyPI are also used for application delivery.

[pganssle]

I realized that this is a duplicate of #1074, which was closed as wontfix.

@jaraco Do you think anything has changed since #1074?

[nedbat]

Just because a feature can be mis-used does not mean it should be forbidden. I have a reason to use this, and I know what I am doing. Is it better that I hack together something with sed and make?

[jaraco]

Do you think anything has changed since #1074?

In my mind, things have changed as there is a more compelling use-case presented above.

I'd be open to a PR that (a) adds support for file: in this field and (b) documents officially the caveat that packagers should resist the temptation to simply reference requirements.txt _especially_ if that file contains pinned requirements.

reality is setuptools [is] used for application delivery

I would recommend that if pinning is done, it should be done outside of setuptools using tools like pip-compile and that the underlying package (even if a final deployment of a local application) should specify its abstract requirements and not some set of concrete requirements, but I won't go as far as to disallow that usage.

I don't think the pipx example is a good one, as applications like tox and twine shouldn't have pinned requirements for the same reason that libraries shouldn't have pinned requirements--the end user may have different demands than the application had at the time it was last released (including honoring environmental variation).

[pganssle]

Just because a feature can be mis-used does not mean it should be forbidden.

I agree that that's true, but it really depends on baseline rates. If we add this and for 99.9% of people it is mis-used and it's still possible to accomplish it without this feature, I think we're doing people a disservice by giving them a feature that's so easy to misuse.

I think that's the case here except that encouraging people to use setup.py for their "escape hatch" in this scenario may impact our ability to provide guarantees about deterministic dependencies.

It seems like the safest way to do this would be to allow setup.cfg to parse a file in a format that pip doesn't understand (so there's no danger of using it with a requirements.txt), and try and get pip-compile changed to allow for compiling . and .[extras] (and possibly "just the dependencies of .").

[layday]

It seems like the safest way to do this would be to allow setup.cfg to parse a file in a format that pip doesn't understand (so there's no danger of using it with a requirements.txt), and try and get pip-compile changed to allow for compiling . and .[extras] (and possibly "just the dependencies of .").

I don't think pip-compile users would see the need to specify their dependencies in a separate file in a format exclusive to setup.cfg if pip-compile were able to extract dependencies from pip requirement specifiers. Would pinned dependencies be forbidden in this format or are you banking on obscurity?

[pganssle]

I don't think pip-compile users would see the need to specify their dependencies in a separate file in a format exclusive to setup.cfg if pip-compile were able to extract dependencies from pip requirement specifiers.

Not sure what you mean by "extract dependencies from pip requirement specifiers", but if you look at Ned's original use case I could definitely imagine someone with that many separate extras dependencies not wanting to combine those into one huge setup.cfg. I suppose the two feature requests are separable.

Would pinned dependencies be forbidden in this format or are you banking on obscurity?

I think neither of these are the case. The problem to solve here is that a lot of people don't understand the difference between requirements.txt and install_requires and want to "single source" their install metadata by having setuptools pull data from their requirements.txt. If we make it easy to do that, people will just do it, not realizing that it's the wrong thing to do. We want the right thing to do to be easy and the right thing to do to be hard (we don't need to make it impossible).

As long as it's not easy to output the setup.cfg format with pip freeze or pip-compile, and it's not easy to pip install the setup.cfg format other than through the normal mechanism of pip install ., then the design objective is satisfied. Sure, someone could write a converter between the two things, but hopefully when people google, "How do I convert between requirements.txt and setuptools requirements file?" they will find some documentation saying, "You shouldn't, that's why they have two different formats."

As an aside, this reminds me of the Susan B. Anthony Dollar, which was a US$1 coin that looks almost identical to a quarter ($0.25). People complained because it was very easy to confuse the two and give someone $4 when you meant to give them $1. It was such a similar shape and size that you could even put the $1 coins in vending machines designed for $0.25 coins. My reasoning for using a different format is similar to the reason for making $1 coins a different color and size than $0.25 coins: install_requires and requirements.txt serve different purposes and we don't want to confuse people by making them similar enough that a lot of tooling almost (but not always) "works".

This is just an idea, though. I'm willing to be persuaded otherwise.

[layday]

Not sure what you mean by "extract dependencies from pip requirement specifiers" ...

I mean these: . and .[extras]. There is an open issue on the pip-tools tracker to support build system-agnostic requirements, meaning 'locked' requirements can be compiled from any pip-installable package.

... but if you look at Ned's original use case I could definitely imagine someone with that many separate extras dependencies not wanting to combine those into one huge setup.cfg. I suppose the two feature requests are separable.

If no other program is able to interoperate with the new file - the only benefit there being to reduce the size of setup.cfg - I think it'd fall kind of flat. readme and version take a file: value because single sourcing the readme and version number _is_ desirable.

The problem to solve here is that a lot of people don't understand the difference between requirements.txt and install_requires ...

I think a lot of people don't appreciate that these are different formats. That you are able to pipe a requirements.txt into install_requires and have it work (for the most part) appears to be an accident of history. But do a lot of people who work with setuptools (package authors) not understand that pinning their dependencies is poor practice? I'm not so sure.

[merwok]

Another idea: pip-compile can read source requirements from setup.py, so it could be extended to support setup.cfg too.

[zed]

Note: requirements.in and requirements.txt are substantially different: the *.in file doesn't pin all versions as a rule otherwise there would be no need to pip-compile it into requirements.txt.

I understand the logic of avoiding requirements.txt in setup.cfg but what is the argument against: install_requires = file: requirements.in?

[ThiefMaster]

CI+caching is another usecase where this would be useful: I'd like to cache my venv folder (using e.g. the hash of requirements.txt as a cache key), but right now this means i have to duplicate the requirements in setup.cfg and requirements.txt for this to work.

[s0undt3ch]

I too have the use case where I want to reuse my .in requirements for packaging while I also maintain a locked .txt file for CI purposes.

At the bare minimum, it would be awesome if setuptools provided an entry point so that a third party library could read requirements files and set the various _require on the distribution instance, similar to how setuptools_scm sets the version attribute. In this case, setuptools would not be "responsible" for the misuse, and that third party library would.

Would you be open for said entry-point?

[s0undt3ch]

So, for those interested, I managed to hack something which might also help you, https://pypi.org/project/setuptools-declarative-requirements

[sevens-ef]

Another idea: pip-compile can read source requirements from setup.py, so it could be extended to support setup.cfg too.

Some more info on this: if you have a setup.py that gets its install_requires from setup.cfg, this seems to already work (did a quick test with pip-compile 5.3.1), and there seems to already be an issue to support setup.cfg directly (https://github.com/jazzband/pip-tools/issues/1047).

[jaraco]

Another idea: pip-compile can read source requirements from setup.py, so it could be extended to support setup.cfg too.

I'd take this a step further and argue that pip-compile (and other projects that need to inspect the requirements for the source of a project) should use pep517.meta.load to extract/parse metadata (and requirements). Such an approach would not only support setup.py and setup.cfg files, but would also support extras and builders other than setuptools.

Example:

tempora master $ pip-run -q pep517 -- -c "from pep517 import meta; print(meta.load('.').metadata.get_all('Requires-Dist'))"
['pytz', 'jaraco.functools (>=1.20)', "sphinx ; extra == 'docs'", "jaraco.packaging (>=3.2) ; extra == 'docs'", "rst.linker (>=1.9) ; extra == 'docs'", "pytest (!=3.7.3,>=3.5) ; extra == 'testing'", "pytest-checkdocs (>=1.2.3) ; extra == 'testing'", "pytest-flake8 ; extra == 'testing'", "pytest-cov ; extra == 'testing'", "jaraco.test (>=3.2.0) ; extra == 'testing'", "backports.unittest-mock ; extra == 'testing'", "freezegun ; extra == 'testing'", "pytest-freezegun ; extra == 'testing'", 'pytest-black (>=0.3.7) ; (platform_python_implementation != "PyPy") and extra == \'testing\'', 'pytest-mypy ; (platform_python_implementation != "PyPy") and extra == \'testing\'']

[nbraud]

CI+caching is another usecase where this would be useful: I'd like to cache my venv folder (using e.g. the hash of requirements.txt as a cache key), but right now this means i have to duplicate the requirements in setup.cfg and requirements.txt for this to work.

Same thing here: preinstalling dependencies in a container image (that's rebuilt when dependencies change) can yield <10s CI times (incl. testing on all supported versions of CPython, linting, etc.) on most of my projects, but doing that in a sensible way requires having the list of dependencies in a separate file.

Quick CI times can do a lot for project health and developer experience:

• waiting on CI means contributors are more tempted to stuff somewhat-unrelated changes in the same PR;
• waiting also interrupts the “flow”, forcing the developer to either stop coding (forcing them out of the productive headspace if they reached it) or juggle more concurrent PRs (higher cognitive load, risk of merge conflicts, ...)

I know it's perfectly [feasible] with setup.py, but given that it's pretty-much always the one thing that requires writing that file, file: support in the *_requires options would let many projects get rid of setup.py altogether.

[nbraud]

PS: I just [added to bork] (a dev/release automation tool for Python projects) the ability to dump the list of dependencies, using pep517.meta as per @jaraco's [comment]. This is far from a complete solution, but at least it will be a stopgap for Bork users.