Pip: setuptools dependencies and freeze

[philtay]

setuptools doesn't vendor its dependencies anymore (ref. https://github.com/pypa/setuptools/issues/581 and https://github.com/pypa/setuptools/commit/ff371f18f0076bc63da05334f7e551c1cc29e10d).
Unfortunately this affects pip freeze and breaks our workflow around it. Below you'll find what happens in a clean environment:

$ pip freeze

appdirs==1.4.0
packaging==16.8
pyparsing==2.1.10
six==1.10.0

Of course those are not my dependencies and I don't want to see them in my requirements.txt file. pip already hides pip (itself), setuptools and wheel packages from freeze. It should do (recursively) the same with the dependencies of these packages. Alternatively we can start to treat setuptools as a regular package and show it (with its dependencies) in freeze. The latter option implies that setuptools should not be automatically installed anymore when installing pip.

[jaraco]

The latter option is preferred, including not installing it by default (which it isn't). It will take until the implementation for PEP 518 support arrives for setuptools to be installed only when declared by a package.

[dstufft]

It's probably not going to be tenable to exclude things like six by default, too many things depend on it and it would break a lot of people's workflows. Probably the only real solutions to this are either a smarter method for filtering out (basically, does anything but setuptools depend on these things?) or just waiting for PEP 518 support to land when these things don't need to be installed by default.

[philtay]

@jaraco The official way to install pip also installs setuptools (and wheel). If setuptools is just a regular package it should not be included in any way during pip installation. Moreover pip freeze should treat it just like any other package, showing it along with its dependencies.

[pfmoore]

@philtay Agreed, that's the ideal. We could do that now, but if setuptools isn't installed, all source installs will break (until PEP 518 arrives). So for the short term, we have to install setuptools by default, to give a reasonable user experience. If you're happy with the consequences, you can of course uninstall setuptools yourself. Equally, we could report setuptools in pip freeze, but that would cause it to show up in a lot of people's workflows where they currently don't expect it. The change in setuptools to unbundle made things less ideal right now, but it's not clear that any short term change to temporarily paper over the issue is worth it.

tl;dr; - It'll work fine once PEP 518 lands. In the meantime, it's a compromise.

[philtay]

@pfmoore Then the breaking changes to setuptools (requiring vs vendoring) should have been made after PEP 518. Now all we have is your idea of "compromise". But in the future it'll be better. Like the daily dependency hell we're exposed to isn't enough. Sigh. I'm going to spend the next few days fixing all the problems this caused and, after PEP 518, I'll do it again. Hurrah. Sorry if this message sounds rude or disrespectful. Not my intention. Just frustration.

[pfmoore]

@philtay TBH, I agree with you that maybe the setuptools changes should have been deferred. But that's not a pip issue, which is really what I was commenting on here. Could you not simply pin setuptools to a pre-unbundling version? (Sorry if I'm offering uninformed advice here...)

[philtay]

@pfmoore In fact the problem isn't pip per se, is setuptools which suddenly started to pretend to be a regular package when clearly it isn't. But it will be in the future...

I appreciate your suggestion. We have a fairly complex build system, lot of steps and moving parts. The "simply pinning down" solution is not going to entirely solve the problem. Other work must be done, unfortunately. Moreover during the (possibly long) period of pinning we're not going to get bug and security fixes, new features, etc.

One thing pip is missing and would help us in this situation (and others too) is an option to exclude packages from freeze. Currently there is only the --all option. Too much blank and white. pip freeze --all --exclude setuptools should output of the same of pip freeze --all but without setuptools and its dependencies. pip freeze --exclude django (note the missing --all) should output of the same of a plain pip freeze but without django and its dependencies. Of course if a dependency of the excluded package is also a dependency of a non excluded package it will be printed. An option like this is going to save a fair amount of scripting.

[pfmoore]

@philtay that (pip freeze --exclude) sounds like a potentially good idea. Of course whether it's implemented before or after PEP 518 is the million-dollar question :-)

[philtay]

@pfmoore It's the same idea expressed by @dstufft in a message above (smarter method for filtering out). The only difference is that it gives to the user the ability to decide what exactly "smart" means.

It's not a million-dollar question, I have the answer. It will be implemented as soon as I don't need it anymore, of course. On that day I'll open an issue here on GitHub to let you know so the work can begin and other users will have the opportunity to benefit from this idea....... ;)

[asottile]

Related: #4256

[alex]

Now that setuptools has gone back to vendoring its dependencies, can this be closed, or at least removed from the 10.0 milestone?