Pip: Install (if possible) distribution packages to satisfy requirements

[mcepl]

What's the problem this feature will solve?
For many users it would be preferable to install packages from maintained distribution, rather than Python packages from PyPI.

Describe the solution you'd like
Would it be possible for pip to gain an option to prefer system packages to the PyPI ones, by running first the system install (e.g., zypper, dnf, or apt-get) with the list of desired system packages and only then installing remaining packages from PyPI.

I am willing to do the work, but I would need a lot of help and advice where to hook this option to and what special functionality would be useful to other distributions (if they are interested in this option).

User runs (even a normal user) runs

$ pip install foo --user --install-distro-pkgs

pip while collecting necessary requirements finds out that it needs LXML. Instead of downloading and building binary from PyPI, it also discovers that the distribution provides package python-lxml, which satisfies the requirement. Because it is run as a normal user it runs the system packager via sudo to install this packages and then continue with installing remaining packages all the way up to foo.

CC: @warsaw, @hroncok

[hroncok]

As the Fedora maintainer, I'm not interested in this functionality at all. In fact, I think it creates false expectations that the distro should package all of PyPI.

pip is for deployments, developers etc. and allows me to install an arbitrary version of anything from PyPI or PyPI like indexes, or git, or bzr, or disk, or HTTP.

The distro package manager allows me to install integrated versions of various software. Some of the software happens to be from PyPI.

Developers should not install libraries directly from their distribution package manager. Users can install applications and some libraries will be pulled in. But pip users have entirely different needs. Please don't go this path. It's a bad idea.

As a side note, it is possible to create a plugin for the distribution package manager that does something like this: https://github.com/barracks510/dnf-plugin-pip

And as a (second) side note, in your particular example, the user has explicitly said to install packages user-wide, installing system RPM packages would be very much contradicting that.

cc @encukou @torsava @stratakis

[pfmoore]

Agreed. We need to make it clearer to users when they are modifying system owned packages and when they are modifying their own personal installations, not muddy the distinction even further.

[mcepl]

OK, let’s not get stuck on this particular: this command MUST be run by root and the combination with --user is not allowed. That is not what this ticket is about.

[pfmoore]

OK, well we've long been promoting the advice that you should never run pip as root, so I'd say no. If you're working as root, you should use the distro package management tools. If the distro hasn't packaged something you need, then you should get it from PyPI using pip and install it as a normal user for your own use. Mixing the two usages is bad, IMO. Yes, it is probably useful for the user to have one tool that "does what you mean" rather than making you check what's available and choose a different tool, but that sort of "intelligent guesswork" is way higher risk than I'd want to include in pip - especially if it's something we'd advocate running as root.

[mcepl]

@hroncok

I think it creates false expectations that the distro should package all of PyPI.

I would argue quite to the contrary. Given that our users want to use some package Foo as a system-wide installation, they have two options:

1. Use pip as root (hopefully now installing to /usr/local so as not to break actual installed system packages), risking including unknown number of packages unmaintained by the system distributor and non-customized for the particular distro (e.g., lxml), or

2. force distributor to package one package from PyPI.

Thus package creep.

[pradyunsg]

Yea, I'm not warm to this idea either.

What would be the usecases that this would enable (while maintaining or improving others), that are currently not possible?

[chrahunt]

I am against implementing this in pip this for the following reasons:

1. The use case proposed, to avoid building native dependencies, already has a working solution: distributing pre-built binary wheels
2. Supporting users is hard enough when we have to consider incidental interactions with system-provided packages, this would make those interactions much more likely.
3. This does not take into account #988 and I think it would make it much harder. And that issue has a lot of preexisting support and ongoing effort dedicated to it.
4. System Python packages and Python packages on PyPI live in completely different namespaces and there's not a 1:1 mapping between them.
5. Anyone who cares about reproducibility in a dev or execution environment would end up specifying everything explicitly (install this from the system package manager, install this from PyPI) and at that point they might as well just do that with the individual tools.

[NeolithEra]

Hi, maybe you can try a dependency diagnosis website http://www.watchman-pypi.com/dependency_analysis, which can help diagnoze dependency conflict problems and automatically visualize your dependency tree and those of your downstream projects.

Best,
Neolith

[chrahunt]

@NeolithEra, that doesn't seem related to the current issue. This is a feature request for pip, not an issue with installing a specific package which has dependency conflicts.

[chrahunt]

@mcepl first off, thank you for contributing. We really appreciate anyone who takes the time to put their thoughts down and participate like this.

Based on the comments above, we don't really see pip as the right place for this functionality so I will close this.

Please don't hesitate to come back with more information or any new ideas for improving pip or Python packaging. For general Python packaging you can also check out discuss.python.org or pypa/packaging-problems.