Pip: Extras not getting installed when the package that includes the extras has already been installed

Created on 5 Jan 2018 · 10Comments · Source: pypa/pip

Pip version: 9.0.1
Python version: 3.5.2
Operating system: Ubuntu 16.04.3 LTS

Description:

Note: I initially reported this here https://github.com/pypa/setuptools/issues/1247, but was advised to report it here instead.

A similar issue was reported and fixed here for pip.

In our case:

We have a repo with multiple packages in it.
We have the zulip_bots package that depends on requests and simple_salesforce. simple_salesforce further depends on requests[security].
The zulip_bots package also depends on the zulip package in the same repo which also depends on/installs requests. So, it is clear that requests does get installed whether you install zulip or zulip_bots.
We usually use our provision script to set up a dev env and install the zulip and zulip_bots packages.
But recently, when we ran this executable (part of the zulip_bots package), we ran into:

.py pkg_resources.DistributionNotFound: The 'cryptography>=1.3.4; extra == "security"' distribution was not found and is required by requests
The above error led us to conclude that for some reason, requests[security] was not getting satisfied as expected.

It would mean a lot to us if someone could please shed some light on whether this is a re-manifestation of https://github.com/pypa/pip/issues/3189 and https://github.com/pypa/pip/pull/3198 or simply a quirk in our understanding of how extra dependencies are resolved! Thanks! :)

dependency resolution extras

Source

eeshangarg

👍8

Most helpful comment

any updates?

SaschaHeyer on 5 Jun 2020

👍9

All 10 comments

I think I have a similar case. Reproduction is as follows. If I have a requirements.txt file like:

git+https://gitlab.com/datadrivendiscovery/[email protected]
scikit-learn

and I do:

pip install --process-dependency-links -r requirements.txt

(--process-dependency-links has nothing with this report, this is just because the package I am demonstrating with requires it)

Then scipy is not installed, despite being and extra dependency alldeps in scikit-learn[alldeps] from d3m_metadata package (installed from the git repo above). scipy installs correctly if there is no scikit-learn entry in requirements.txt. It seems extras do not get combined as one would expect. This is really tricky because one cannot really know all dependencies packages can have. So If in my project I need d3m_metadata package and just basic scikit-learn, I would like to list only those. But now what happens i that d3m_metadata package does not get all expected dependencies and it fails.

mitar on 9 Jan 2018

I am having the following problem (commented on #3516). Is this issue the cause of that problem?

I am also still having this problem in Pip 9.0.3 and 10.0.1. I'll provide a specific example. PySOA provides a PyTest plugin, and that plugin has a set of requirements, represented in the extra pysoa[pytest]. Projects that use PySOA would put pysoa in their normal requirements (so that the PyTest plugin extras aren't included in the distribution), but would put pysoa[pytest] in their testing extras so that the PyTest plugin extras are installed for testing:
   install_requires={ ... 'pysoa' ...},
   tests_require={ ... 'pysoa[pytest]' ... },
   extras_require={
       'testing': { ... 'pysoa[pytest]' ...},
   },
In our CI environment, when we run python setup.py test, it works perfectly. The pysoa[pytest] requirement, with extras, is installed, and tests pass. However, locally, we install it with pip install -e .[testing], and that approach does not install the extras from pysoa[pytest], so our tests fail locally unless we manually install the extra requirements.

nickwilliams-eventbrite on 26 Apr 2018

Same issue here 😞

Extract from my setup.py:

    extras_require = {
        'api': [
            'Some-Dep[api]'
            ],
        },
    install_requires = [
        'Some-Dep',
        ],

pip install --editable ".[api]" does not install the extra dependencies api of Some-Dep.

fpagnoux on 2 Aug 2018

👍1

This just bit us as well. I've run a few test cases:

$ pip install 'requests[security]' 'requests'  # installs `requests[security]`
$ pip install 'requests' 'requests[security]'  # BUG: only installs `requests`
$ pip install 'requests' 'requests[security]' && pip install 'requests[security]' # BUG: only installs `requests`, then installs `requests[security]` in the second command
$ pip install 'requests[security]' 'requests[socks]'  # BUG: only installs `requests[security]`
$ pip install 'requests[socks]' 'requests[security]'  # BUG: only installs `requests[socks]`

This bug also applies to setup.py dependencies, even if the local setup.py is not itself making use of extras:

$ cat setup.py
import setuptools; setuptools.setup(
    name='test-0', version='1.0.0', description='test 0',
    author='me', author_email='[email protected]', url='https://domain.tld',
    packages=[],
    install_requires=['requests[security]'])
$ pip install .  # installs `requests[security]`
$ pip install requests .  # BUG: only installs `requests` (and `test-0`)

The above test cases have been replicated on:

OSX 10.12.6, Python 3.7, pip 18.0
python:3.5.4 docker image, pip 9.0.1
python:2.7.11 docker image, pip 8.1.2
python:2.7.10 docker image, pip 8.0.0 (updated with pip install pip==8.0.0)

And versions of pip prior to 8.0.0 raise an error when passing requirements with two different extras, as described in #3189.

After going through this exercise, It seems clear to me that #3189 as well as its associated PR #3198 solve only a small portion of this issue, namely running rather than erroring out. It is worth noting that #988 "build a dependency resolver" is still open and describes the above issues quite clearly in point 2:

a. "first found, wins" (where the order is breadth first)

Given the above, I'm pretty sure this boils down to #988, #4957, etc, and many other bugs on various other repos (eg. GoogleCloudPlatform/google-cloud-python#5023, GoogleCloudPlatform/google-cloud-python#5776) all being dupes of this.

TheKevJames on 9 Aug 2018

👍4

I'm seeing this same issue with pip version 19.1 as well. As an example, if I have two extras specified, prod and test, with extras_require that look something like the following:

setup(
    [...],
    extras_require={
        'prod': ['foo'],
        'test': ['foo[bar]'],
    },
)

then I can verify that both pip install .[prod,test] and pip install .[test,prod] will install foo but fail to install it with extras as foo[bar].

I don't have a good solution for the case when two extras specify the same package but with different extras (e.g. for the above example, if test were to require foo[bar] and prod required foo[baz]) other than either merging all of the specified extras together and requiring all of them (e.g. foo[bar,baz]), or simply raising an exception because of the conflict, but either would be better than the current behavior of silent failure.

springheeledjak on 2 May 2019

Hi, maybe you can try a dependency diagnosis website http://www.watchman-pypi.com/dependency_analysis, which can help diagnoze dependency conflict problems and automatically visualize your full dependency tree and those of your downstream projects.

Best,
Neolith