Pip-tools: Pip-compile ignores extras_require when parsing setup.py

[davidjlloyd]

The documentation for python-wheel describes ways to define conditional dependencies: http://wheel.readthedocs.io/en/stable/index.html?highlight=extras_require#defining-conditional-dependencies

The most well-supported of these is to use the parameter extras_require to specify dependencies for specific environment markers. For example:

from setuptools import setup
setup(
    name='mypackage',
    extras_require={
        ':platform_python_implementation=="CPython"': ['python-cjson'],
    }
)

This allows us to install optimisations for CPython which are not available on other implementations of Python. This works correctly for pip/setuptools (e.g. pip install . or python setup.py install) but is ignored by pip-compile.

Environment Versions

1. OS Type (Linux/Ubuntu)
2. Python version: 2.7
3. pip version: 9.0.1
4. pip-tools version: 1.11.0

Steps to replicate

1. Create a package with a conditional dependency specified using extras_require in setup.py.
2. Run pip-compile on the package in an environment which meets the conditions, and in one that does not.

Expected result

The generated requirements.txt should contain the conditional packages for the environment which meets the conditions, and not otherwise.

Actual result

The conditional dependencies are not present in any requirements files.

[suutari]

Does it make a difference if you specify the conditional dependency in the install_requires argument?

Here's an example:

setup(
    name='mypackage',
    install_requires=[
        'python-cjson ; platform_python_implementation=="CPython"',
        ...
    ])

[davidjlloyd]

@suutari Turns out it doesn't make any difference. Setuptools Distribution class separates out any installation requirements with environment markers, and then merges them into extras_requires, so the end result is the same from the perspective of the returned Distribution object! On the plus side, this means that any solution to this issue should work for both methods of specifying conditions on requirements. I was using the older method (via extras_require) because this is supported since setuptools 18, whereas the method you suggest is only available since setuptools 36.2 and is not fully supported by pip.

The Distribution object provides an extras_require attribute containing the necessary information, but I'm not sure how we would want to process it. We either need to evaluate if the marker conditions are met and include/omit based on that, or we need the markers to exist in the generated requirements file. The latter is nice as it allows single requirements files to work across all environments, but then this should probably be applied to inherited dependencies too. Currently inherited dependencies are included based on marker conditions, and markers are not preserved in the generated requirements file.

[vphilippon]

Hi @davidjlloyd, thanks for the details.

I did a small test and can point out this: you will get the proper result if you use a requirements.in file like this:
requirements.in:

-e .

Having pip parse the requirements.in file, and then process the setup.py, works fine.
The "direct" setup.py parse done by pip-compile is incomplete and would need improvement.

I would generally advise to use requirements files, its the battle-tested flow. Hindsight being 20/20, I might have voted against parsing the setup.py directly in retrospective.

Contributions are welcomed of course 😄

[Hernrup]

We just ran head first into this one. We are not a package but rather have a requirements.in that needs to be resolved into frozen packages. In this case one of our dependencies have a extras_require section. This issue means that one dependency is going to be missing completely in the resulting requirements.txt and fail the entire install as all packages needs to be frozen in --require-hashes mode.

We cant see any workaround other then depending on the package specified in extras_require ourselvs. This is very inconvenient and makes upgrades a lot less flexible.

We would very much like to see a fix for this as the whole purpose of pip-compile seams to be to solve this kind problem.

I might try to get a PR in for it if I get some time on my hands but in the meantime please be advised that this issue has a real and serious impact without workaround.

[abravalheri]

@vphilippon thank you very much for your comments and suggested approach (and for maintaining such a good package)!

I have a little doubt left though: since, as you said, the support to setup.py in incomplete
(for example I am getting a RuntimeError: 'distutils.core.setup()' was never called -- perhaps 'setup.py' is not a Distutils setup script?, when trying to run pip-compile on both example1 and example2) and adding -e . to requirements.in, as you suggested, create an absolute path in requirements.txt (see example3) and therefore prevents sharing it between multiple developers of a same project, what would be your recommendation for being able to use the pip-tools workflow while relying on setuptools for packaging without double bookkeeping the dependencies?

Is there an alternative workaround for this?

[jedie]

Have the same problem. In my case it's https://github.com/stefanfoulis/django-phonenumber-field it used install_requires and extras_require in setup:

    install_requires=["Django>=1.11.3", "babel"],
    extras_require={
        "phonenumbers": ["phonenumbers>=7.0.2"],
        "phonenumberslite": ["phonenumberslite>=7.0.2"],
    },

install_requires is considered but extras_require not:

/tmp$ echo "django-phonenumber-field" > test.in
/tmp$ pip-compile test.in 
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile test.in
#
babel==2.7.0              # via django-phonenumber-field
django-phonenumber-field==3.0.1
django==2.2.2             # via django-phonenumber-field
pytz==2019.1              # via babel, django
sqlparse==0.3.0           # via django

[atugushev]

Hello @jedie! What about this:

$ echo "django-phonenumber-field[phonenumbers]" | pip-compile - -qo-
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file=- -
#
babel==2.7.0              # via django-phonenumber-field
django-phonenumber-field[phonenumbers]==3.0.1
django==2.2.2             # via django-phonenumber-field
phonenumbers==8.10.13     # via django-phonenumber-field
pytz==2019.1              # via babel, django
sqlparse==0.3.0           # via django

[jedie]

Yes. But in my case: django-phonenumber-field is a dependency of a other package. So i didn't have this in my .in file...

[JulienPalard]

What about something like:

pip-compile setup.py --with-extra dev building a requirements-dev.txt file using the extras_require["dev"] section?

[atugushev]

@jedie

Yes. But in my case: django-phonenumber-field is a dependency of a other package. So i didn't have this in my .in file...

That's author's decision to force developers to install extra requirements manually. See changelog and PR https://github.com/stefanfoulis/django-phonenumber-field/pull/236. So in your case you have to put phonenumbers to requirements.in (even if it is an extra dependency of another package you are using). IMO pip-compile has nothing to do with this case.

[barrywhart]

Can someone clarify the expected behavior?

I see above that -e . works. My question is, what about more complex variants such as -e .[dev,quality,test]? It seems they don't work. Is that the expected behavior?

[barrywhart]

Related question: I have a library project where the library's own requirements.in lists -e . as a requirement. In requirements.txt, this became: -e file:///opt/app/src, the absolute local path of the library source. Then when I setup the development environment using requirements.txt, pip freeze shows that it actually installed something like -e [email protected]:barry-hart/my-library.git@ef30e106ce844377b59b82c23943f16d8cbbb5a7#egg=mylib. This is not at all what I wanted -- it's installing remote source, using the last commit of the code. I'm happy to create a separate issue for this if that seems appropriate -- please let me know.

[jaraco]

Hi @barrywhart. I don't see how your question relates at all to "extras_require", which is key to this topic. It sounds like you have another issue.

[atugushev]

I've filed an issue #908 which describes how to resolve a bunch of problems with "extras_require" in setup.py. Please comment if someone is interested in.

[eedwards-sk]

I'm confused how -e . (in a requirements.in) is a valid workaround?

This results in the package being added to the requirements file itself, which is not what happens if you run pip-compile . on a package (setup.py).

[merwok]

I think this is a workflow used by developers of apps that require the app (site, etc) to be itself an installed Python project, for entry points or other integrations (e.g. Pyramid sites need this): developers want to document one command to install project dependencies and the project itself (I guess), so having -e . in requirements.txt can be more convenient than having two install commands.

I am not sure about having -e . in requirements.in though!

[raabf]

Sounds reasonable. But just to note, the “two install commands” can be used in one line e.g.

pip install -r requirements.txt -e .

which works fine and does not require to be -e . in the requirements.in .

I also think, the above line in the documentation, instead putting -e . into the requirements.in . The package itself is not a requirement and the user might want to use it in another way—such as extending PYTHONPATH—although the package might require most times entry points or so. Additionally, -e . does not work with --generate-hashes .
At the moment I manually edit the requirements.txt file after pip-compile and remove the package itself by hand, which is the best thing I could find up until now.

[eedwards-sk]

I also think, the above line in the documentation, instead putting -e . into the requirements.in . The package itself is not a requirement and the user might want to use it in another way—such as extending PYTHONPATH—although the package might require most times entry points or so. Additionally, -e . does not work with --generate-hashes .
At the moment I manually edit the requirements.txt file after pip-compile and remove the package itself by hand, which is the best thing I could find up until now.

Yep. I have a package that should be able to be 'frozen' with hashes, for both its regular dependencies, and foo[dev]. Adding the package itself as a dependency into requirements.txt makes no sense.

[graingert]

I think something like:

pip-compile .[dev]

[atugushev]

pip-tools v6.1.0 is released 🚀 Try out new option pip-compile {setup.py,pyproject.toml,setup.cfg} --extra dev.