Pip-tools: pip-compile to use abstract dependencies declared in setup.py install_requires and extras_require

[gotgenes]

This issue is similar to #282, however, because setup.py is the de-facto way of declaring packages presently in Python packaging, today, could pip-compile support using the install_requires and extras_require defined in setup.py files to determine abstract dependencies? This is based off of the recommendations from Donald Stufft's blog post outlining abstract dependencies in setup.py versus concrete dependencies in requirements.txt.

[ryanhiebert]

My requirements.in just has a reference to ., and then reads my setup.py, so it is possible even now.

[estan]

@ryanhiebert: Could you explain further? Can I just put "." in requirements.in and it'll just work? Sorry for my ignorance :)

I still think this feature has merit, because often you want to maintain a separate test-requirements.txt with additional deps needed only for testing, and those should be generated from tests_require, so some way of specifying what args to setup to consider would be needed. Or maybe that's possible with your approach as well?

[ryanhiebert]

I have a line in my requirements.inthat is just -e ..

[omarkohl]

That's very nice :-) It works for me (at least with 'install_requires' in setup.py).

This ticket could probably the closed.

[jredwards]

Found this thread searching for the same thing. Might be worth mentioning this in the readme.

[majuscule]

This feature would be very useful! Regarding the workaround (-e . in requirements.in), it almost works, but then incorrectly injects a reference to the cwd in the output, i.e.: -e file:///~/src/my/project.

[ryanhiebert]

@majuscule : You're correct. I believe there's another related issue to -e with relative paths that's open but I don't remember which one. My current workaround is this script:

#!/bin/sh

# Ensure that pip-tools is installed and upgraded
pip install --upgrade pip-tools | grep 'Successfully installed pip-tools'

# Remove current requirements.txt, because pip-compile
# doesn't seem to work properly with it present.
rm requirements.txt > /dev/null 2>&1

pip-compile --output-file requirements.txt requirements.in > /dev/null

# pip-compile likes to swap out editable relative paths for
# absolute paths. That's not gonna work on production.
sed -i '' "s|-e file://`pwd`|-e .|" requirements.txt

# pip-compile is adding duplicate lines for --extra-index-url
sed -i '' '$!N; /^\(.*\)\n\1$/!P; D' requirements.txt

# Show the diff
git diff requirements.txt

It does a couple things differently because of things I've had to work around, but it works pretty well for me, apart from regularly switching the case of the initial "D" in my Django requirement.

[majuscule]

Thanks @ryanhiebert, I appreciate you pasting that here. Hopefully I'll have shortened some of that for us if PR https://github.com/jazzband/pip-tools/pull/468 is merged, but the fact that such workarounds are being created should be another testament to the validity of this issue and PR https://github.com/jazzband/pip-tools/pull/418.

[ryanhiebert]

421 / #416 too.

[davidovich]

I think this is safe to close because of #418 being merged.

[jaraco]

I don't believe this issue is fully fixed. The title mentions both install_requires and extras_require, but #418 only addressed install requires. If one wants extras, one still needs to use the hacky workaround of a requirements.in with -e .[extra1,extra2] and still ends up with the local working directory added to requirements.txt. I see #625 was opened to address that issue.