Pip-tools: Dependencies are droped when using a combination of -r & -c in layered requirements

[mgab]

First of all, thanks a lot for developing and maintaining the package! It is a great tool!

Background

Regarding the issue, when using layered requirements (eg test.txt being a superset of main.txt), two referencing modes are available -r that defines requirements, and -c that defines constraints.

Including both -r main.in and -c main.txt in test.in should ensure that test.txt is a superset of main.txt. It should produce the same results as including just -r main.txt except by the fact that it keeps the comments indicating where indirect dependencies came from.

Issue description

However, using both -r main.in and -c main.txt at the same time drops some dependencies from main.txt that are not propagated to test.txt.

When installing pip install -r test.txt, the missing packages are installed anyway (they were dependencies, indeed). And in fact, the --verbose flag of pip-tools shows that the missing packages are taken into account until the very end of the compilation process but are not included in the output.

On the other hand, using only -r main.in does not miss any package (but does not ensure that the versions are the same).

Environment Versions

1. OS version: docker image python:3.7-slim, but also in other environments.
2. Python version: Python 3.7.6
3. pip version: 19.3.1
4. pip-tools version: pip-compile, version 4.4.0

Steps to replicate

To ensure reproducibility I ran this in a docker, but I was able to reproduce it elsewhere:

docker run -it --rm python:3.7-slim bash

1. Install pip-tools

pip install pip-tools

1. Create two simple layered requirements with

echo pandas > main.in
echo -e "-r main.in\n-c main.txt" > test.in

1. Compile main.in requirements with: -> [captured output]

python -m piptools compile --verbose --allow-unsafe --build-isolation --no-header --no-index main.in --output-file main.txt

1. Compile test.in requirements with: -> [captured output]

python -m piptools compile --verbose --allow-unsafe --build-isolation --no-header --no-index test.in --output-file test.txt

Expected result

Every package listed in main.txt and its version should appear also in test.txt. test.txt could include additional packages if test.in specified additional dependencies.

This is the content of main.txt and the expected content of test.txt:

numpy==1.18.1             # via pandas
pandas==0.25.3
python-dateutil==2.8.1    # via pandas
pytz==2019.3              # via pandas
six==1.14.0               # via python-dateutil

Actual result

Instead, test.txt has less packages than main.txt. The actual content of test.txt is:

numpy==1.18.1             # via pandas
pandas==0.25.3
python-dateutil==2.8.1    # via pandas
pytz==2019.3              # via pandas

This happened with other library combinations, have not figured out the pattern yet.

As mentioned, pip install -r test.txt will also install six, and --verbose shows that pip-tools is considering six until the very end of the compilation process.

[atugushev]

Hello @mgab,

Thanks for the perfectly described issue! It's candy to my eyes ☺️ Regarding the issue, it's not a bug and this is how pip-compile currently works with constraints files. Please see this section and related comment https://github.com/jazzband/pip-tools/pull/1037#issuecomment-578470265.

[jeevb]

@atugushev: In this section, I think it would be very reasonable for users to expect pytz==2019.2 to be added to dev-requirements.txt, via this lineage: django-debug-toolbar==2.0 -> django==2.1.12 -> pytz==2019.2.

This example fails if used with --generate-hashes:

# requirements.in
django<2.2

# requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes requirements.in
#
django==2.1.15 \
    --hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
    --hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c
pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
    # via django



md5-676686554dff239cce2e842ee13bd3a5



# dev-requirements.in
-c requirements.txt
django-debug-toolbar



md5-676686554dff239cce2e842ee13bd3a5



# dev-requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes dev-requirements.in
#
django-debug-toolbar==2.1 \
    --hash=sha256:24c157bc6c0e1648e0a6587511ecb1b007a00a354ce716950bff2de12693e7a8 \
    --hash=sha256:77cfba1d6e91b9bc3d36dc7dc74a9bb80be351948db5f880f2562a0cbf20b6c5
django==2.1.15 \
    --hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
    --hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c \
    # via django-debug-toolbar
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
    # via django-debug-toolbar



md5-9d9c6e14e97f426d3f1d4665ebefdde3



$ pip install -r dev-requirements.txt
Collecting django-debug-toolbar==2.1
  Using cached django_debug_toolbar-2.1-py3-none-any.whl (198 kB)
Collecting django==2.1.15
  Using cached Django-2.1.15-py3-none-any.whl (7.3 MB)
Collecting sqlparse==0.3.0
  Using cached sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Collecting pytz
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    pytz from https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl#sha256=1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d (from django==2.1.15->-r dev-requirements.txt (line 10))

[atugushev]

@jeevb

In this case pip install -r requirements.txt -r dev-requirements.txt must be used. It would be nice to have it mentioned in README too.

[jeevb]

@atugushev

If a constraints file is NOT specified in dev-requirements.in, the resulting dev-requirements.txt works on its own:

# dev-requirements.in
django-debug-toolbar

Notice how the below file is different from the one generated with -c requirements.txt:

# dev-requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes dev-requirements.in
#
django-debug-toolbar==2.1 \
    --hash=sha256:24c157bc6c0e1648e0a6587511ecb1b007a00a354ce716950bff2de12693e7a8 \
    --hash=sha256:77cfba1d6e91b9bc3d36dc7dc74a9bb80be351948db5f880f2562a0cbf20b6c5
django==2.1.15 \
    --hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
    --hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c \
    # via django-debug-toolbar
pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
    # via django
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
    # via django-debug-toolbar

This works:

$ pip install -r dev-requirements.txt
Collecting django-debug-toolbar==2.1
  Using cached django_debug_toolbar-2.1-py3-none-any.whl (198 kB)
Collecting django==2.1.15
  Using cached Django-2.1.15-py3-none-any.whl (7.3 MB)
Collecting pytz==2019.3
  Using cached pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting sqlparse==0.3.0
  Using cached sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Installing collected packages: sqlparse, pytz, django, django-debug-toolbar
Successfully installed django-2.1.15 django-debug-toolbar-2.1 pytz-2019.3 sqlparse-0.3.0

Is it expected that adding -c requirements.txt to dev-requirements.in breaks the install of the dev-requirements.txt file on its own?

[atugushev]

@jeevb

Is it expected that adding -c requirements.txt to dev-requirements.in breaks the install of the individual dev-requirements.txt file?

In this workflow, yes it is. If you use -r requirements.txt, then you'd be able to install dev-requirements.txt file on its own.

[atugushev]

In this case pip install -r requirements.txt -r dev-requirements.txt must be used. It would be nice to have it mentioned in README too.

FTR, addressed in a tracking issue #1043.

[jeevb]

@atugushev Thanks for the clarification!

[atugushev]

@mgab

FYI, this bug should be fixed by #1037.

[mgab]

Thanks @atugushev and @jeevb! And sorry for the late response, I wasn't able to come back to you earlier!

Reading the following discussion in the #1037 pull request I see that you managed to understand what was the problem I was trying to point at despite my silence 😄. Thanks for the perseverance and fast reaction! And yes, it seems to solve the bug!

[atugushev]

pip-tools v4.4.1 is released.