Pip-tools: Sequential pip-compile calls leads to missed dependecies

[pohmelie]

We have subnormal requirements.in:

file:./krontech-core
file:./krontech-web

aiohttp >= 3.0.0, < 4.0.0
alembic >= 0.9.2
asyncpg >= 0.20.1, < 1.0.0
geojson >= 2.5.0
h5py >= 2.0.0
injector >= 0.13.0, < 0.16.0
jpeg4py >= 0.1.3
pillow >= 6.2.0, < 7.0.0
pypika >= 0.37.1, < 1.0.0
python-docx >= 0.8.10
pytz >= 2019.3
sqlalchemy[postgresql_psycopg2binary] >= 1.3.13, < 2.0.0
yarl >= 1.1.0

So, first two lines links to submodules with simple setup.py package description.

Environment Versions

1. Ubuntu 20.04
2. Python version: 3.8.3
3. pip version: 20.1.1
4. pip-tools version: 5.2.0

Steps to replicate

1. pip-compile requirements.in -o requirements.txt
2. Copy requirements.txt somewhere
3. pip-compile requirements.in -o requirements.txt
4. Compare copied and generated versions.

Expected result

They are the same

Actual result

Packages from submodules krontech-core and krontech-web are missed.

[atugushev]

Hello @pohmelie,

Thanks for the report. Couldn't reproduce the bug. See details below:

$ cat foo/setup.py
from setuptools import setup
setup(name='foo', version='0.1', install_requires=['pytz'])

$ cat bar/setup.py
from setuptools import setup
setup(name='bar', version='0.1', install_requires=['six'])

$ cat requirements.in
file:./foo
click

$ pip-compile --no-header
file:./bar                # via -r requirements.in
click==7.1.2              # via -r requirements.in
file:./foo                # via -r requirements.in
pytz==2020.1              # via foo
six==1.15.0               # via bar

$ cp requirements.txt requirements.txt.orig

$ pip-compile --no-header
file:./bar                # via -r requirements.in
click==7.1.2              # via -r requirements.in
file:./foo                # via -r requirements.in
pytz==2020.1              # via foo
six==1.15.0               # via bar

$ sha1sum requirements.txt*
ac59f831da785a68612372a69d633742eb356085  requirements.txt
ac59f831da785a68612372a69d633742eb356085  requirements.txt.orig

[pohmelie]

Maybe, it depends that origin setup.py have local reqs:

krontech-core @ file://localhost/path/to/package/krontech-core
krontech-web @ file://localhost/path/to/package/krontech-web

Or is it better to create github repo to repoduce this thing?

[atugushev]

@pohmelie

Or is it better to create github repo to repoduce this thing?

That'd be super helpful!

[atugushev]

@pohmelie

Also, could you reproduce the bug on pip-tools==5.1.0?

[pohmelie]

Ok, here is our exact case skeleton: https://github.com/pohmelie/pip-tools-1154

[pohmelie]

5.1.0 gives:

$ pip-compile requirements/production.in -o requirements/production.txt
Could not find a version that matches krontech-core>=3.0.0 (from krontech-web==0.13.2->-r requirements/production.in (line 2))
No versions found
Was https://pypi.org/simple reachable?

[Dandi91]

Looking at the changes in production.txt closely, it appears that only dependencies from krontech-core are missing. Packages from krontech-web are still present.

Maybe the 5.1.0 output is a hint to what is happening: pip-compile can't find krontech-core, but 5.2.0 is silencing this error somehow?

[atugushev]

Ok, here is our exact case skeleton: https://github.com/pohmelie/pip-tools-1154

Thanks a lot, that's perfect! Will look into that.

[atugushev]

Looking at the changes in production.txt closely, it appears that only dependencies from krontech-core are missing. Packages from krontech-web are still present.

Maybe the 5.1.0 output is a hint to what is happening: pip-compile can't find krontech-core, but 5.2.0 is silencing this error somehow?

Looks like it is a regression that was introduced in c0b33e7 (#1132). There is also a related issue #1155.

/cc @richafrank

[atugushev]

Could not find a version that matches krontech-core>=3.0.0 (from krontech-web==0.13.2->-r requirements/production.in (line 2))
No versions found
Was https://pypi.org/simple reachable?

So this is the real bug we have to figure out.

[atugushev]

@pohmelie

The workaround is to add an egg fragment to krontech-core in production.in:

...
file:./krontech-core#egg=krontech-core
...

UPDATE: and temporarily downgrade to pip-tools-5.1.0

[pohmelie]

Okay, this is not critical for us, so watching this issue :eyes:

[atugushev]

pip-tools v5.2.1 is released