Pip-tools: 5.1.2 regression: lost dependencies using same zip file with different subdirectories

[andersk]

In pip-tools 5.1.2 and later, when I provide two copies of the same zip file URL with different subdirectories, the dependencies of the first are lost:

https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots

git bisect shows that this regression was introduced in commit c0b33e7f2224758524001e755cc338965ca0b2c0 (#1132).

Cc @richafrank

Environment Versions

1. OS Type: Ubuntu 18.04
2. Python version: 3.6.9
3. pip version: 20.1.1
4. pip-tools version: 5.2.0

Steps to replicate

cat > z.in <<EOF
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots
EOF
pip-compile z.in

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile z.in
#
beautifulsoup4==4.9.1     # via zulip-bots
certifi==2020.4.5.1       # via requests
cffi==1.14.0              # via cryptography
chardet==3.0.4            # via requests
cryptography==2.9.2       # via pyopenssl, requests
distro==1.5.0             # via zulip
html2text==2020.1.16      # via zulip-bots
idna==2.9                 # via requests
lxml==4.5.1               # via zulip-bots
matrix-client==0.3.2      # via zulip
pycparser==2.20           # via cffi
pyopenssl==19.1.0         # via requests
requests[security]==2.23.0  # via matrix-client, zulip
six==1.15.0               # via cryptography, pyopenssl, zulip
soupsieve==2.0.1          # via beautifulsoup4
urllib3==1.25.9           # via requests
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots  # via -r z.in
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip  # via -r z.in, zulip-bots

# The following packages are considered to be unsafe in a requirements file:
# pip

Actual result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile z.in
#
beautifulsoup4==4.9.1     # via zulip-bots
html2text==2020.1.16      # via zulip-bots
lxml==4.5.1               # via zulip-bots
soupsieve==2.0.1          # via beautifulsoup4
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots  # via -r z.in
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip  # via -r z.in, zulip-bots

# The following packages are considered to be unsafe in a requirements file:
# pip

[andersk]

I haven’t figured out why, but it works correctly if I delete this optimization:

--- a/piptools/resolver.py
+++ b/piptools/resolver.py
@@ -55,10 +55,6 @@ def combine_install_requirements(repository, ireqs):
     for ireq in ireqs:
         source_ireqs.extend(getattr(ireq, "_source_ireqs", [ireq]))

-    # Optimization. Don't bother with combination logic.
-    if len(source_ireqs) == 1:
-        return source_ireqs[0]
-
     # deepcopy the accumulator so as to not modify the inputs
     combined_ireq = copy.deepcopy(source_ireqs[0])
     repository.copy_ireq_dependencies(source_ireqs[0], combined_ireq)

but only with egg= in the URLs.

[atugushev]

Related to #1154.

[richafrank]

That the optimization breaks it is surprising, but this whole combination step keeps yielding many surprises...

If no one else has figured it out, I'll take a look at this and the related issue as well, hopefully in the next week.

[atugushev]

@richafrank thank you!

[richafrank]

@andersk I was only able to reproduce this issue once there is an existing output file, so I think it's the same as https://github.com/jazzband/pip-tools/issues/1154, but let me know if that's not the case for you, i.e. the first time I compile, I get the expected result.

[andersk]

@richafrank I think you’re correct, and #1159 does seem to solve the problem. Thanks for taking a look!

[atugushev]

pip-tools v5.2.1 is released