Pip-tools: Fails to recongize Multiple URL filnks (`pip @ git+https://github.com/...`)

[MinchinWeb]

PEP 508 allows version controlled requirements to be listed like this:

pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686

So I'm trying to do this (I am using a couple of private forks that aren't available on PyPI), but only one of them is making it into the pip-compiled requirements file.

My input:

pip @ git+https://github.com/MinchinWeb/django-import-export.git@b406c6f3a247cc587d77f91459571df6cfa536fc#egg=import_export
pip @ git+https://github.com/kysela-petr/django-naturalsortfield.git@e4037372adfe9a109d7fef5bfdb6b1c3a9c4136a#egg=django-naturalsortfield

If I run pip-compile, it will warn me (4 or 5 times):

WARNING:pip._internal.req.req_install:Generating metadata for package pip produced metadata for project name django-import-export. Fix your #egg=pip fragments.

(but I'm not sure what that means.)

and the output file will only contain the django-import-export, but not django-naturalsortfield. Order in the input file doesn't seem to matter, and I can add other "normal" requirements, as found on PyPI, and they all seem to work (i.e. they are added to the output file).

Environment Versions

1. OS Type: Windows
2. Python version: Python 3.8.1
3. pip version: pip 20.0.2
4. pip-tools version: pip-compile, version 4.4.1

Steps to replicate

1. testing.in

pip @ git+https://github.com/MinchinWeb/django-import-export.git@b406c6f3a247cc587d77f91459571df6cfa536fc#egg=import_export
pip @ git+https://github.com/kysela-petr/django-naturalsortfield.git@e4037372adfe9a109d7fef5bfdb6b1c3a9c4136a#egg=django-naturalsortfield

1. pip-compile testing.in

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile '.\.requirements\testing.in'
#
asgiref==3.2.3            # via django
defusedxml==0.6.0         # via odfpy
diff-match-patch==20181111  # via django-import-export
django==3.0.3           # via django-import-export
git+https://github.com/MinchinWeb/django-import-export.git@b406c6f3a247cc587d77f91459571df6cfa536fc#egg=import_export-export
git+https://github.com/kysela-petr/django-naturalsortfield.git@e4037372adfe9a109d7fef5bfdb6b1c3a9c4136a#egg=django-naturalsortfield
et-xmlfile==1.0.1         # via openpyxl
jdcal==1.4.1              # via openpyxl
markuppy==1.14            # via tablib
numpy==1.18.1             # via pandas
odfpy==1.4.1              # via tablib
openpyxl==3.0.3           # via tablib
pandas==1.0.1             # via tablib
python-dateutil==2.8.1    # via pandas
pytz==2019.3              # via django, pandas
pyyaml==5.3               # via tablib
six==1.14.0               # via python-dateutil
sqlparse==0.3.0           # via django
tablib[all]==1.0.0        # via django-import-export
tabulate==0.8.6           # via tablib
xlrd==1.2.0               # via tablib
xlwt==1.3.0               # via tablib

Actual result

Command line errors:

WARNING:pip._internal.req.req_install:Generating metadata for package pip produced metadata for project name django-import-export. Fix your #egg=pip fragments.
WARNING:pip._internal.req.req_install:Generating metadata for package pip produced metadata for project name django-import-export. Fix your #egg=pip fragments.
WARNING:pip._internal.req.req_install:Generating metadata for package pip produced metadata for project name django-import-export. Fix your #egg=pip fragments.
WARNING:pip._internal.req.req_install:Generating metadata for package pip produced metadata for project name django-import-export. Fix your #egg=pip fragments.
WARNING:pip._internal.req.req_install:Generating metadata for package pip produced metadata for project name django-import-export. Fix your #egg=pip fragments.

Output file:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile '.\.requirements\testing.in'
#
asgiref==3.2.3            # via django
defusedxml==0.6.0         # via odfpy
diff-match-patch==20181111
git+https://github.com/MinchinWeb/django-import-export.git@b406c6f3a247cc587d77f91459571df6cfa536fc#egg=import_export   django==3.0.3
et-xmlfile==1.0.1         # via openpyxl
jdcal==1.4.1              # via openpyxl
markuppy==1.14            # via tablib
numpy==1.18.1             # via pandas
odfpy==1.4.1              # via tablib
openpyxl==3.0.3           # via tablib
pandas==1.0.1             # via tablib
python-dateutil==2.8.1    # via pandas
pytz==2019.3              # via django, pandas
pyyaml==5.3               # via tablib
six==1.14.0               # via python-dateutil
sqlparse==0.3.0           # via django
tablib[all]==1.0.0
tabulate==0.8.6           # via tablib
xlrd==1.2.0               # via tablib
xlwt==1.3.0               # via tablib

[atugushev]

Hello @MinchinWeb,

Thanks for the detailed issue! I believe the issue is in the name @ url pattern. It should be project_name @ vcs_url, no eggs param needed in this case by the way, but the project_name should be the same as package name in vcs_url/setup.py. Try the requirements.in below for the instance which compiles without any warnings:

$ cat requirements.in
django-import-export @ git+https://github.com/MinchinWeb/django-import-export.git@b406c6f3a247cc587d77f91459571df6cfa536fc
django-naturalsortfield @ git+https://github.com/kysela-petr/django-naturalsortfield.git@e4037372adfe9a109d7fef5bfdb6b1c3a9c4136a

$ pip-compile --no-header
asgiref==3.2.3            # via django
defusedxml==0.6.0         # via odfpy
diff-match-patch==20181111
git+https://github.com/MinchinWeb/django-import-export.git@b406c6f3a247cc587d77f91459571df6cfa536fc
git+https://github.com/kysela-petr/django-naturalsortfield.git@e4037372adfe9a109d7fef5bfdb6b1c3a9c4136a
django==3.0.3
et-xmlfile==1.0.1         # via openpyxl
jdcal==1.4.1              # via openpyxl
markuppy==1.14            # via tablib
numpy==1.18.1             # via pandas
odfpy==1.4.1              # via tablib
openpyxl==3.0.3           # via tablib
pandas==1.0.1             # via tablib
python-dateutil==2.8.1    # via pandas
pytz==2019.3              # via django, pandas
pyyaml==5.3               # via tablib
six==1.14.0               # via python-dateutil
sqlparse==0.3.0           # via django
tablib[all]==1.0.0
tabulate==0.8.6           # via tablib
xlrd==1.2.0               # via tablib
xlwt==1.3.0               # via tablib

Hope this helps!

[MinchinWeb]

Thank you @atugushev ! With your suggestion, it works on my end. I had spent hours trying to figure this out, and your 4 lines are the best explanation I've found of this anywhere. I see my error was prefacing both with "pip @". This was based on the sample line in PEP508, but there, the example is for pip, and I assumed it meant install with pip.

A couple questions:

1. Where can I add this to the documentation? Or is there a better way to specify dependencies from GitHub?
2. The output doesn't have the "# via..." notation for an packages that depend on GitHub-hosted packages. Is this intentional? Should I raise a separate bug report for this?
3. Sorting of packaged is kind of weird, as "=" is after "-". So all the django packages get listed before django itself. If you can point me towards the sorting code, I could probably make the Pull Request myself...

[atugushev]

@MinchinWeb

1. Where can I add this to the documentation?

I think it's better to raise an issue on pep's tracker to improve PEP-508 documentation. Also, you may found useful a requirement specifiers section in pip documentation.

Or is there a better way to specify dependencies from GitHub?

There are a few ways:

1. editable: -e git+https://github.com/benjaminp/six.git@3a3db7#egg=six
2. VCS URL: git+https://github.com/benjaminp/six.git@3a3db7
3. archive: https://github.com/benjaminp/six/archive/3a3db7.zip#egg=six

1. The output doesn't have the "# via..." notation for an packages that depend on GitHub-hosted packages. Is this intentional? Should I raise a separate bug report for this?

This is a known bug, here is the tracking issue — #293.

1. Sorting of packaged is kind of weird, as "=" is after "-". So all the django packages get listed before django itself. If you can point me towards the sorting code, I could probably make the Pull Request myself...

Yes, this seems odd. It would be nice to have fixed it. Could you raise a separate issue? I'm pretty sure it's related to these lines: https://github.com/jazzband/pip-tools/blob/d405bf9c35e9fd37439e33b1f93a708d299bf329/piptools/writer.py#L76-L77

[MinchinWeb]

These are great suggestions, thanks you! I'm going to be away from my computer for the next three weeks, but I will follow up when I return!

[electrofelix]

Unfortunately it appears this doesn't work so well for requirements.txt that are read into setuptools which still requires the format <package> @ <vcs_url>.

When trying to just use the syntax referred to above I get the following error:

error in test-repo setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+ssh://g'"

We can work around it by having the file that prepends the package name to the vcs_url, and then pass the modified list as an argument to setuptools.

However as this shows other tools maybe expecting <name> @ <vcs_url> and don't all perform parsing to detect if it's just <vcs_url> by itself unless prefixed with the <name> @ would it be better if pip-compile retained the original input line in such a case rather than transforming it?

[atugushev]

@MinchinWeb

1. The output doesn't have the "# via..." notation for an packages that depend on GitHub-hosted packages. Is this intentional? Should I raise a separate bug report for this?

This is a known bug, here is the tracking issue — #293.

FYI, it was fixed in pip-tools-4.5.0.

---

@electrofelix

Unfortunately it appears this doesn't work so well for requirements.txt that are read into setuptools which still requires the format <package> @ <vcs_url>.

Yes, this is not well supported yet. #1165 - is a tracking issue for this feature.

[atugushev]

I'll close this in favor of following-up #1165. Thanks for the issue!