tox 4: pip requirements parsing doesn't support --hash arguments

[adamchainz]

For my open source project tests I use pip-compile's --generate-hashes to pass the --hash argument to pip (example). This works in tox 3 but with tox 4 it fails where tox tries to parse the line:

$ python3.9 -m tox -e py39
py39: internal error
Traceback (most recent call last):
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/packaging/requirements.py", line 98, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/pyparsing.py", line 1955, in parseString
    raise exc
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/pyparsing.py", line 3814, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected stringEnd, found '-'  (at char 18), (line:1, col:19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/tox_env/python/req_file.py", line 108, in validate_and_expand
    req = Requirement(line)
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/packaging/requirements.py", line 100, in __init__
    raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'--hash=s'": Expected stringEnd

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/session/cmd/run/single.py", line 41, in _evaluate
    tox_env.ensure_setup(recreate=recreate)
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/tox_env/api.py", line 169, in ensure_setup
    self.setup()
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/tox_env/python/runner.py", line 44, in setup
    self.install_deps()
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/tox_env/python/runner.py", line 68, in install_deps
    requirement_file_content = requirements_file.validate_and_expand()
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/tox_env/python/req_file.py", line 101, in validate_and_expand
    result.extend(req_file.validate_and_expand())
  File "/.../.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/tox_env/python/req_file.py", line 120, in validate_and_expand
    raise ValueError(f"{at}: {line}") from exc
ValueError: 7: attrs==20.3.0     --hash=sha256:31b2eced602aa8423c2aea9c76a724617ed67cf9513173fd3a4f03e3a929c7e6     --hash=sha256:832aa3cde19744e49938b91fea06d69ecb9e649c93ba974535d08ad92164f700
py39: FAIL ✖ in 0.06 seconds
  py39: FAIL code 2 (0.06 seconds)
  evaluation failed :( (0.11 seconds)

[gaborbernat]

Can you please post a link to a reproducible repository? Or full requirements.txt example?

[adamchainz]

Linked in the above post in the word "example" (requirements.txt in reproducible repo)

[adamchainz]

I tried again with 4.0.0a6 and now the error shows the --hash arguments not being split off from the package name, e.g.:

ERROR: Invalid requirement: 'attrs==20.3.0 --hash=sha256:31b2eced602aa8423c2aea9c76a724617ed67cf9513173fd3a4f03e3a929c7e6 --hash=sha256:832aa3cde19744e49938b91fea06d69ecb9e649c93ba974535d08ad92164f700'

[gaborbernat]

Hmm, must be a bug, it shouldn't.

[pradyunsg]

https://pip--9474.org.readthedocs.build/en/9474/reference/requirements-files/#supported-options seems relevant here.

[r2dan]

Hi, I tried the tox4 in of of my projects and found another issue with the same possible root cause - just adding a minimal failing example here for reference:

$ tox4
py: install_deps> python -I -m pip install '--index-url https://m.devpi.net/fschulze/dev' devpi-client==6.0.0dev3

Usage:
  /Users/mrusi/.tox/4/py/bin/python -m pip install [options] <requirement specifier> [package-index-options] ...
  /Users/mrusi/.tox/4/py/bin/python -m pip install [options] -r <requirements file> [package-index-options] ...
  /Users/mrusi/.tox/4/py/bin/python -m pip install [options] [-e] <vcs project url> ...
  /Users/mrusi/.tox/4/py/bin/python -m pip install [options] [-e] <local project path> ...
  /Users/mrusi/.tox/4/py/bin/python -m pip install [options] <archive url/path> ...

no such option: --index-url https://m.devpi.net/fschulze/dev
py: exit 2 (1.12 seconds) /Users/mrusi> python -I -m pip install '--index-url https://m.devpi.net/fschulze/dev' devpi-client==6.0.0dev3 pid=85989
  py: FAIL code 2 (1.22 seconds)
  evaluation failed :( (1.32 seconds)

$ cat tox.ini
[testenv]
deps=-rrequirements.txt
commands=echo "commands"

$ cat requirements.txt
--index-url https://m.devpi.net/fschulze/dev
devpi-client==6.0.0dev3

[jugmac00]

@r2dan Could you please create a new issue? This seems pretty unrelated, as in your case it is about --index-url and not --hash.

Also, there should be already some support for --index-url in tox4, considering when I look at req_file.py and its associated test suite.

When you create a new issue, please also make sure to follow the instructions, ie show us the verbose output by adding -vv.

[r2dan]

Sorry - I thought you're about to include the support for requirements.txt file support with this ticket as suggested by @pradyunsg:

https://pip--9474.org.readthedocs.build/en/9474/reference/requirements-files/#supported-options seems relevant here.

Creating a new one, thanks!

[jugmac00]

This has been fixed on the current rewrite branch. Thanks @gaborbernat!

❯ tox4dev -e py39 -rvv
...
========================================================= test session starts =========================================================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
Using --randomly-seed=2295313076
rootdir: /tmp/apig-wsgi
plugins: randomly-3.5.0
collected 73 items

tests/test_apig_wsgi.py .........................................................................                               [100%]

========================================================= 73 passed in 0.23s ==========================================================
ROOT: 6940 I exit 0 (0.50 seconds) /tmp/apig-wsgi> python -W error::DeprecationWarning -W error::PendingDeprecationWarning -m coverage run --parallel -m pytest pid=19860 [tox/execute/api.py:212]
ROOT: 6941 W _exit> python /home/jugmac00/Projects/tox/src/tox/util/pep517/backend.py True setuptools.build_meta [tox/tox_env/api.py:349]
Backend: run command _exit with args {}
Backend: Wrote response {'return': 0} to /tmp/pep517__exit-im6gciui.json
ROOT: 6943 I exit None (0.00 seconds) /tmp/apig-wsgi> python /home/jugmac00/Projects/tox/src/tox/util/pep517/backend.py True setuptools.build_meta pid=19785 [tox/execute/api.py:212]
  py39: OK (6.84=setup[6.33]+cmd[0.50] seconds)
  congratulations :) (6.87 seconds)

[adamchainz]

👍 Thanks!

[jugmac00]

@adamchainz Could you please close the issue? I have no triage permission. Thank you!

[gaborbernat]

@adamchainz Could you please close the issue? I have no triage permission. Thank you!

In light of recent contributions and involvement invited you to be part of the maintainers group, which should give you all the rights you need 👌

[adamchainz]

I'll leave this open so you can test your permission @jugmac00

[jugmac00]

@adamchainz Could you please close the issue? I have no triage permission. Thank you!

In light of recent contributions and involvement invited you to be part of the maintainers group, which should give you all the rights you need

Holy moly! This makes me super happy and super proud!

[gaborbernat]

@adamchainz Could you please close the issue? I have no triage permission. Thank you!

In light of recent contributions and involvement invited you to be part of the maintainers group, which should give you all the rights you need

Holy moly! This makes me super happy and super proud!

You deserve it!😊 Use your great new powers responsibly 😎