Pipenv: Extras don't get converted to requirements correctly

[greysteil]

I'm not sure how to run the latest pipenv git commit (e135f7c) locally (and have terrible internet for the next couple of days), but when I test locking a Pipfile with a path dependencies and extras I get the following error:

Creating a virtualenv for this project…
       Pipfile: /Users/greysteil/code/dependabot-core/tmp/dependabot_20180531-17342-193skc0/Pipfile
       Using /Users/greysteil/.pyenv/versions/3.6.5/bin/python3.6 (3.6.5) to create virtualenv…
       Already using interpreter /Users/greysteil/.pyenv/versions/3.6.5/bin/python3.6
       Using base prefix '/Users/greysteil/.pyenv/versions/3.6.5'
       New python executable in /Users/greysteil/.local/share/virtualenvs/dependabot_20180531-17342-193skc0-bpL9EMBx/bin/python3.6
       Also creating executable in /Users/greysteil/.local/share/virtualenvs/dependabot_20180531-17342-193skc0-bpL9EMBx/bin/python
       Installing setuptools, pip, wheel...done.
       Setting project for dependabot_20180531-17342-193skc0-bpL9EMBx to /Users/greysteil/code/dependabot-core/tmp/dependabot_20180531-17342-193skc0

       Virtualenv location: /Users/greysteil/.local/share/virtualenvs/dependabot_20180531-17342-193skc0-bpL9EMBx
       Locking [dev-packages] dependencies…
       Locking [packages] dependencies…
       MBx/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2949, in __init__
           super(Requirement, self).__init__(requirement_string)
         File "/Users/greysteil/.local/share/virtualenvs/dependabot_20180531-17342-193skc0-bpL9EMBx/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
           requirement_string[e.loc:e.loc + 8]))
       pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'.[socks]'"

       During handling of the above exception, another exception occurred:

       Traceback (most recent call last):
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/resolver.py", line 83, in <module>
           main()
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/resolver.py", line 72, in main
           system=system,
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/resolver.py", line 63, in resolve
           allow_global=system,
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/utils.py", line 401, in resolve_deps
           req_dir=req_dir,
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/utils.py", line 249, in actually_resolve_reps
           req = Requirement.from_line(dep)
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/vendor/requirementslib/requirements.py", line 672, in from_line
           r = NamedRequirement.from_line(stripped_line)
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/vendor/requirementslib/requirements.py", line 246, in from_line
           req = first(requirements.parse(line))
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/vendor/first.py", line 70, in first
           for el in iterable:
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/vendor/requirements/parser.py", line 50, in parse
           yield Requirement.parse(line)
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/vendor/requirements/requirement.py", line 220, in parse
           return cls.parse_line(line)
         File "/Users/greysteil/code/dependabot-core/helpers/python/src/pipenv/pipenv/vendor/requirements/requirement.py", line 198, in parse_line
           pkg_req = Req.parse(line)
         File "/Users/greysteil/.local/share/virtualenvs/dependabot_20180531-17342-193skc0-bpL9EMBx/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2995, in parse
           req, = parse_requirements(s)
         File "/Users/greysteil/.local/share/virtualenvs/dependabot_20180531-17342-193skc0-bpL9EMBx/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2942, in parse_requirements
           yield Requirement(line)
         File "/Users/greysteil/.local/share/virtualenvs/dependabot_20180531-17342-193skc0-bpL9EMBx/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2951, in __init__
           raise RequirementParseError(str(e))
       pkg_resources.RequirementParseError: Invalid requirement, parse error at "'.[socks]'"

To replicate, run pipenv lock with these files:

# Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
pytest = "==3.4.0"

[packages]
requests = ">=2.18.0"
"e1839a8" = {path = ".", editable = true, extras=["socks"]}

# setup.py
from setuptools import setup, find_packages

setup(name='python-package',
      version='0.0',
      description='Example setup.py',
      url='httos://github.com/example/python-package',
      author='Dependabot',
      scripts=[],
      packages=find_packages(),
      setup_requires=[
          'pytest-runner',
      ],
      install_requires=[
          'raven == 5.32.0',
      ],
      tests_require=[
          'pytest==2.9.1',
          'responses==0.5.1',
      ],
      extras_require=dict(
          API=[
              'flask==0.12.2',
          ],
      ),
)

# Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "1fa9cd2757cc385c836a93a7e9480428b5d00d0f9bddbca159cdff0798c73e05"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "contextlib2": {
            "hashes": [
                "sha256:509f9419ee91cdd00ba34443217d5ca51f5a364a404e1dce9e8979cea969ca48",
                "sha256:f5260a6e679d2ff42ec91ec5252f4eeffdcf21053db9113bd0a8e4d953769c00"
            ],
            "version": "==0.5.5"
        },
        "e1839a8": {
            "editable": true,
            "extras": [
                "socks"
            ],
            "path": "."
        },
        "raven": {
            "hashes": [
                "sha256:13e68bbf21d4d6f0cae4458da5be2d0d538733beabc5f93cb185048b28a77457",
                "sha256:a06517e9b7858ac41963f9741cc8358005d37511475aaa4c8b692d29ae0a7d94"
            ],
            "version": "==5.32.0"
        },
        "requests": {
            "hashes": [
                "sha256:1cdbed1f0e236f35ef54e919982c7a338e4fea3786310933d3a7887a04b74d75",
                "sha256:fdb9af60d47ca57a80df0a213336019a34ff6192d8fff361c349f2c8398fe460"
            ],
            "version": "==2.6.0"
        }
    },
    "develop": {
        "attrs": {
            "hashes": [
                "sha256:1c7960ccfd6a005cd9f7ba884e6316b5e430a3f1a6c37c5f87d8b43f83b54ec9",
                "sha256:a17a9573a6f475c99b551c0e0a812707ddda1ec9653bed04c13841404ed6f450"
            ],
            "version": "==17.4.0"
        },
        "pluggy": {
            "hashes": [
                "sha256:7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff"
            ],
            "version": "==0.6.0"
        },
        "py": {
            "hashes": [
                "sha256:29c9fab495d7528e80ba1e343b958684f4ace687327e6f789a94bf3d1915f881",
                "sha256:983f77f3331356039fdd792e9220b7b8ee1aa6bd2b25f567a963ff1de5a64f6a"
            ],
            "version": "==1.5.3"
        },
        "pytest": {
            "hashes": [
                "sha256:6074ea3b9c999bd6d0df5fa9d12dd95ccd23550df2a582f5f5b848331d2e82ca",
                "sha256:95fa025cd6deb5d937e04e368a00552332b58cae23f63b76c8c540ff1733ab6d"
            ],
            "version": "==3.4.0"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        }
    }
}

[techalchemy]

To be clear, this doesn't happen with any path, right? Only when you install .[extras]?

[greysteil]

I think so. Midnight here but I’ll double check tomorrow morning.

[techalchemy]

No rush. We pushed this off into a new library to simplify the logic and the library is part of an org I own with @uranusjr

Debugging should be easier than it used to be at least

[greysteil]

Just checked - it's only a problem when the path dependency has extras.

[greysteil]

Just did a bit more digging on this: it's a problem with any editable path with extras, and the offending line appears to be this one. I'm a bit out of my depth on exactly how to fix, though.

[techalchemy]

@greysteil it was just a problem with parsing before passing stuff around, thanks for the tip!

[techalchemy]

closed by #2302