Pipenv: extra-index-url URL in generated requirements.txt syntax error

[wichert]

This bug seems to be introduced between pipenv 9.0.3 and 11.8.3. I have a Pipfile with two sources:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[[source]]

url = "https://pypi.fury.io/TOKEN/jzoo/"
verify_ssl = true
name = "jzoo"


[packages]

"e1839a8" = {path = ".", editable = true}

When installing from this using a basic pipenv install --dev I get the following:

…
Installing 'jsonschema==2.6.0 --hash=sha256:000e68abd33c972a5248544925a0cae7d1125f9bf6c58280d37546b946769a08  --hash=sha256:6ff5f3180870836cae40f06fa10419f557208175f13ad7bc26caa77beb1f6e02   --extra-index-url https://pypi.fury.io/TOKEN/jzoo/'
$ "/Users/wichert/.local/share/virtualenvs/whitelabel.api-aXG16dXx/bin/pip" install   --verbose  --no-deps  -r /var/folders/mq/cq4zpf2n6qz_8x4jqsmmv6b80000gn/T/pipenv-t2i8sda7-requirements/pipenv-zygz4qgt-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
Installing 'jzoo.utils==1.17.0  --extra-index-url https://pypi.fury.io/TOKEN/jzoo/'
WARNING: Invalid requirement, parse error at "'--extra-'"
ABORTING INSTALL... You will have to reinstall any packages that failed to install.

[wichert]

Trying to narrow this down:

• 9.0.3: works
• 9.1.0: works
• 10.0.0: works
• 11.0.0: works
• 11.5.*: failed to start
• 11.6.0: works
• 11.6.1: works
• 11.6.2: fails
• 11.6.4: fails
• 11.6.9: fails
• 11.7.0: fails
• 11.8.3: fails

So something in the 11.6.2 release introduced this particular error.

[techalchemy]

@wichert thanks for bringing this to our attention! Was the requirements.txt generated with the same version of pipenv you are testing with? If not, can you regenerate it and will it work?

Or, are you saying that any pipfile with two sources is now failing when you try to pass --dev?

If you dont mind, we would need to see as much of your pipfile and lockfile as you are willing to share or an equivalent reproducible test case or it will be hard to untangle

[wichert]

I removed all generated files between pipenv runs. The requirements.txt was always the one generated by pipenv to pass to pip install.

I think the —dev argument does not matter, that just happened to be the command CI was running.

[wichert]

The complete Pipfile looks like this:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[[source]]

url = "https://pypi.fury.io/TOKEN/jzoo/"
verify_ssl = true
name = "jzoo"


[packages]

"e1839a8" = {path = ".", editable = true}


[dev-packages]

"e1839a8" = {path = ".", extras = ["tests"], editable = true}
"flake8" = "*"
lingua = "*"


[requires]

python_version = "3.6"

Pipfile.lock looks like this:

{
    "_meta": {
        "hash": {
            "sha256": "397f9f2805a6b8b3d7d4709acfabae6bace647bec98a2182221ac519c42c4b18"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.4",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "17.4.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64",
            "python_full_version": "3.6.4",
            "python_version": "3.6",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            },
            {
                "name": "jzoo",
                "url": "https://pypi.fury.io/TOKEN/jzoo/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
       …
    }
}

[techalchemy]

@wichert you shouldn't have the same entry appearing twice in your pipfile, the one in dev-packages will never be resolved. I can't reproduce anything from what you've provided though, can you supply the full outputs without truncating them? It's really hard to troubleshoot without seeing what pipenv is actually producing

[wichert]

@techalchemy The two entries are very different: the dev entry includes the test extra. That extra adds a bunch of stuff that the tests need, but should not be install in production.

I have created a gist with the full output. The error occurs when it hits the first package that is not available on pypi but only on our private index.

[wichert]

1723 looks very similar to this one. I'll check if regenerating the lockfile works.

[techalchemy]

I get that they are different. They have the same name, which means the non-dev one always has priority. Not making things up here.

[cristynkells]

removing the --skip-lock option eliminated the below issue for me. But now I need to remove the lock file in between builds...

WARNING: Invalid requirement, parse error at "'--extra-'"
ABORTING INSTALL... You will have to reinstall any packages that failed to install.

[wichert]

Regenerating the lockfile indeed fixes this. I have no idea why that helps - are there any backwards compatibility guarantees for lockfiles? I couldn't find any things in the documentation.

[techalchemy]

@wichert as an open source project run by volunteers we make no guarantees that we don't sometimes release bugs. Seems likely that we did that somewhere along the line. That's why lockfiles are reproducible.