Pipenv: Pipenv ignores custom index on the command line... or does it?

Created on 12 Aug 2019 · 5Comments · Source: pypa/pipenv

I'm fairly new to this, so I'm not sure if I'm using pipenv right. I encountered several issues with pipenv.

I'm trying to install a package (that i've just made myself) from test.pypi.org. Basically, I want to run pip install my-package -i https://test.pypi.org/simple, which works fine, but in pipenv.

Problem 0: this doesn't seem to be possible...ish:

% pipenv install my-package -i https://test.pypi.org/simple
Creating a virtualenv for this project...
Pipfile: /home/oakkitten/test7/Pipfile
Using /usr/bin/python (2.7.13) to create virtualenv...
⠋ Creating virtual environment...Already using interpreter /usr/bin/python
New python executable in /home/oakkitten/.local/share/virtualenvs/test7-wnMdLOpT/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment! 
Virtualenv location: /home/oakkitten/.local/share/virtualenvs/test7-wnMdLOpT
Creating a Pipfile for this project...
Installing my-package...
⠸ Installing...
Adding my-package to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
✘ Locking Failed! 
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 69, in resolve
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       ResolutionFailure: ERROR: ERROR: Could not find a version that matches my-package
[pipenv.exceptions.ResolutionFailure]:       No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches my-package
No versions found
Was https://pypi.org/simple reachable?
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       ResolutionFailure: ERROR: ERROR: Could not find a version that matches my-package
[pipenv.exceptions.ResolutionFailure]:       No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches my-package
No versions found
Was https://pypi.org/simple reachable?

Problem 1: Judging by the last line, it looks like pipenv only checked out https://pypi.org/simple and didn't look at https://test.pypi.org/, right? Except python -c "import mymodule" works fine, and it is listed by pip freeze.

Looking at what pipenv launches with --verbose when i run it without -i, I see

pip install --verbose --upgrade my-package -i https://pypi.org/simple

But running it with -i https://test.pypi.org/simple results in an extra source added (even with no Pipfile):

pip install --verbose --upgrade my-package -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple

And running it with -i https://f.org --extra-index-url https://b.org --verbose results in no extra url added (although the documentation says there can be several?):

pip install --verbose --upgrade my-package -i https://f.org --extra-index-url https://b.org

During all of the mentioned calls, pipenv says:

Using pip: -i https://pypi.org/simple

This must be a mistake, right?

Problem 2: even though pipenv returns an error, it puts my-package = {index = "https://test.pypi.org/simple",version = "*"} into Pipfile. Likely because of that, all subsequent operations fail as well (such as pipenv clean). Related issue: #3808

I tried editing the Pipfile as follows:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "test"
url = "https://test.pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]

[requires]
python_version = "2.7"

Running pipenv install my-package -i test then worked without issues, with my-package = {index = "test",version = "*"}. Obviously, due to problem 1, pipenv install my-package -i pypi worked just as well, as did pipenv install my-package -ipypi (note the absence of the space). Manually adding my-package = {index = "wtf",version = "*"} and syncing worked as well. So, the

Problem 3 is that in practice index in Pipfile has no effect. This seems to be already covered in #2730, #2159 and other issues; just mentioning here for the sake of completeness.

So what are -i an --extra-index-url options, anyway? According to the documentation, these should be url(s):

--extra-index-url <extra_index_url>

    URLs to the extra PyPI compatible indexes to query for package lookups.

-i, --index <index>

    Target PyPI-compatible package index url.

But the latter makes it to the index = "test" parameter, which is not an url?..

All of this doesn't make any sense to me, please advise.

Also, I'd like to mention these regarding the first error message:

Problem 4: the error message is duplicated (for some reason line with “69” is not). Also note the "ERROR: ERROR:" thing.

Problem 5: the error message is cryptic, including traceback for no apparent reason. it's even worse when i tried to do pipenv install lxml, while the latter had an mismatching glibc version; the output looked like this:

An error occurred while installing lxml==4.4.0 --hash=sha256:06e5599b9c54f797a3c0f384c67705a0d621031007aa2400a6c7d17300fdb995 --ha
(many lines omitted)
7fdd9f39892ba05 --hash=sha256:f840dddded8b046edc774c88ed8d2442cdb231a68894c42c74e3a809450fae76! Will try again.
Installing initially failed dependencies...
(many lines omitted)
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting lxml==4.4.0 (from -r /tmp/pipenv-4i3Q8I-requirements/pipenv-jmkuYm-requirement.txt
(line 1))', '  Using cached https://files.pythonhosted.org/packages/e1/f5/5eb3b491958dcfdcfa5daae3c655ab59276bc216ca015e44743c9c22
0e9e/lxml-4.4.0.tar.gz', 'Building wheels for collected packages: lxml', '  Building wheel for lxml (setup.py): started', "  Build
ing wheel for lxml (setup.py): finished with status 'error'", '  Running setup.py clean for lxml', 'Failed to build lxml', 'Insta$
ling collected packages: lxml', '  Running setup.py install for lxml: started', "    Running setup.py install for lxml: finished w
ith status 'error'"]
(many lines omitted)
h:24:0,', '                     from src/lxml/etree.c:723:', '    /usr/include/libxslt/xsltlocale.h:20:21: fatal error: xlocale.h:
 No such file or directory', '     #include <xlocale.h>', '                         ^', '    compilation terminated.', "    Compil
e failed: command 'gcc' failed with exit status 1", '    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInituub0
Dh.c -o tmp/xmlXPathInituub0Dh.o', '    /tmp/xmlXPathInituub0Dh.c:2:1: warning: return type defaults to \xe2\x80\x98int\xe2\x80\x9
9 [-Wimplicit-int]', '     main (int argc, char **argv) {', '     ^~~~', '    cc tmp/xmlXPathInituub0Dh.o -lxml2 -o a.out', "    e
rror: command 'gcc' failed with exit status 1", '    ----------------------------------------', 'ERROR: Command errored out with e
xit status 1: /home/bot/.local/share/virtualenvs/boot2-JoRJ2-zy/bin/python -u -c \'import sys, setuptools, tokenize; sys.argv[0] =
 \'"\'"\'/tmp/pip-install-_LZNij/lxml/setup.py\'"\'"\'; __file__=\'"\'"\'/tmp/pip-install-_LZNij/lxml/setup.py\'"\'"\';f=getattr(t
okenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(c
ompile(code, __file__, \'"\'"\'exec\'"\'"\'))\' install --record /tmp/pip-record-XRL7Jv/install-record.txt --single-version-extern
ally-managed --compile --install-headers /home/bot/.local/share/virtualenvs/boot2-JoRJ2-zy/include/site/python2.7/lxml Check the l
ogs for full command output.']

As you can see, the compiler output is quite impossible to read.

Not including the output of $ pipenv --support as I tested this on several devices with clean installations of everything

Possible Bug

Source