Pipenv: Public dependencies are not resolved properly on private packages

[gzzo]

When a public package is listed as a dependency in a private package, the package will fail to install because it doesn't exist in the private index.

For example, if you specify an additional source in your Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[[source]]

url = "http://localhost:8080/"
name = "local"

This is the following output from running pipenv install --verbose local_package

Installing local_package…
⠋Installing u'local_package'
$ "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/bin/pip" install   --verbose    "local_package" -i https://pypi.python.org/simple --exists-action w
⠏$ "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/bin/pip" install   --verbose    "local_package" -i http://localhost:8080/ --exists-action w
Collecting local_package
  1 location(s) to search for versions of local-package:
  * http://localhost:8080/local-package/
  Getting page http://localhost:8080/local-package/
  Starting new HTTP connection (1): localhost
  "GET /local-package/ HTTP/1.1" 200 None
  Analyzing links from page http://localhost:8080/local-package/
    Found link http://localhost:8080/local-package/local_package-0.1.1-py3-none-any.whl#md5=ce978ae83f1fec0526e7f34c45d27810 (from http://localhost:8080/local-package/), version: 0.1.1
  Using version 0.1.1 (newest of versions: 0.1.1)
  "GET /local-package/local_package-0.1.1-py3-none-any.whl HTTP/1.1" 200 7273
  Downloading http://localhost:8080/local-package/local_package-0.1.1-py3-none-any.whl
  Downloading from URL http://localhost:8080/local-package/local_package-0.1.1-py3-none-any.whl#md5=ce978ae83f1fec0526e7f34c45d27810 (from http://localhost:8080/local-package/)
Collecting boto3 (from local_package)
  1 location(s) to search for versions of boto3:
  * http://localhost:8080/boto3/
  Getting page http://localhost:8080/boto3/
  "GET /boto3/ HTTP/1.1" 403 None
  Could not fetch URL http://localhost:8080/boto3/: 403 Client Error: Forbidden for url: http://localhost:8080/boto3/ - skipping
Cleaning up...
Exception information:
Traceback (most recent call last):
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
pip.exceptions.DistributionNotFound: No matching distribution found for boto3 (from local_package)

Error:  An error occurred while installing local_package!
  Could not find a version that satisfies the requirement boto3 (from local_package) (from versions: )
No matching distribution found for boto3 (from local_package)

It seems that the issue is with pipenv using -i instead of --extra-index-url

If I run pip install --extra-index-url http://localhost:8080 local_package everything is installed correctly.

Specifying the index for local_package under [packages] doesn't help.

Describe your environment

1. OS Type: macOS Sierra
2. Python version: Python 3.6.3
3. Pipenv version: pipenv, version 10.1.2

Expected result

I would expect for the public package (in the case above, boto3), to be fetched from the public source by default.

Actual result

The public package is fetched from the private source.

Steps to replicate

Set up a private pypi index, add source to Pipfile, and run install command from above.

[festinuz]

I have faced a similar problem (#1291) and am facing that problem again at this very moment.
I guess you can help by specifying the following:

• Python version you used to install pipenv
• Python version used in pipenv
• Pipenv version
• Your OS

Maybe one of the issues should be closed, but i will leave that to the actual contributors to decide.

[wichert]

I ran into this as well. Version information is:

• pipenv installed using Python 3.6, from homebrew
• pipenv version 11.0.2
• venv using Python 3.6
• macOS high sierra

[gzzo]

@festinuz Thanks, I edited my description to include my environment.

[techalchemy]

I think the default behavior was supposed to be to try each index and fail silently before moving on. Not sure but this seems like a regression

[wichert]

@kennethreitz I don't see a commit that references this ticket, so I'm wondering if something changed to close this?

[kennethreitz42]

cleaning house, it'll come back up if it's an issue.

[wichert]

It certainly was still an issue for me 2 days ago.

[kennethreitz42]

Cannot reproduce with this Pipfile:

[[source]]

url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "pypi"


[packages]

requests = "*"


[dev-packages]



[requires]

python_version = "3.6"

[gzzo]

@kennethreitz this was incorrectly closed.

Your test is inadequate because the dependencies of requests exist in testpypi.

The issue, as is stated in the description, is "When a public package is listed as a dependency in a private package, the package will fail to install because it doesn't exist in the private index."

[wichert]

@kennethreitz The problems happens when you have multiple sources in the Pipfile, and you are installing a mix of packages each of which only exists in one of those sources. Most commonly you have a private source, which has packages that depend on packages on a public source (generally pypi).

[wichert]

@kennethreitz Will you reopen this ticket, or should we file a new one?

[jtratner]

good first step to resolution is a failing test case :)

[gzzo]

@jratner I described how to reproduce this in the description, but if it helps, I've broken it down for you:

1. Create a new Python package, give it at least one public dependency (like boto3).
2. Create your own pypi index. This can be as simple as creating the correct folder structure and running python -mSimpleHTTPServer in that directory.
3. Build your package and place it in your index.
4. Create a new folder elsewhere, and initialize pipenv. Add your local index to the sources section.
5. Try to install that package with pipenv. It will fail because boto3 is not in your local index.

[jtratner]

@jratner I described how to reproduce this in the description, but if it helps, I've broken it down for you:

gotcha - thanks for the instructions. what I was suggesting was posting a gist that sets all of this up correctly and reproduces the problem (i.e., a script that sets up a pypi index, uploads a file to it (or just stores the file in the folder as you said) and then attempts to pipenv install from it, reproducing the issue you've shown above).

[techalchemy]

I think we understand the issue. Here is my proposed solution, if there are any concerns raise them here and we can discuss:

Our current implementation iterates over the indexes and attempts to install each top level package using each index. It used to fail silently, obviously that doesn't work anymore. This is problematic for a couple of reasons

1. We have to manually iterate over each package and each index unless it's explicitly stated
2. We don't resolve dependencies across indexes (i.e. if they are not on the same index, or if a dependency is on a private index)

My proposal is this:

1. The first listed source will become the primary index
2. The subsequent sources will become arguments to EXTRA_INDEX_URL
3. Perhaps we can add an optional flag that indicates that the location should be treated as a FIND_LINKS location rather than an EXTRA_INDEX_URL which would enable local wheel repositories and the like

Thoughts?

[kennethreitz42]

+1

[kennethreitz42]

We use --extra-index-url for the second and third (etc) source entries.

[kennethreitz42]

ah only for locking

[kennethreitz42]

reproduced. fixing.

[kennethreitz42]

I actually can't get this to reproduce locally. I note a line here:

Could not fetch URL http://localhost:8080/boto3/: 403 Client Error: Forbidden for url: http://localhost:8080/boto3/ - skipping

I see what's going on here.

[kennethreitz42]

So you're providing a package from one index that depends on packages from another index.

[kennethreitz42]

I believe https://github.com/pypa/pipenv/commit/d5b9b36713826746d69cde6d9256d33a528881f0 should fix this right up :)

[ryanwilsonperkin]

Hey @kennethreitz, I don't think d5b9b36 has quite solved this issue. I've attempted to reproduce by publishing a package to Test PyPI that references a package on regular PyPI.

---

Expected behaviour

Installs pipenv-test-private-package from TestPyPI and discovers dependency on pipenv-test-public-package which is available on regular PyPI.

Works when running pip directly:

/private/tmp/pip  (venv) pip install --extra-index-url https://testpypi.python.org/pypi pipenv-test-private-package
Collecting pipenv-test-private-package
  Downloading pipenv-test-private-package-1.0.tar.gz
Collecting pipenv-test-public-package (from pipenv-test-private-package)
  Downloading pipenv-test-public-package-1.0.tar.gz
Building wheels for collected packages: pipenv-test-private-package, pipenv-test-public-package
  Running setup.py bdist_wheel for pipenv-test-private-package ... done
  Stored in directory: /Users/ryan/Library/Caches/pip/wheels/2e/79/07/9f90adc981af1f8dd8fb1478c14f40228d0820b619d6e6b507
  Running setup.py bdist_wheel for pipenv-test-public-package ... done
  Stored in directory: /Users/ryan/Library/Caches/pip/wheels/21/0f/7d/581265ed90f2c7539e6ea64e5eb118ffbe652a4d6bade55d44
Successfully built pipenv-test-private-package pipenv-test-public-package
Installing collected packages: pipenv-test-public-package, pipenv-test-private-package
Successfully installed pipenv-test-private-package-1.0 pipenv-test-public-package-1.0

Actual behaviour

Fails with the following error message

/p/t/pipenv pipenv install --verbose
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://testpypi.python.org/pypi

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://testpypi.python.org/pypi

                          ROUND 1
Current constraints:
  pipenv-test-private-package

Finding the best candidates:
  found candidate pipenv-test-private-package==1.0 (constraint was <any>)

Finding secondary dependencies:
  pipenv-test-private-package==1.0 not in cache, need to check index
  pipenv-test-private-package==1.0 requires pipenv-test-public-package

New dependencies found in this round:
  adding [u'pipenv-test-public-package', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  pipenv-test-private-package
  pipenv-test-public-package

Finding the best candidates:
  found candidate pipenv-test-private-package==1.0 (constraint was <any>)

INFO:pip._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): testpypi.python.org
INFO:pip._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org
INFO:pip.download:Using cached pipenv-test-private-package-1.0.tar.gz
INFO:pip.download:Saved /Users/ryan/Library/Caches/pipenv/pkgs/pipenv-test-private-package-1.0.tar.gz
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pipenv/resolver.py", line 66, in <module>
    main()
  File "/usr/local/lib/python2.7/site-packages/pipenv/resolver.py", line 54, in main
    results = resolve(packages, pre=do_pre, sources=project.sources, verbose=is_verbose, clear=do_clear)
  File "/usr/local/lib/python2.7/site-packages/pipenv/resolver.py", line 52, in resolve
    return pipenv.utils.resolve_deps(packages, which, project=project, pre=pre, sources=sources, clear=clear, verbose=verbose)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/utils.py", line 398, in resolve_deps
    resolved_tree, resolver = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/utils.py", line 328, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/repositories/pypi.py", line 114, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/repositories/pypi.py", line 99, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 605, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 719, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 831, in get_page
    resp.raise_for_status()
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/../pip/_vendor/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://testpypi.python.org/pypi/pipenv-test-public-package/

/usr/local/lib/python2.7/site-packages/pipenv/utils.py:1157: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/k_/yfvkl0_n2y9_t7xr9gd16r740000gn/T/pipenv-LtrMA1-requirements'>
  warnings.warn(warn_message, ResourceWarning)

Steps to reproduce

Using pipenv version 1.15.2, run pipenv install using the following Pipfile

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "testpypi"

[packages]
pipenv-test-private-package = "*"

---

For simplicity's sake, I've hosted the contents of the private and public packages at the following gists:

• Private: https://gist.github.com/ryanwilsonperkin/585250b0558e7fbc380536c40d0b9005
• Public: https://gist.github.com/ryanwilsonperkin/d8d591e08e0996414eea27b25f4bb68c

My python -m pipenv.help is:

Pipenv version: '11.5.2'

Pipenv location: '/usr/local/lib/python2.7/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python2.7'

Other Python installations in PATH:

• 2.6: /usr/bin/python2.6
• 2.6: /usr/bin/python2.6
• 2.7: /usr/local/bin/python2.7
• 2.7: /usr/local/bin/python2.7
• 2.7: /usr/bin/python2.7
• 3.6: /usr/local/bin/python3.6m

• 3.6: /usr/local/bin/python3.6

• 2.7.13: /usr/local/bin/python

• 2.7.10: /usr/bin/python
• 2.7.13: /usr/local/bin/python2
• 3.6.4: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '15.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 15.6.0: Mon Nov 13 21:58:35 PST 2017; root:xnu-3248.72.11~1/RELEASE_X86_64',
 'python_full_version': '2.7.13',
 'python_version': '2.7',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_PROGRAM_VERSION
• LOGNAME
• USER
• NVM_DIR
• HOME
• PATH
• PYTHONUNBUFFERED
• DISPLAY
• TERM_PROGRAM
• LANG
• TERM
• Apple_PubSub_Socket_Render
• COLORFGBG
• SHLVL
• SECURITYSESSIONID
• XPC_FLAGS
• ITERM_SESSION_ID
• EDITOR
• TERM_SESSION_ID
• XPC_SERVICE_NAME
• SSH_AUTH_SOCK
• SHELL
• ITERM_PROFILE
• PIP_PYTHON_PATH
• TMPDIR
• __CF_USER_TEXT_ENCODING
• PWD
• COLORTERM

[gzzo]

@kennethreitz again, I think this has been incorrectly closed.

Not installing deps, as in d5b9b36, is not a proper solution. We obviously still want dependencies installed.

What @techalchemy said makes sense, pipenv should use EXTRA_INDEX_URL to install packages instead of INDEX_URL.

[gzzo]

Thanks! Confirmed it's working.

[techalchemy]

@gzzo note that as of 2399080 you may need to regenerate your lockfiles as there was a bug in the original implementation which caused us to write the wrong index names!

Sorry about that!