pipenv install fails when any [[source]] other than pypi is used

[dpratt]

Describe your environment

1. OS Type MacOS 10.12.6, Ubuntu 16.04
2. Python version: 3.6.4
3. Pipenv version: 9.0.3

Expected result

'pipenv install' with a custom source should work.

Actual result

Traceback (most recent call last):
File "/usr/local/bin/pipenv", line 11, in
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
return self.main(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
return callback(args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1865, in install
do_init(dev=dev, allow_global=system, ignore_pipfile=ignore_pipfile, system=system, skip_lock=skip_lock, verbose=verbose, concurrent=concurrent, deploy=deploy, pre=pre)
File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1306, in do_init
do_lock(system=system, pre=pre)
File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1102, in do_lock
pre=pre
File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 545, in resolve_deps
resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 507, in actually_resolve_reps
resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 200, in _resolve_one_round
for dep in self._iter_dependencies(best_match):
File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 296, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 153, in get_dependencies
result = reqset._prepare_file(self.finder, ireq)
File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/req/req_set.py", line 671, in _prepare_file
check_dist_requires_python(dist)
File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/utils/packaging.py", line 56, in check_dist_requires_python
os.environ['PIP_PYTHON_VERSION']
pip.exceptions.UnsupportedPythonVersion: futures requires Python '>=2.6, <3' but the running Python is 3.6.4

Steps to replicate

Create a pipfile that looks like the following -

[[source]]
url = "https://my.mirror.host/mirror/simple"
verify_ssl = true
name = "pypi-mirror"

[packages]
keras = "*"

[dev-packages]

[requires]
python_version = "3.6"

'pipenv install' fails with the above error - if you replace the source section with the default, it works perfectly. This is a duplicate of #1166, but that was closed by the submitter, but it is still present.

[nateprewitt]

@dpratt can you provide the output of pipenv run python --version and pipenv run pip --version for me and report the output? What method did you install pipenv with and what version of python did you use? Thanks!

[dpratt]

'pipenv run python --version'

Python 3.6.4

'pipenv run pip --version'

pip 9.0.1 from /Users/dpratt/.local/share/virtualenvs/keras-room-classifier-Gplti04v/lib/python3.6/site-packages (python 3.6)

I've replicated this by installing by doing both 'pip3 install --user pipenv' and 'pip3 install pipenv' on my Mac and on a fresh ubuntu 16.04 machine.

[kennethreitz42]

this looks like an installation issue, not a pipenv/pypi issue.

[dpratt]

I don’t really understand how it could be.

If I leave the URL as the default in my Pipfile, everything works perfectly, but changing it to anything but the default causes the above problem.

I have replicated this on my local MacOS environment as well as several different EC2 instances. If it’s an installation issue, what is installed incorrectly?

On Feb 17, 2018, at 7:26 AM, Kenneth Reitz [e-mail] wrote:

this looks like an installation issue, not a pipenv/pypi issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[kennethreitz42]

Maybe I closed too early

[dpratt]

Has there been a fix for this? I can still reproduce it across several environments.

[kennethreitz42]

we're aware of the issue and are just cleaning up issues

[kennethreitz42]

cannot reproduce with the following Pipfile:

[[source]]

url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "pypi"


[packages]

requests = "*"


[dev-packages]



[requires]

python_version = "3.6"

[jtratner]

@dpratt -if you're using devpi to host your pypi mirror, try upgrading the version. I think subtle API issues can cause this problem (and has to do with pip/internal PyPI interactions)

[kennethreitz42]

Update: we also have all of PyPI mocked out in our test suite, and have run into zero issues.

[kennethreitz42]

I wrote the Flask app that mimics PyPI myself in about 20 minutes.

[kennethreitz42]

Point: this is very well tested and is not problematic, as far as we can tell.

[dpratt]

Thanks for the update - we currently use Nexus to host our mirror, I’ll start looking there. Is there a way to turn on verbose logging for pipenv/pip?

[techalchemy]

P.S are you still using pipenv 9?

[kalfa]

I'm still getting the same exception in a very similar context. Using PyPiCloud and I can reproduce it on OSX and Centos6 EC2 instances

It wasn't clear if it is aIready known what's causing this, so I investigated a bit and found out that ultimately also pip CLI shows different behaviours depending on the server contacted, probably it's the server itself returning different information?

pipenv, version 9.0.3 # should be the latest one from pypi
python 3.3.6 # required, cannot change
pip 9.0.1 from /users/kalfa/.local/share/virtualenvs/ADS-network-model/lib/python3.3/site-packages (python 3.3) # should be the latest one from pypi

Against PyPiCloud:

$ pip install --pre    --trusted-host PYPYCLOUDHOST --index-url https://PYPYCLOUDHOST/simple pytest
Collecting pytest
  Downloading https://PYPYCLOUDHOST/api/package/pytest/pytest-3.4.2-py2.py3-none-any.whl (189kB)
    100% |████████████████████████████████| 194kB 37.8MB/s
pytest requires Python '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*' but the running Python is 3.3.6

Against the default pip pypi server:

$ pip install --pre pytest
Collecting pytest
  Using cached pytest-3.2.5-py2.py3-none-any.whl
Requirement already satisfied: py>=1.4.33 in /users/kalfa/.local/share/virtualenvs/ADS-network-model/lib/python3.3/site-packages (from pytest)
Requirement already satisfied: setuptools in /users/kalfa/.local/share/virtualenvs/ADS-network-model/lib/python3.3/site-packages (from pytest)
Installing collected packages: pytest
Successfully installed pytest-3.2.5

Not sure yet if it's actually the server replying differently, but I supsect so, because it's the only thing that changes in the two scenarios.

In fact, pkg_resources.DistInfoDistribution metadata in check_dist_requires_python() in the former case returns "Requires-Python" info, which is unmet if I'm using python 3.3.6 and tyring to install something with "Requires-Python: >=2.7, !=3.0., !=3.1., !=3.2., !=3.3.".
In the latter, the header/metadata is lacking, and pip actually is happy to continue and install pytest against py 3.3.6.

Unless anyone objects, I'll try to draft a patch for an option to ignore Requires-Python even if it's passed among the pkg metadata.

[kennethreitz42]

please do. I had this working, actually, but had to remove it as it was causing bugs.

[kennethreitz42]

e.g. please do it properly (as i did it improperly)

[kennethreitz42]

✨🍰✨

[kalfa]

@kennethreitz any suggestion about how you'd like to see it 'properly'? or what you would not like to see ;-)

[kalfa]

anyway, patching only pipenv wouldn't resovle the problem, unless we can issue a 'pip install' telling pip to ignore the metadata as well.

Otherwise it will simply postpone the rejection when pip is actually executed. Am I getting something wrong?