pipenv doesn't find correct python version on $PATH without pyenv

[jarshwah]

This is related to https://github.com/pypa/pipenv/issues/729 but I think I've found a different cause. When multiple versions of python 3 are installed on $PATH and pyenv is not available, then pipenv will only look at /usr/bin/python3 and not more explicit versions of python3 like /usr/bin/python3.6 even if python_version declares 3.6 as the target.

We're using pipenv on our build server (jenkins) and haven't quite figured out how to get pyenv working with jenkins pipelines yet.

Describe your environment

1. OS Type: Ubuntu 16.04
2. Python versions:

jenkins@jenkin:~/workspace$ /usr/bin/python3 --version
Python 3.5.2
jenkins@jenkins:~/workspace$ /usr/bin/python3.5 --version
Python 3.5.2
jenkins@jenkins:~/workspace$ /usr/bin/python3.6 --version
Python 3.6.3

1. Pipenv version: 9.0.3

Expected result

Virtualenv should be created using /usr/bin/python3.6 if it's available on $PATH

Actual result

$ pipenv install --dev --three --verbose
Creating a virtualenv for this project…
Using /usr/bin/python3 to create virtualenv…
⠋Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/lib/jenkins/.local/share/virtualenvs/venvnamehere/bin/python3
Also creating executable in /var/lib/jenkins/.local/share/virtualenvsvenvnamehere/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /var/lib/jenkins/.local/share/virtualenvs/venvnamehere

Warning: Your Pipfile requires python_version 3.6, but you are using 3.5.2 (/var/lib/j/.local/share/v/r/bin/python)

Files

Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[dev-packages]

"flake8" = "*"
isort = "*"
werkzeug = "*"
pyopenssl = "*"
factory-boy = "<3.0,>=2.9"
"flake8-isort" = "*"
"flake8-builtins" = "*"
"flake8-comprehensions" = "*"
ipdb = "*"
responses = "*"


[packages]

django = ">=1.11.0,<2.0.0"
wagtail = "<1.14,>=1.13"
dj-database-url = "*"
gunicorn = "*"
"psycopg2" = "*"
whitenoise = "*"
raven = "*"
django-extensions = "*"
xmltodict = "*"
django-cloudinary-storage = "*"
newrelic = "*"
pygments = "<2.3.0,>=2.2.0"
django-fsm = "<3.0,>=2.5"
django-debug-toolbar = "*"


[requires]

python_version = "3.6"

[uranusjr]

The current implementation does not consider the content of Pipfile when building the environment, so if you specify --three it will use the first available python3 command. If Python 3.6 is needed specifically, you need to specify --python 3.6 instead. I’m not sure if this fits the core devs’ design goals, but agree it would be reasonable if --three honours requires.python_version in Pipfile, if available.

[jarshwah]

Thanks @uranusjr we ended up resolving our issues by passing --python as suggested. Under what circumstances is requires.python_version honoured? If I understand correctly requires.python_version works properly with pyenv. I wouldn't imagine it'd be that much of a leap to do the equivalent of command -v python{requires.python_version}?

Hah just did some testing. It looks like requires.python_version works if you leave off --three altogether.

[uranusjr]

@jarshwah My wild guess is it’s simply because requires.python_version was implemented after the --three and --python options, and nobody actually thought of reusing this new information on those flags.

[techalchemy]

@jarshwah if you pass a python version explicitly as a CLI flag it will always act as a hard override of whatever is in the Pipfile. That behavior is intentional, since if you just want to install according to a pipfiles specification you would just use pipenv install. If you use —three I believe pipenv will use the first version of python 3 available to it on your path.

Edit: thanks for reporting this and sorry for the confusion. Hopefully this helps someone else out if they run into it. I’m going to close this out for now but let us know if you run into other issues!

[jarshwah]

I think having --three and --python and requires.python_version all being independent confusing. Would a docs update to the requires.python_version or --three be welcome to explain the precedence and actual behaviour?

[uranusjr]

The current rule works like this (if pyenv is not present):

1. --three is essentially a shorthand for --python=3 (and --two is --python=2).
2. If --python=X is specified, look for python{X} in PATH (if X looks like a version number).
3. Otherwise, if requires.python_version = Y look for python{Y} in PATH.
4. Otherwise use whatever Python Pipenv is on (sys.executable).

The logic is relatively straightforward by itself IMO, but not really what users would expect.