Pipenv: Pipenv should check implementation version when choosing a runtime

[joshfriend]

Describe your environment

1. OS Type: macOS 10.13.3
2. Python version: $ python -V

Python 2.7.13 (c925e7381036, Jun 05 2017, 20:53:58)
[PyPy 5.8.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]

1. Pipenv version: $ pipenv --version: pipenv, version 9.1.0

Expected result

$ pyenv local pypy2.7-5.8.0
# Creates .python-version that sets the project's python 
# version to the specific pypy version
$ pipenv install --python pypy
# Installs packages in virtualenv created with pypy2.7-5.8.0

Actual result

$ pipenv install --python pypy
Creating a virtualenv for this project…
Using /Users/josh/.pyenv/versions/pypy2.7-5.9.0/bin/pypy to create virtualenv…
# Installs packages in virtualenv created with pypy2.7-5.9.0, 
# or whatever the latest version of pypy2.7 I have installed is

I have to uninstall pypy2.7-5.9.0 and any other later versions to be able to use v5.8.0 with pipenv. I need to test with 5.8.0 specifically because that is the latest version that the heroku python buildpack supports.

Perhaps pipenv should defer to .python-version when choosing the python version to use, unless it conflicts with python_version or python_full_version in the Pipfile, in which case throw an error.

[jtratner]

Could you clarify one thing: if you do which pypy in that directory, what
does it resolve to?

My understanding is that the --python flag is supposed to get path to
binary (or do a lookup based on current path), but possibly there needs to
be additional magic for pyenv, I’m unclear.
On Thu, Feb 22, 2018 at 1:29 PM Josh Friend [e-mail]
wrote:

Describe your environment

1. OS Type: macOS 10.13.3
2. Python version: $ python -V

Python 2.7.13 (c925e7381036, Jun 05 2017, 20:53:58)
[PyPy 5.8.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]

1. Pipenv version: $ pipenv --version: pipenv, version 9.1.0

Expected result

$ pyenv local pypy2.7-5.8.0

Creates .python-version that sets the project's python

version to the specific pypy version

$ pipenv install --python pypy

Installs packages in virtualenv created with pypy2.7-5.8.0

Actual result

$ pipenv install --python pypy
Creating a virtualenv for this project…
Using /Users/josh/.pyenv/versions/pypy2.7-5.9.0/bin/pypy to create virtualenv…

Installs packages in virtualenv created with pypy2.7-5.9.0,

or whatever the latest version of pypy2.7 I have installed is

I have to uninstall pypy2.7-5.9.0 and any other later versions to be able
to use v5.8.0 with pipenv. I need to test with 5.8.0 specifically because
that is the latest version that the heroku python buildpack supports.

Perhaps pipenv should defer to .python-version when choosing the python
version to use, unless it conflicts with python_version or
python_full_version in the Pipfile, in which case throw an error.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pipenv/issues/1473, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABhjq87CbYkadZMlKdzHgRiF_RByLvFEks5tXdxFgaJpZM4SQBDa
.

[techalchemy]

We've touched upon this in a few places I'm pretty sure, if you pass --python at all pipenv will find the first instance of the supplied python on your path, it was never intended to parse your pyenv local settings files or call pyenv shims. If you want a specific executable you can use the absolute path rather than just the name, or in this case you can likely use pipenv install --python pypy2.7-5.8.0 since it will crawl your .pyenv/versions directory.

possibly there needs to be additional magic for pyenv, I’m unclear

Yep, there certainly does, and there is -- we compare the user input against everything under $PYENV_ROOT/versions

_edit:_ See also: #715

[joshfriend]

which pypy points to the pyenv shims directory (as you would expect):

$ which pypy
/Users/josh/.pyenv/shims/pypy

you can likely use pipenv install --python pypy2.7-5.8.0

I tried this and pipenv assumes that pypy2.7-5.8.0 is a version of CPython:

$ pipenv install --python pypy2.7-5.8.0
Warning: Python pypy2.7-5.8.0 was not found on your system…
Would you like us to install CPython pypy2.7-5.8.0 with pyenv? [Y/n]: lol no

You _can_ use the filesystem path to a directory containing a python installation, which I found out from an "advanced" section about use with Conda. Unfortunately this path will be different depending on where pyenv is installed and can't be assumed.

[uranusjr]

Does pypy --version in the project root indicate the correct version number? Since pypy points to the shim, Pipenv would simply use it, and the shim should in turn point Pipenv to the correct PyPy executable. At least that’s how I would expect things to work, reading the source (without actually experimenting).

[joshfriend]

pypy --version returns Python 2.7.13 for both v5.8.0 and v5.9.0. It outputs the PyPy version number below that, but I don't think pipenv looks at that. As long as platform_python_implementation is PyPy and the python_version or python_full_version markers match, I believe pipenv just picks the first match it finds in the pyenv installed versions.

pipenv does not look at the shims to determine which python to use, it scans $PYENV_ROOT/versions for what is installed, adds them to the $PATH and finds the first one that matches the environment markers.

[uranusjr]

@joshfriend Ah, I see, that makes sense. Sorry for my lack of understanding toward PyPy’s versioning. Funny thing is, the pyenv shim is actually able to do the right thing automatically. This would probably work:

$ pipenv install --python=$(which pypy)

But Pipenv’s pyenv integration adds bin directories before the shim, and prevents it from being found. It would require some careful deliberation how we can handle this correctly.

---

By the way, being curious here since I’m really not familiar with PyPy, what is behind the choice of 5.8 over 5.9? Shouldn’t 5.9 be feature-compatible to 5.8, but better?

[joshfriend]

$(which pypy) is a good idea, I had not thought of that :+1:

I need to test with 5.8.0 specifically because that is the latest version that the heroku python buildpack supports.

I'm just trying to minimize surprises when deploying to production ;)

I would say that perhaps the implementation_name and implementation_version should also be included in the check for the right python along with python_full_version. Unfortunately that info is only available in Python 3.3+. Also the only place i can find that looks at those markers is pipfile.assert_requirements(), which is not used in pipenv AFAICT...

[uranusjr]

implementation_name and implementation_version checks sounds like a good idea though. It wouldn’t help this particular use case, but in the long run would be very useful imo. What do you think @kennethreitz?

[techalchemy]

I'm fairly sure we intentionally decided not to be too fancy about how we handle this because of the sheer number of runtimes and the inconsistencies in how they are handled across platforms, systems, etc. Also, we opted by bypass pyenv shims intentionally and consciously.

I'd be curious to see an implementation that handles any runtime and version on any platform without requiring a massive amount of maintenance overhead

[kennethreitz42]

we ignore shims because they are problematic

[kennethreitz42]

the solution to this is to simply use --python=/path/to/python

[joshfriend]

I think this should be reopened because as is, pipenv install creates environments that immediately fail check. For example:

Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
testpackage = "*"

[requires]
python_full_version = "3.5.3"
implementation_name = "pypy"
implementation_version = "5.10.1"

pipenv install will create the venv with CPython (probably, unless you _only_ have pypy installed), but pipenv check will then immediately fail:

$ pipenv check
Checking PEP 508 requirements…
Specifier implementation_name does not match pypy (cpython).
Specifier implementation_version does not match 5.10.1 (3.5.3).
Failed!

Pipfile.lock:

{
    "_meta": {
        "hash": {
            "sha256": "c637dfe39ab17bf44ba426526e66497c8e9d4663eafa9214c7e5edcb13b0d7e2"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.5.3",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "17.4.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64",
            "python_full_version": "3.5.3",
            "python_version": "3.5",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {
            "implementation_name": "pypy",
            "implementation_version": "5.10.1",
            "python_full_version": "3.5.3"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "testpackage": {
            "hashes": [
                "sha256:fd0076de3a2c8c695826e466cbb0f9febc6ea8474302ec3e673dbe46cd278b6a"
            ],
            "version": "==2.27"
        }
    },
    "develop": {}
}

The expected behavior would be that anything in the requires section of the Pipfile should be considered for install such that pipenv doesn't end up creating environments that fail the check process with no changes. For this example, that would mean looking at the python implementation type when finding a matching python executable. I believe only python_full_version and python_version are currently used during install.

Of course, not all PEP508 markers will be useful as a requires directive, but if they are given in the Pipfile and don't match, an error should be given if the criteria are not able to be satisfied.

This feature would be very useful for anyone that requires non CPython runtimes.

@techalchemy I don't think this would need to use pyenv shims. I think when selecting the python runtime, pipenv will just have to additionally examine implementation_version and implementation_name. That's something I might try out soon when I get the chance. Limitations on that would be that any runtime with python_version less than 3.3 will report those markers as cpython and 0. In that case the only recourse is to specify the full path to the executable.

[techalchemy]

At a glance you are simply suggesting improvements to our existing handling of this which seem straightforward. While the issue is closed (think of it as triaged -> not prioritized yet), I suspect we’d entertain a PR on this