Pipenv: markers 'platform_python_implementation == "cpython"' don't match your environment when installing gevent

Created on 1 May 2018  ·  19Comments  ·  Source: pypa/pipenv

When installing gevent using pipenv, greenlet is not being installed.

$ python -m pipenv.help output

Pipenv version: '11.10.1'

Pipenv location: '/usr/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python'

Other Python installations in PATH:

  • 2.7: /usr/bin/python2.7
  • 2.7: /usr/bin/python2.7
  • 2.7: /home/jeremy/.pyenv/shims/python2.7
  • 2.7: /usr/bin/python2.7
  • 3.6: /usr/bin/python3.6m
  • 3.6: /usr/bin/python3.6
  • 3.6: /home/jeremy/.pyenv/shims/python3.6

  • 3.6: /usr/bin/python3.6

  • 3.6.5: /usr/bin/python

  • 3.6.5: /home/jeremy/.pyenv/shims/python
  • 3.6.5: /usr/bin/python
  • 2.7.14: /usr/bin/python2
  • 2.7.14: /home/jeremy/.pyenv/shims/python2
  • 2.7.14: /usr/bin/python2
  • 3.6.5: /usr/bin/python3
  • 3.6.5: /home/jeremy/.pyenv/shims/python3
  • 3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.16.5-1-ARCH',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Thu Apr 26 16:53:40 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

  • LS_COLORS
  • LANG
  • LESS
  • FZF_DEFAULT_COMMAND
  • DISPLAY
  • PYENV_ROOT
  • OLDPWD
  • INVOCATION_ID
  • EDITOR
  • COLORTERM
  • PYENV_VIRTUALENV_INIT
  • MOZ_PLUGIN_PATH
  • PYENV_HOOK_PATH
  • XDG_VTNR
  • ZSH
  • SSH_AUTH_SOCK
  • FZF_CTRL_T_COMMAND
  • XDG_SESSION_ID
  • USER
  • PYENV_DIR
  • PAGER
  • LSCOLORS
  • PWD
  • HOME
  • LC_CTYPE
  • JOURNAL_STREAM
  • PYENV_VERSION
  • MAIL
  • VISUAL
  • WINDOWPATH
  • TERM
  • SHELL
  • VTE_VERSION
  • XDG_SEAT
  • SHLVL
  • PYENV_SHELL
  • WINDOWID
  • MAVEN_OPTS
  • LOGNAME
  • DBUS_SESSION_BUS_ADDRESS
  • XDG_RUNTIME_DIR
  • XAUTHORITY
  • PATH
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/bin:/home/jeremy/.pyenv/libexec:/home/jeremy/.pyenv/plugins/python-build/bin:/home/jeremy/.pyenv/plugins/pyenv-virtualenv/bin:/home/jeremy/.pyenv/plugins/pyenv-virtualenv/shims:/home/jeremy/.pyenv/shims:/home/jeremy/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jeremy/bin:/home/jeremy/.gem/ruby/2.5.0/bin:/home/jeremy/.local/bin/:/home/jeremy/thirdrepos/google-cloud-sdk/bin
  • SHELL: /usr/bin/zsh
  • EDITOR: vim
  • LANG: en_US.UTF-8
  • PWD: /home/jeremy

Expected result

I expect installing gevent to also result in greenlet being installed.

Actual result

Instead of having greenlet installed, I get a warning about "Ignoring greenlet: markers 'platform_python_implementation == "cpython"' don't match your environment".

Steps to replicate
➜  ~/tmp/pipenving echo '[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gevent = "*"

[dev-packages]

[requires]
python_version = "2.7"

[pipenv]
allow_prereleases = true' > Pipfile
➜  ~/tmp/pipenving pipenv install
Creating a virtualenv for this project…
Using /home/jeremy/.pyenv/versions/jfly-2.7.12/bin/python2.7 (2.7.12) to create virtualenv…
⠋Running virtualenv with interpreter /home/jeremy/.pyenv/versions/jfly-2.7.12/bin/python2.7
Using real prefix '/home/jeremy/.pyenv/versions/2.7.12'
New python executable in /home/jeremy/.local/share/virtualenvs/pipenving-IYbJmdaL/bin/python2.7
Also creating executable in /home/jeremy/.local/share/virtualenvs/pipenving-IYbJmdaL/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/jeremy/.local/share/virtualenvs/pipenving-IYbJmdaL
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (6d8673)!
Installing dependencies from Pipfile.lock (6d8673)…
Ignoring greenlet: markers 'platform_python_implementation == "cpython"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
➜  ~/tmp/pipenving pip freeze | grep gevent
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
➜  ~/tmp/pipenving pipenv run pip freeze | grep greenlet
➜  ~/tmp/pipenving pipenv run pip freeze | grep gevent
gevent==1.3b1

Note the warning above about platform_python_implementation == "cpython" not matching my environment. I looked into this a bit, and according to PEP508, platform_python_implementation corresponds to platform.python_implementation(), which is equal to "CPython" for me:

➜  ~/tmp/pipenving pipenv run python -c "import platform; print platform.python_implementation()"
CPython

I verified that if I manually change the "markers": "platform_python_implementation == 'cpython'" line in my Pipfile.lock from "cpython" to "CPython", then pipenv install works happily:

➜  ~/tmp/pipenving sed -i 's/cpython/CPython/' Pipfile.lock 
➜  ~/tmp/pipenving pipenv install                           
Installing dependencies from Pipfile.lock (6d8673)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
➜  ~/tmp/pipenving pipenv run pip freeze | grep greenlet
greenlet==0.4.13

However, this isn't a permanent solution for me, as subsequent runs of pipenv lock reset my Pipfile.lock to have a lowercase "cpython":

➜  ~/tmp/pipenving pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (6d8673)!
➜  ~/tmp/pipenving grep cpython Pipfile.lock 
            "markers": "platform_python_implementation == 'cpython'",

I confirmed that gevent 1.3b1 is correctly capitalizing "CPython" in its setup.py: https://github.com/gevent/gevent/blob/1.3b1/setup.py#L190, so I concluded that pipenv must be lowercasing this marker somewhere. I stepped through pipenv with a debugger to track down where the platform_python_implementation marker is getting lowercased, and I tracked it down to this line:

https://github.com/pypa/pipenv/blob/a16b1d06515b26db929e87905304ba00a9e1f33b/pipenv/patched/piptools/resolver.py#L299

which invokes format_requirement, which lowercases the entire requirement, including the markers:

https://github.com/pypa/pipenv/blob/a16b1d06515b26db929e87905304ba00a9e1f33b/pipenv/patched/piptools/utils.py#L69

It looks like the format_requirement method was written with the intention of markers being specified as an optional second argument, so maybe the right fix is to change our invocation of format_requirement to actually split off the markers and pass them in separately?

picture jfly
👍4

Most helpful comment

I've sent in a PR here: https://github.com/pypa/pipenv/pull/2123.

Until we get this fixed, I am able to workaround this by adding the following line to our Pipfile:

greenlet = {version = "*", platform_python_implementation="=='CPython'"}

(I got the idea for this hack from https://github.com/pypa/pipenv/issues/1757#issuecomment-373558866)

picture jfly on 2 May 2018
👍4

All 19 comments

If I read PEP 508 correctly, the comparison should be case insensitive (it even lists 'cpython' as an acceptable value). The lower-casing shouldn’t be a problem; it’s the comparing part that should be fixed.

uranusjr picture uranusjr on 2 May 2018

Are you sure about that? The only mention of case sensitivity I see in PEP 508 is about python distribution names. As I understood it, this is about environment markers, which I believe should be using regular Python string comparison (emphasis mine):

The operators use the PEP-440 [4] version comparison rules when those are defined (that is when both sides have a valid version specifier). If there is no defined PEP-440 behaviour and the operator exists in Python, then the operator falls back to the Python behaviour.

The example you give of 'cpython' in the PEP is for the implementation_name environment marker. The issue I raised is about the platform_python_implementation environment marker, which PEP 508 gives "CPython, Jython" as example values.

jfly picture jfly on 2 May 2018

Yes, you are correct, I misread. The code you mentioned is the correct place to fix this in that case. Would you mind convert it to a pull request?

uranusjr picture uranusjr on 2 May 2018

Sure, I will try to send in a PR tomorrow. Thanks!

jfly picture jfly on 2 May 2018

Jeez you got into some code on this. Here’s the underlying code from the platform module:

def python_implementation():

    """ Returns a string identifying the Python implementation.
        Currently, the following implementations are identified:
          'CPython' (C implementation of Python),
          'IronPython' (.NET implementation of Python),
          'Jython' (Java implementation of Python),
          'PyPy' (Python implementation of Python).
    """
    return _sys_version()[0]
techalchemy picture techalchemy on 2 May 2018

I've sent in a PR here: https://github.com/pypa/pipenv/pull/2123.

Until we get this fixed, I am able to workaround this by adding the following line to our Pipfile:

greenlet = {version = "*", platform_python_implementation="=='CPython'"}

(I got the idea for this hack from https://github.com/pypa/pipenv/issues/1757#issuecomment-373558866)

jfly picture jfly on 2 May 2018
👍4

Just an FYI: this should probably not have been closed yet since the fix hasn't been released (we just got bit by it).

Bogdanp picture Bogdanp on 24 May 2018

@Bogdanp it's fixed on master though and keeping issues open until the fix is released puts quite a bit more workload on the maintainers.

Turbo87 picture Turbo87 on 24 May 2018

@Turbo87 if that's the project's policy, then fair enough. I thought it was worth pointing out since GitHub most likely closed this automatically. That said, the fact that it adds some maintenance burden doesn't mean it's not the right thing to do. :)

Bogdanp picture Bogdanp on 24 May 2018

We try our best not to fight too hard against the tools we are using, none of us is doing this full time and this tracker is more for our own organization than for making announcements or anything like that.

techalchemy picture techalchemy on 24 May 2018

Yeah, GitHub is better suited for the current workflow (closing on resolution, not release). Maybe we can add a tag to all un-released issues, but that would still require quite some administration overhead. I wish we have an ops maintainer to handle these kind of things. You don’t even need to know how to code, but alas people are better at pointing out issues than actually trying to solve them 🙂

uranusjr picture uranusjr on 25 May 2018

Is there an ETA for when a version containing this fix might be released?

judgeaxl picture judgeaxl on 4 Jun 2018

@judgeaxl From what I’ve heard a version would be out this week.

uranusjr picture uranusjr on 4 Jun 2018

Depending on the ongoing merges etc going smoothly it should be in the next day or two

techalchemy picture techalchemy on 4 Jun 2018
🎉1

Any news on when the next release happens? Would be nice to get this fix finally available.

bigunyak picture bigunyak on 12 Jun 2018

Track https://github.com/pypa/pipenv/projects/2 to find out! We still have a few pressing bugs need fixing.

uranusjr picture uranusjr on 12 Jun 2018
👍1

@uranusjr would it not be possible to release a version with the existing bugfixes instead of waiting for all bugs to be fixed?

Turbo87 picture Turbo87 on 12 Jun 2018

@Turbo87 It’s not that simple (unfortunately). Some bugs are introduced by the bugfix. The release will fix your problem, and introduce problems to others :(

uranusjr picture uranusjr on 12 Jun 2018

I see, thanks for the explanation!

Turbo87 picture Turbo87 on 12 Jun 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jakul picture jakul  ·  3Comments
ipmb picture ipmb  ·  3Comments
marc-fez picture marc-fez  ·  3Comments
jacebrowning picture jacebrowning  ·  3Comments
ansrivas picture ansrivas  ·  3Comments