pipenv respects version constraint from incorrect python version

[dspechnikov]

drf-yasg package has 2 Django version constraints:

Django (>=1.11.7); python_version >= "3.4"
Django (<2.0,>=1.11.7); python_version <= "2.7"

pipenv uses 2.7 one even when invoked in python 3.6 virtual env.

$ python -m pipenv.help output

Pipenv version: '11.10.0'

Pipenv location: '/Users/dspeck/work/test/.venv/lib/python3.6/site-packages/pipenv'

Python location: '/Users/dspeck/work/test/.venv/bin/python'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7
• 2.7: /usr/bin/python2.7
• 3.6: /usr/local/bin/python3.6m
• 3.6: /Users/dspeck/work/test/.venv/bin/python3.6

• 3.6: /usr/local/bin/python3.6

• 3.6.4: /Users/dspeck/work/test/.venv/bin/python

• 3.6.4: /usr/local/bin/python
• 2.7.10: /usr/bin/python
• 3.6.4: /Users/dspeck/work/test/.venv/bin/python3
• 3.6.4: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST '
                     '2018; root:xnu-4570.51.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.4',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_SESSION_ID
• SSH_AUTH_SOCK
• Apple_PubSub_Socket_Render
• COLORFGBG
• ITERM_PROFILE
• XPC_FLAGS
• PWD
• SHELL
• SECURITYSESSIONID
• LC_CTYPE
• TERM_PROGRAM_VERSION
• TERM_PROGRAM
• PATH
• COLORTERM
• COMMAND_MODE
• TERM
• HOME
• TMPDIR
• USER
• XPC_SERVICE_NAME
• LOGNAME
• __CF_USER_TEXT_ENCODING
• ITERM_SESSION_ID
• SHLVL
• OLDPWD
• ZSH
• PAGER
• LESS
• LSCOLORS
• PROMPT_EOL_MARK
• VIRTUAL_ENV_DISABLE_PROMPT
• LC_ALL
• LANG
• PIPENV_VENV_IN_PROJECT
• ANDROID_HOME
• VIRTUAL_ENV
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONUNBUFFERED

Pipenv–specific environment variables:

• PIPENV_VENV_IN_PROJECT: yes

Debug–specific environment variables:

• PATH: /Users/dspeck/work/test/.venv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/dspeck/Library/Android/sdk/tools:/Users/dspeck/Library/Android/sdk/platform-tools
• SHELL: /bin/zsh
• LANG: en_US.UTF-8
• PWD: /Users/dspeck/work/test
• VIRTUAL_ENV: /Users/dspeck/work/test/.venv

---

Contents of Pipfile ('/Users/dspeck/work/test/Pipfile'):

[packages]
django = ">=2.0"
drf-yasg = "*"

Expected result

pipenv lock produces Pipfile.lock

Actual result

Pipenv cannot resolve dependencies with the following error:

verbose output

.venv ❯ pipenv lock --verbose
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  django>=2.0
  drf-yasg

Finding the best candidates:
  found candidate django==2.0.4 (constraint was >=2.0)
  found candidate drf-yasg==1.6.1 (constraint was <any>)

Finding secondary dependencies:
  drf-yasg==1.6.1           requires coreapi>=2.3.3, coreapi>=2.3.3, coreschema>=0.0.4, coreschema>=0.0.4, Django<2.0,>=1.11.7; python_version <= "2.7", Django>=1.11.7; python_version >= "3.4", Django>=1.11.7; python_version >= "3.4"; python_version >= "3.4", djangorestframework>=3.7.7, djangorestframework>=3.7.7, future>=0.16.0, future>=0.16.0, inflection>=0.3.1, inflection>=0.3.1, openapi-codec>=1.3.2, openapi-codec>=1.3.2, ruamel.yaml>=0.15.34, ruamel.yaml>=0.15.34, six>=1.10.0, six>=1.10.0, uritemplate>=3.0.0, uritemplate>=3.0.0
  django==2.0.4             requires pytz

New dependencies found in this round:
  adding ['coreapi', '>=2.3.3', '[]']
  adding ['coreschema', '>=0.0.4', '[]']
  adding ['django', '<2.0,>=1.11.7', '[]']
  adding ['djangorestframework', '>=3.7.7', '[]']
  adding ['future', '>=0.16.0', '[]']
  adding ['inflection', '>=0.3.1', '[]']
  adding ['openapi-codec', '>=1.3.2', '[]']
  adding ['pytz', '', '[]']
  adding ['ruamel.yaml', '>=0.15.34', '[]']
  adding ['six', '>=1.10.0', '[]']
  adding ['uritemplate', '>=3.0.0', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  coreapi>=2.3.3
  coreschema>=0.0.4
  django<2.0,>=1.11.7,>=2.0
  djangorestframework>=3.7.7
  drf-yasg
  future>=0.16.0
  inflection>=0.3.1
  openapi-codec>=1.3.2
  pytz
  ruamel.yaml>=0.15.34
  six>=1.10.0
  uritemplate>=3.0.0

Finding the best candidates:
  found candidate coreapi==2.3.3 (constraint was >=2.3.3)
  found candidate coreschema==0.0.4 (constraint was >=0.0.4)
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  django>=2.0
  drf-yasg

Finding the best candidates:
  found candidate django==2.0.4 (constraint was >=2.0)
  found candidate drf-yasg==1.6.1 (constraint was <any>)

Finding secondary dependencies:
  django==2.0.4             requires pytz
  drf-yasg==1.6.1           requires coreapi>=2.3.3, coreapi>=2.3.3, coreschema>=0.0.4, coreschema>=0.0.4, Django<2.0,>=1.11.7; python_version <= "2.7", Django>=1.11.7; python_version >= "3.4", Django>=1.11.7; python_version >= "3.4"; python_version >= "3.4", djangorestframework>=3.7.7, djangorestframework>=3.7.7, future>=0.16.0, future>=0.16.0, inflection>=0.3.1, inflection>=0.3.1, openapi-codec>=1.3.2, openapi-codec>=1.3.2, ruamel.yaml>=0.15.34, ruamel.yaml>=0.15.34, six>=1.10.0, six>=1.10.0, uritemplate>=3.0.0, uritemplate>=3.0.0

New dependencies found in this round:
  adding ['coreapi', '>=2.3.3', '[]']
  adding ['coreschema', '>=0.0.4', '[]']
  adding ['django', '<2.0,>=1.11.7', '[]']
  adding ['djangorestframework', '>=3.7.7', '[]']
  adding ['future', '>=0.16.0', '[]']
  adding ['inflection', '>=0.3.1', '[]']
  adding ['openapi-codec', '>=1.3.2', '[]']
  adding ['pytz', '', '[]']
  adding ['ruamel.yaml', '>=0.15.34', '[]']
  adding ['six', '>=1.10.0', '[]']
  adding ['uritemplate', '>=3.0.0', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  coreapi>=2.3.3
  coreschema>=0.0.4
  django<2.0,>=1.11.7,>=2.0
  djangorestframework>=3.7.7
  drf-yasg
  future>=0.16.0
  inflection>=0.3.1
  openapi-codec>=1.3.2
  pytz
  ruamel.yaml>=0.15.34
  six>=1.10.0
  uritemplate>=3.0.0

Finding the best candidates:
  found candidate coreapi==2.3.3 (constraint was >=2.3.3)
  found candidate coreschema==0.0.4 (constraint was >=0.0.4)

INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.python.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Resetting dropped connection: pypi.python.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Resetting dropped connection: pypi.python.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Resetting dropped connection: pypi.python.org
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches django<2.0,>=1.11.7,>=2.0
Tried: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.12, 1.6, 1.6, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 1.6.6, 1.6.6, 1.6.7, 1.6.7, 1.6.8, 1.6.8, 1.6.9, 1.6.9, 1.6.10, 1.6.10, 1.6.11, 1.6.11, 1.7, 1.7, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.7.4, 1.7.4, 1.7.5, 1.7.5, 1.7.6, 1.7.6, 1.7.7, 1.7.7, 1.7.8, 1.7.8, 1.7.9, 1.7.9, 1.7.10, 1.7.10, 1.7.11, 1.7.11, 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.8, 1.8, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.8.3, 1.8.3, 1.8.4, 1.8.4, 1.8.5, 1.8.5, 1.8.6, 1.8.6, 1.8.7, 1.8.7, 1.8.8, 1.8.8, 1.8.9, 1.8.9, 1.8.10, 1.8.10, 1.8.11, 1.8.11, 1.8.12, 1.8.12, 1.8.13, 1.8.13, 1.8.14, 1.8.14, 1.8.15, 1.8.15, 1.8.16, 1.8.16, 1.8.17, 1.8.17, 1.8.18, 1.8.18, 1.8.19, 1.8.19, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.2, 1.9.2, 1.9.3, 1.9.3, 1.9.4, 1.9.4, 1.9.5, 1.9.5, 1.9.6, 1.9.6, 1.9.7, 1.9.7, 1.9.8, 1.9.8, 1.9.9, 1.9.9, 1.9.10, 1.9.10, 1.9.11, 1.9.11, 1.9.12, 1.9.12, 1.9.13, 1.9.13, 1.10a1, 1.10a1, 1.10b1, 1.10b1, 1.10rc1, 1.10rc1, 1.10, 1.10, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.10.3, 1.10.3, 1.10.4, 1.10.4, 1.10.5, 1.10.5, 1.10.6, 1.10.6, 1.10.7, 1.10.7, 1.10.8, 1.10.8, 1.11a1, 1.11b1, 1.11rc1, 1.11rc1, 1.11, 1.11, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.11.4, 1.11.4, 1.11.5, 1.11.5, 1.11.6, 1.11.6, 1.11.7, 1.11.7, 1.11.8, 1.11.8, 1.11.9, 1.11.9, 1.11.10, 1.11.10, 1.11.11, 1.11.11, 1.11.12, 1.11.12, 2.0a1, 2.0b1, 2.0rc1, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.2, 2.0.2, 2.0.3, 2.0.3, 2.0.4, 2.0.4
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.python.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Resetting dropped connection: pypi.python.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Resetting dropped connection: pypi.python.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Resetting dropped connection: pypi.python.org
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches django<2.0,>=1.11.7,>=2.0
Tried: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.12, 1.6, 1.6, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 1.6.6, 1.6.6, 1.6.7, 1.6.7, 1.6.8, 1.6.8, 1.6.9, 1.6.9, 1.6.10, 1.6.10, 1.6.11, 1.6.11, 1.7, 1.7, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.7.4, 1.7.4, 1.7.5, 1.7.5, 1.7.6, 1.7.6, 1.7.7, 1.7.7, 1.7.8, 1.7.8, 1.7.9, 1.7.9, 1.7.10, 1.7.10, 1.7.11, 1.7.11, 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.8, 1.8, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.8.3, 1.8.3, 1.8.4, 1.8.4, 1.8.5, 1.8.5, 1.8.6, 1.8.6, 1.8.7, 1.8.7, 1.8.8, 1.8.8, 1.8.9, 1.8.9, 1.8.10, 1.8.10, 1.8.11, 1.8.11, 1.8.12, 1.8.12, 1.8.13, 1.8.13, 1.8.14, 1.8.14, 1.8.15, 1.8.15, 1.8.16, 1.8.16, 1.8.17, 1.8.17, 1.8.18, 1.8.18, 1.8.19, 1.8.19, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.2, 1.9.2, 1.9.3, 1.9.3, 1.9.4, 1.9.4, 1.9.5, 1.9.5, 1.9.6, 1.9.6, 1.9.7, 1.9.7, 1.9.8, 1.9.8, 1.9.9, 1.9.9, 1.9.10, 1.9.10, 1.9.11, 1.9.11, 1.9.12, 1.9.12, 1.9.13, 1.9.13, 1.10a1, 1.10a1, 1.10b1, 1.10b1, 1.10rc1, 1.10rc1, 1.10, 1.10, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.10.3, 1.10.3, 1.10.4, 1.10.4, 1.10.5, 1.10.5, 1.10.6, 1.10.6, 1.10.7, 1.10.7, 1.10.8, 1.10.8, 1.11a1, 1.11b1, 1.11rc1, 1.11rc1, 1.11, 1.11, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.11.4, 1.11.4, 1.11.5, 1.11.5, 1.11.6, 1.11.6, 1.11.7, 1.11.7, 1.11.8, 1.11.8, 1.11.9, 1.11.9, 1.11.10, 1.11.10, 1.11.11, 1.11.11, 1.11.12, 1.11.12, 2.0a1, 2.0b1, 2.0rc1, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.2, 2.0.2, 2.0.3, 2.0.3, 2.0.4, 2.0.4

Steps to replicate

Run pipenv lock with the Pipfile provided above.

[techalchemy]

Pipenv uses whichever thing is called python and has precedence unless you tell it specifically which python to create the environment with or you set PIPENV_DEFAULT_PYTHON_VERSION. Use pipenv install --python=3.6 or pipenv install --three or pipenv install --python=/usr/local/bin/python3.6

[dspechnikov]

@techalchemy as you can see from pipenv.help output, the thing called python in my virtualenv is actually python3.6. Also, options you suggested didn't work. Pipenv still somehow uses python 2.7.

Can you please reopen the issue until we sort it out?

[techalchemy]

@dspechnikov oooh yes, okay, I see the bug here:

drf-yasg==1.6.1    Django<2.0,>=1.11.7; python_version <= "2.7", Django>=1.11.7; python_version >= "3.4",...

We are currently picking up both of these... I'm not sure lockfiles currently actually have a way to handle this in the spec (duplicate entries would be the only way I guess right now? or flattening the markers and including all hashes found?)

Otherwise we can just pick the one that matches the specified python version and resolve that one only as a fix -- @ncoghlan @jtratner @uranusjr @frostming -- thoughts? We have markers for 2 different python versions & a different specifier for each, what approach makes the most sense?

[uranusjr]

I didn’t even know this was possible. People really can get creative with requirement specification. Flattening + merging hashes would cause a problem with the version key, I think? The only non-hack solution I can think of is to turn the dependency content into a list

{
    "django": [
        {
            "hashes": ["..."],
            "markers": "python_version <= '2.7'",
            "version": "1.11.7"
        },
        {
            "hashes": ["..."],
            "markers": "python_version >= '3.4'",
            "version": "2.0.3"
        }
    ]
}

[frostming]

So we should allow duplicate entries with different markers in Pipfile too. (which is not supported by my previous commit)

[ncoghlan]

Huh, I thought the lockfile generation had successfully been changed to always use the venv Python, so I'm surprised it was picking up both markers.

I suspect that longer term pipenv's going to have to carry environment markers all the way through to the lock file, but for this particular issue, there may be a simpler solution based on ignoring markers that the target environment doesn't meet.

After all, pipenv's current target scope of use is specifically "known target deployment environment", as even getting that working reliably is a relatively complex task.

[techalchemy]

I like @uranusjr’s suggestion but in the short term let’s just pick the one that works in the specific environment

[techalchemy]

@ncoghlan that’s correct although if (as in this case) someone includes all markers on the same line, that’s the line we pick up. We just split the markers up. That makes me think — maybe we don’t have to do that

[techalchemy]

I can't actually reproduce this on the current version of pipenv; maybe my marker fix caught it. @dspechnikov can you confirm that you are still seeing this

[techalchemy]

Hm yeah this isn't happening with the latest released version of pipenv

[dspechnikov]

@techalchemy what do you mean by current version? 11.10.0? If so, then my initial report was already for that version (see pipenv.help output). I tried everything (clean installation, different version specifiers, etc.), but the problem remains and I can't find any workarounds :(

Can I somehow help to reproduce it? I really want this to be fixed.

[techalchemy]

Are you sure though? Is it possible you have multiple pipenv installations? Try to completely purge the one that’s on your path with python -m pip uninstall pipenv and see if you still have one. Keep doing that until you are out of pipenvs. Then (outside of a virtualenv) python3.6 -m pip install —upgrade —no-cache-dir —user pipenv and make sure ~/.local/bin is on your path. Let me know if this helps at all, if not we will have to revisit other solutions

[techalchemy]

As an additional FYI we don’t recommend putting secondary dependencies in your pipfile. I.e. django is a dependency of django-rest-framework and drf-yasg already so you shouldn’t put it in your pipfile in the first place.

[dspechnikov]

@techalchemy uninstalling pipenv didn't help, but clearing the dependency cache did! Thanks a lot for the help everyone, I'll remember to clear the cache after upgrade just in case :)

[techalchemy]

@dspechnikov awesome, I didn't even recommend that for some reason but I'm glad you stumbled across it :clap: