Pipenv: incorrect dependencies (versions) locked by pip

[jerzyk]

tried to add flower to the project, but pipenv lock is picking newest tornado version (5.1) instead one specified in requirements (tornado>=4.2.0,<5.0.0) (tested on 2.7.15 & 3.7)

Expected result

proper tornado version to be picked

Steps to replicate

$ mkdir tmp1
$ cd tmp1
$ pipenv install flower

---

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/Users/jerzyk/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pipenv'

Python location: '/Users/jerzyk/.pyenv/versions/3.7.0/bin/python3.7'

Other Python installations in PATH:

• 2.7: /usr/local/bin/python2.7
• 2.7: /Users/jerzyk/.pyenv/shims/python2.7
• 2.7: /usr/local/bin/python2.7
• 2.7: /usr/local/bin/python2.7
• 2.7: /usr/bin/python2.7
• 3.6: /Users/jerzyk/.pyenv/shims/python3.6
• 3.7: /Users/jerzyk/.pyenv/versions/3.7.0/bin/python3.7m
• 3.7: /Users/jerzyk/.pyenv/versions/3.7.0/bin/python3.7
• 3.7: /Users/jerzyk/.pyenv/shims/python3.7
• 3.7: /usr/local/bin/python3.7

• 3.7: /usr/local/bin/python3.7

• 3.7.0: /Users/jerzyk/.pyenv/versions/3.7.0/bin/python

• 3.7.0: /Users/jerzyk/.pyenv/shims/python
• 2.7.15: /usr/local/bin/python
• 2.7.15: /usr/local/bin/python
• 2.7.10: /usr/bin/python
• None: /Users/jerzyk/.pyenv/shims/python2
• 2.7.15: /usr/local/bin/python2
• 2.7.15: /usr/local/bin/python2
• 3.7.0: /Users/jerzyk/.pyenv/versions/3.7.0/bin/python3
• 3.7.0: /Users/jerzyk/.pyenv/shims/python3
• 3.7.0: /usr/local/bin/python3
• 3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT '
                     '2018; root:xnu-4570.71.2~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_PROGRAM
• PYENV_ROOT
• TERM
• SHELL
• TMPDIR
• Apple_PubSub_Socket_Render
• TERM_PROGRAM_VERSION
• TERM_SESSION_ID
• PYENV_VERSION
• USER
• SSH_AUTH_SOCK
• PYENV_DIR
• __CF_USER_TEXT_ENCODING
• PATH
• PWD
• LANG
• PGHOST
• ITERM_PROFILE
• PYENV_HOOK_PATH
• XPC_FLAGS
• XPC_SERVICE_NAME
• SHLVL
• HOME
• COLORFGBG
• ITERM_SESSION_ID
• LOGNAME
• PGDATA
• DISPLAY
• COLORTERM
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /Users/jerzyk/.pyenv/versions/3.7.0/bin:/usr/local/Cellar/pyenv/1.2.6/libexec:/Users/jerzyk/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/Applications/Xcode.app/Contents/Developer/usr/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/MacGPG2/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /Users/jerzyk/tmp

---

Contents of Pipfile ('/Users/jerzyk/tmp/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
flower = "*"

[dev-packages]

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/Users/jerzyk/tmp/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "e224f38cf9f41ab210a6546986368cfee3a9087dc1c0d756024e2f6aea1f7e9a"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "amqp": {
            "hashes": [
                "sha256:073dd02fdd73041bffc913b767866015147b61f2a9bc104daef172fc1a0066eb",
                "sha256:eed41946890cd43e8dee44a316b85cf6fee5a1a34bb4a562b660a358eb529e1b"
            ],
            "version": "==2.3.2"
        },
        "babel": {
            "hashes": [
                "sha256:6778d85147d5d85345c14a26aada5e478ab04e39b078b0745ee6870c2b5cf669",
                "sha256:8cba50f48c529ca3fa18cf81fa9403be176d374ac4d60738b839122dfaaa3d23"
            ],
            "version": "==2.6.0"
        },
        "billiard": {
            "hashes": [
                "sha256:ed65448da5877b5558f19d2f7f11f8355ea76b3e63e1c0a6059f47cfae5f1c84"
            ],
            "version": "==3.5.0.4"
        },
        "celery": {
            "hashes": [
                "sha256:77dab4677e24dc654d42dfbdfed65fa760455b6bb563a0877ecc35f4cfcfc678",
                "sha256:ad7a7411772b80a4d6c64f2f7f723200e39fb66cf614a7fdfab76d345acc7b13"
            ],
            "version": "==4.2.1"
        },
        "flower": {
            "hashes": [
                "sha256:a7a828c2dbea7e9cff1c86d63626f0eeb047b1b1e9a0ee5daad30771fb51e6d0"
            ],
            "index": "pypi",
            "version": "==0.9.2"
        },
        "kombu": {
            "hashes": [
                "sha256:86adec6c60f63124e2082ea8481bbe4ebe04fde8ebed32c177c7f0cd2c1c9082",
                "sha256:b274db3a4eacc4789aeb24e1de3e460586db7c4fc8610f7adcc7a3a1709a60af"
            ],
            "version": "==4.2.1"
        },
        "pytz": {
            "hashes": [
                "sha256:a061aa0a9e06881eb8b3b2b43f05b9439d6583c206d0a6c340ff72a7b6669053",
                "sha256:ffb9ef1de172603304d9d2819af6f5ece76f2e85ec10692a524dd876e72bf277"
            ],
            "version": "==2018.5"
        },
        "tornado": {
            "hashes": [
                "sha256:1c0816fc32b7d31b98781bd8ebc7a9726d7dce67407dc353a2e66e697e138448",
                "sha256:4f66a2172cb947387193ca4c2c3e19131f1c70fa8be470ddbbd9317fd0801582",
                "sha256:5327ba1a6c694e0149e7d9126426b3704b1d9d520852a3e4aa9fc8fe989e4046",
                "sha256:6a7e8657618268bb007646b9eae7661d0b57f13efc94faa33cd2588eae5912c9",
                "sha256:a9b14804783a1d77c0bd6c66f7a9b1196cbddfbdf8bceb64683c5ae60bd1ec6f",
                "sha256:c58757e37c4a3172949c99099d4d5106e4d7b63aa0617f9bb24bfbff712c7866",
                "sha256:d8984742ce86c0855cccecd5c6f54a9f7532c983947cff06f3a0e2115b47f85c"
            ],
            "markers": "python_version != '3.2.*' and python_version != '3.1.*' and python_version != '3.3.*' and python_version != '3.0.*' and python_version >= '2.7'",
            "version": "==5.1"
        },
        "vine": {
            "hashes": [
                "sha256:52116d59bc45392af9fdd3b75ed98ae48a93e822cee21e5fda249105c59a7a72",
                "sha256:6849544be74ec3638e84d90bc1cf2e1e9224cc10d96cd4383ec3f69e9bce077b"
            ],
            "version": "==1.1.4"
        }
    },
    "develop": {}
}

[r-darwish]

```[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
flower = "*"

[dev-packages]
tornado = "<5"

[requires]
python_version = "3.7"
```

This file reproduces the problem. pipenv lock seems to lock packages and dev-packages in different stages. When it finds out that flower depends on tornado>=4.2.0 is ignores the constraint defined in dev-packages and installs tornado 5. Running pipenv lock -d doesn't help either.

[mizi]

As a workaround, I had to move all [dev-packages] requirements to [packages]. Or I guess I could've duplicated all [packages] entries in [dev-packages].

[techalchemy]

Sorry for the delay getting back to you all, to answer at least one part of the inquiry, dev-packages take a back seat to packages to prevent developing accidentally in an environment that won't mirror whatever you deploy.

Since you have a top level package here that depends on tornado (flower), and a strict constraint on what version of tornado is OK to install, it is pretty likely you would want to pin the tornado entry in packages since it's being installed either way. If the version pin is conditional, you should consider adding an environment marker.

Roughly speaking, this Pipfile says you will be using tornado 5.x in production, but <5 in development, which is something we designed around (i.e. this was a design decision -- unless you use environment markers your local dependencies will match the ones in production no matter what).

Closing for now as this particular dependency resolution issue is a design decision, you'll have to move your requirement into the packages section and use environment markers