Pipenv: Incorrect index in lockfile

Created on 18 Nov 2018 · 7Comments · Source: pypa/pipenv

Issue description

When using a custom index for a package that is also on PyPI, but with different versions, it uses the custom index for the version (which is the higher version, so this is correct), but the lock file tags it as coming from the PyPI index. The version that is locked does not exist on the pypi index, and the PyPI versions do not exist on the custom index.

Expected result

I expected it to get the latest version, and annotate the lockfile with the correct index that the version was retrieved from.

Actual result

Locking works cleanly, but produced the odd output in the lockfile.

Steps to replicate

Use this Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "aspiredu"
verify_ssl = true
url = "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/"

[dev-packages]

[packages]
django-enumfields = "*"

[requires]
python_version = "3.7"

to lock and get this Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "8e1f91149b67ec93c8a959eb66c89860a7dcfba35b07c058890607a0bc16eb7c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "aspiredu",
                "url": "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django-enumfields": {
            "hashes": [
                "sha256:b32faacf10ac20201dc2910a5f440d8132e7cae80e689e14a97bd1394e3a8dd4"
            ],
            "index": "pypi",
            "version": "==0.10.1+aspiredu1"
        }
    },
    "develop": {}
}

$ pipenv --support

Pipenv version: '2018.11.14'

Pipenv location: '/home/ryan/.local/lib/python3.6/site-packages/pipenv'

Python location: '/home/ryan/.pyenv/versions/3.6.6/bin/python3.6'

Python installations found:

3.7.1: /home/ryan/.pyenv/versions/3.7.1/bin/python3
3.7.1: /home/ryan/.pyenv/versions/3.7.1/bin/python3.7m
3.7.0: /home/ryan/.pyenv/versions/3.7.0/bin/python3
3.7.0: /home/ryan/.pyenv/versions/3.7.0/bin/python3.7m
3.6.7: /home/ryan/.pyenv/versions/3.6.7/bin/python3
3.6.7: /home/ryan/.pyenv/versions/3.6.7/bin/python3.6m
3.6.6: /home/ryan/.pyenv/versions/3.6.6/bin/python3
3.6.6: /home/ryan/.pyenv/versions/3.6.6/bin/python3.6m
3.6.6: /usr/bin/python3
3.6.6: /usr/bin/python3.6m
2.7.15rc1: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-38-generic',
 'platform_system': 'Linux',
 'platform_version': '#41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

CLUTTER_IM_MODULE
LS_COLORS
LESSCLOSE
XDG_MENU_PREFIX
LANG
GDM_LANG
DISPLAY
QT_STYLE_OVERRIDE
COLORTERM
PYENV_VIRTUALENV_INIT
XDG_VTNR
SSH_AUTH_SOCK
MANDATORY_PATH
XDG_SESSION_ID
XDG_GREETER_DATA_DIR
USER
DESKTOP_SESSION
QT4_IM_MODULE
TEXTDOMAINDIR
GNOME_TERMINAL_SCREEN
DEFAULTS_PATH
QT_QPA_PLATFORMTHEME
PWD
HOME
TEXTDOMAIN
SSH_AGENT_PID
QT_ACCESSIBILITY
XDG_SESSION_TYPE
XDG_DATA_DIRS
XDG_SESSION_DESKTOP
GTK_MODULES
TERM
SHELL
VTE_VERSION
XDG_SEAT_PATH
QT_IM_MODULE
XMODIFIERS
IM_CONFIG_PHASE
XDG_CURRENT_DESKTOP
GPG_AGENT_INFO
GNOME_TERMINAL_SERVICE
XDG_SEAT
SHLVL
PYENV_SHELL
LANGUAGE
GDMSESSION
GNOME_DESKTOP_SESSION_ID
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_SESSION_PATH
XDG_CONFIG_DIRS
PATH
SESSION_MANAGER
LESSOPEN
GTK_IM_MODULE
OLDPWD
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/ryan/bin:/home/ryan/.local/bin:/home/ryan/.pyenv/plugins/pyenv-virtualenv/shims:/home/ryan/.pyenv/shims:/home/ryan/.pyenv/bin:/home/ryan/.local/bin:/home/ryan/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/ryan/Code/aspiredu/pipenv-indices

Contents of Pipfile ('/home/ryan/Code/aspiredu/pipenv-indices/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "aspiredu"
verify_ssl = true
url = "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/"

[dev-packages]

[packages]
django-enumfields = "*"

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/home/ryan/Code/aspiredu/pipenv-indices/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "8e1f91149b67ec93c8a959eb66c89860a7dcfba35b07c058890607a0bc16eb7c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "aspiredu",
                "url": "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django-enumfields": {
            "hashes": [
                "sha256:b32faacf10ac20201dc2910a5f440d8132e7cae80e689e14a97bd1394e3a8dd4"
            ],
            "index": "pypi",
            "version": "==0.10.1+aspiredu1"
        }
    },
    "develop": {}
}

Dependency Resolution Needs More Information Type

Source

ryanhiebert

👍5

Most helpful comment

I'm also seeing this problem. Here's a bit more info:

Our project Pipfiles have two sources at the top:

[[source]]
url = "https://our.jenkins.server/job/wheelhouse-index/Indexes/16.04/simple/"
verify_ssl = true
name = "our_wheelhouse"

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

Our private wheelhouse holds closed-source projects that we don't release publicly. When we run pipenv lock on a project, _every dependency in the Pipfile.lock_ ends up having "index": "our_wheelhouse" in it, including all the open source projects that actually come from PyPi. This is despite the fact that we don't mirror anything in our private wheelhouse: It contains _only_ our closed-source projects.

The order of the [[source]]s does not matter. If we put PyPi first, our private wheelhouse is still listed as the index for every dependency, even those on PyPi.

Interestingly, it doesn't seem to affect anything. Future installs using the lock file still work fine. There are no errors. But this is clearly not accurate data in the lock file, and I'm concerned that, in future pipenv versions, it _could_ break something.

nickwilliams-eventbrite on 30 Nov 2018

👍2

All 7 comments

It just lists pypi as the index specifically because it locked it from the alternate index despite the fact that pypi is listed first (i.e. is the primary index), so it knows it should check there first

I am curious to know if this causes issues though, I'm not sure we thought this through much

techalchemy on 19 Nov 2018

My thought process is that if it's in the lockfile, it's because it's part of what needs to be considered as locked. If we don't actually care which index it comes from, should it even be in the lockfile?

ryanhiebert on 19 Nov 2018

I do sometimes have issues with multiple indexes where the index chosen to install from first doesn't have the package, so it does seem to me that the index being specified is important to get correct.

ryanhiebert on 19 Nov 2018

I'm also seeing this problem. Here's a bit more info:

Our project Pipfiles have two sources at the top:

[[source]]
url = "https://our.jenkins.server/job/wheelhouse-index/Indexes/16.04/simple/"
verify_ssl = true
name = "our_wheelhouse"

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

The order of the [[source]]s does not matter. If we put PyPi first, our private wheelhouse is still listed as the index for every dependency, even those on PyPi.

nickwilliams-eventbrite on 30 Nov 2018

👍2

That's pretty much the way it's happening for me as well, except that, for reasons I don't understand, it doesn't lock everything to the internal index, some things get set to PyPI, and some to the internal index. I don't have a clue how it chooses, and it's very commonly incorrect.

I also don't see it affecting the installs for the most part, except in the case I describe in #3250, which is related to the extent that it is an example of the index specified in the lockfile being ignored during the install.

ryanhiebert on 4 Dec 2018

I have the exact same behavior as @nickwilliams-eventbrite . Is this a problem? is this normal? What does this index entry actually means (subdependencies do not have it)?

erdnaxeli on 19 Jun 2019

Just come across this problem.

In my case, it appears to be a bit random. Some packages from pypi are being attributed to my private (mainly containing prebuilt wheels for our environment) repository but not all.

The main side effect is the install stage takes longer as packages will initially fail to install, but are then subsequently installed when they are retried (presumably using the other repository listed).

While the outcome is all expected packages are installed, it takes longer than required and causes a few double takes when reading through the lock file seeing packages that I know are not in the private repository listed. Our junior developer was confused by the issue when they first encountered it.