Pipenv: Using VCS dependencies throws KeyError when writing Pipfile.lock

Created on 21 Dec 2018  Â·  14Comments  Â·  Source: pypa/pipenv

When converting an existing project from requirements.txt to use pipenv, I performed a pipenv install, which created the following Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
boto = "==2.36.0"
boto3 = "==1.4.5"
braintree = "==3.37.2"
cachetools = "==1.1.6"
celery = "==4.2.1"
certifi = "*"
cffi = "==1.5.2"
cloudinary = "==1.5.0"
coverage = "==4.0.3"
customerio = "==0.1.10"
dj-static = "==0.0.6"
django-ace = "==1.0.2"
django-admin-sortable = "==2.0.20"
django-admin-tools = "==0.8.1"
django-admin-view-permission = "==1.8"
django-allauth = "==0.32.0"
django-annoying = "==0.8.7"
django-cors-headers = "==2.0.2"
django-dbsettings = {ref = "2eb24b58b50c3d47c185f09c17b99c36a4ec2d76",git = "git://github.com/Anton-Shutik/django-dbsettings.git",editable = true}
django-debug-toolbar = "==1.6"
django-filter = "==1.0.2"
django-mailgun = "==0.8.0"
django-nose = "==1.4.4"
django-oauth-toolkit = "==0.10.0"
django-pylibmc = "==0.6.1"
django-reversion = "==1.10.1"
django-seo-js = {ref = "9a0db3856c2647fe6a09dd0cf4103f9f8b1740f5",git = "git://github.com/groveco/django-seo-js.git",editable = true}
django-storages = "==1.6.5"
djangorestframework = "==3.8.2"
djangorestframework-csv = "==1.4.1"
djangorestframework-jsonapi = "==2.6.0"
django-watson = "==1.3.1"
drf-extensions = "==0.3.1"
freezegun = "==0.3.6"
elasticsearch = "<7.0.0,>=6.0.0"
elasticsearch-dsl = "==6.1.0"
gunicorn = "==19.4.5"
hasoffers = "==0.2.1"
jsmin = "==2.2.0"
kombu = "==4.2.0"
librato-metrics = "==0.8.6"
lxml = "==3.5.0"
mailchimp = "==2.0.9"
mock = "==1.3.0"
psycopg2 = "==2.7.3.2"
pylibmc = "==1.5.1"
pytz = "==2018.5"
raven = "==5.32.0"
redis = "==2.10.5"
requests-aws4auth = "==0.9"
requests = "==2.9.1"
selenium = "==3.3.1"
sauceclient = "==0.2.1"
slacker = "==0.9.9"
sqlparse = "==0.2.2"
stripe = "==1.56.0"
tinys3 = "==0.1.11"
user-agents = "==1.1.0"
wsgiref = "==0.1.2"
zenpy = "==2.0.7"
django-sprinklers = {editable = true,git = "https://github.com/chrisclark/django-sprinklers",ref = "e53053f53e0c5a3ff45a3c6c6b8c70af374854b1"}
django-segments = {ref = "00c9591ab72b099b567f855ca0c3b86367164910",git = "https://github.com/groveco/django-segments",editable = true}
sailthru-client = "==2.3.1"
nose-exclude = "==0.5.0"
ddtrace = "==0.12.0"
zeep = "==2.4.0"
flake8 = "==3.5.0"
urllib3 = "==1.22"
Avalara = "==18.5.2"
Django = "==1.11.6"
Beaker = "==1.9.0"
factory_boy = "==2.8.1"
django_celery_beat = "==1.3.0"
Pillow = "==3.3.1"
rest_condition = "==1.0.3"
XlsxWriter = "==0.8.5"
nose_ittr = "==0.0.4"
Explorer = {ref = "7373383598ce9bba2c6e43e3d32342fef11035ba",git = "https://github.com/groveco/django-sql-explorer",editable = true}

[requires]
python_version = "2.7"

Then pipenv attempts to generate the Pipenv.lock file and generates the following error:

Locking [dev-packages] dependencies…
Locking [packages] dependencies…
âś” Success!
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==2018.11.26', 'console_scripts', 'pipenv')()
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/cli/command.py", line 332, in lock
    pypi_mirror=state.pypi_mirror,
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1068, in do_lock
    lockfile=lockfile
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 682, in venv_resolve_deps
    lockfile[lockfile_section][k].update(v)
KeyError: 'idna'

I have run pipenv lock --clear multiple times and have gotten different KeyError failures for chardet as well, but mostly for idna. The stack trace is the same, regardless of the key that fails the lookup. When I used pipenv install --skip-lock, all the dependencies installed except for one that needed native headers from libmemcached. The error message that pip reported from that failure was very different and Google-able, so I was able to resolve and get it installed. But I still have no Pipfile.lock and pipenv lock --clear any number of times just generates the same error as above. Help?

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.11.26/libexec/bin/python3.7'

Python installations found:

  • 3.7.1: /usr/local/bin/python3
  • 3.7.1: /usr/local/bin/python3.7m
  • 2.7.15: /Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov/bin/python
  • 2.7.15: /usr/local/bin/python
  • 2.7.15: /usr/local/bin/pythonw
  • 2.7.10: /usr/bin/python
  • 2.7.10: /usr/bin/pythonw
  • 2.7.10: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.1',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '18.2.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST '
                     '2018; root:xnu-4903.231.4~2/RELEASE_X86_64',
 'python_full_version': '3.7.1',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

  • PATH
  • GREP_COLOR
  • LESS_TERMCAP_mb
  • TERM_PROGRAM
  • LESS_TERMCAP_md
  • NVM_CD_FLAGS
  • LESS_TERMCAP_me
  • PIP_PYTHON_PATH
  • SHELL
  • TERM
  • TMPDIR
  • Apple_PubSub_Socket_Render
  • TERM_PROGRAM_VERSION
  • DIRENV_DIR
  • TERM_SESSION_ID
  • LESS_TERMCAP_ue
  • NVM_DIR
  • USER
  • LS_COLORS
  • GREP_COLORS
  • RPROMPT
  • SSH_AUTH_SOCK
  • __CF_USER_TEXT_ENCODING
  • VIRTUAL_ENV
  • VIRTUAL_ENV_DISABLE_PROMPT
  • PAGER
  • WORKON_HOME
  • TMUX
  • DIRENV_WATCHES
  • LESS_TERMCAP_us
  • VIRTUALENVWRAPPER_PYTHON
  • PIPENV_ACTIVE
  • _git_log_oneline_format
  • _
  • PWD
  • EDITOR
  • LANG
  • TMUX_PANE
  • XPC_FLAGS
  • PYTHONDONTWRITEBYTECODE
  • XPC_SERVICE_NAME
  • HOME
  • SHLVL
  • LESS
  • LOGNAME
  • PIP_DISABLE_PIP_VERSION_CHECK
  • VISUAL
  • LESS_TERMCAP_so
  • TMUX_PLUGIN_MANAGER_PATH
  • NVM_BIN
  • BROWSER
  • _git_log_medium_format
  • DIRENV_DIFF
  • SECURITYSESSIONID
  • LESS_TERMCAP_se
  • PIP_SHIMS_BASE_MODULE
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

  • PIPENV_ACTIVE: 1

Debug–specific environment variables:

  • PATH: /usr/local/Cellar/pipenv/2018.11.26/libexec/tools:/usr/local/opt/nvm/versions/node/v10.4.1/bin:/Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov/bin:/Users/al_the_x/bin:/usr/local/Cellar/pipenv/2018.11.26/libexec/tools:/Users/al_the_x/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin
  • SHELL: /usr/local/bin/zsh
  • EDITOR: /usr/local/bin/nvim
  • LANG: en_US.UTF-8
  • PWD: /Users/al_the_x/Projects/grove.co
  • VIRTUAL_ENV: /Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov

Contents of Pipfile ('/Users/al_the_x/Projects/grove.co/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
boto = "==2.36.0"
boto3 = "==1.4.5"
braintree = "==3.37.2"
cachetools = "==1.1.6"
celery = "==4.2.1"
certifi = "*"
cffi = "==1.5.2"
cloudinary = "==1.5.0"
coverage = "==4.0.3"
customerio = "==0.1.10"
dj-static = "==0.0.6"
django-ace = "==1.0.2"
django-admin-sortable = "==2.0.20"
django-admin-tools = "==0.8.1"
django-admin-view-permission = "==1.8"
django-allauth = "==0.32.0"
django-annoying = "==0.8.7"
django-cors-headers = "==2.0.2"
django-dbsettings = {ref = "2eb24b58b50c3d47c185f09c17b99c36a4ec2d76",git = "git://github.com/Anton-Shutik/django-dbsettings.git",editable = true}
django-debug-toolbar = "==1.6"
django-filter = "==1.0.2"
django-mailgun = "==0.8.0"
django-nose = "==1.4.4"
django-oauth-toolkit = "==0.10.0"
django-pylibmc = "==0.6.1"
django-reversion = "==1.10.1"
django-seo-js = {ref = "9a0db3856c2647fe6a09dd0cf4103f9f8b1740f5",git = "git://github.com/groveco/django-seo-js.git",editable = true}
django-storages = "==1.6.5"
djangorestframework = "==3.8.2"
djangorestframework-csv = "==1.4.1"
djangorestframework-jsonapi = "==2.6.0"
django-watson = "==1.3.1"
drf-extensions = "==0.3.1"
freezegun = "==0.3.6"
elasticsearch = "<7.0.0,>=6.0.0"
elasticsearch-dsl = "==6.1.0"
gunicorn = "==19.4.5"
hasoffers = "==0.2.1"
jsmin = "==2.2.0"
kombu = "==4.2.0"
librato-metrics = "==0.8.6"
lxml = "==3.5.0"
mailchimp = "==2.0.9"
mock = "==1.3.0"
psycopg2 = "==2.7.3.2"
pylibmc = "==1.5.1"
pytz = "==2018.5"
raven = "==5.32.0"
redis = "==2.10.5"
requests-aws4auth = "==0.9"
requests = "==2.9.1"
selenium = "==3.3.1"
sauceclient = "==0.2.1"
slacker = "==0.9.9"
sqlparse = "==0.2.2"
stripe = "==1.56.0"
tinys3 = "==0.1.11"
user-agents = "==1.1.0"
wsgiref = "==0.1.2"
zenpy = "==2.0.7"
django-sprinklers = {editable = true,git = "https://github.com/chrisclark/django-sprinklers",ref = "e53053f53e0c5a3ff45a3c6c6b8c70af374854b1"}
django-segments = {ref = "00c9591ab72b099b567f855ca0c3b86367164910",git = "https://github.com/groveco/django-segments",editable = true}
sailthru-client = "==2.3.1"
nose-exclude = "==0.5.0"
ddtrace = "==0.12.0"
zeep = "==2.4.0"
flake8 = "==3.5.0"
urllib3 = "==1.22"
Avalara = "==18.5.2"
Django = "==1.11.6"
Beaker = "==1.9.0"
factory_boy = "==2.8.1"
django_celery_beat = "==1.3.0"
Pillow = "==3.3.1"
rest_condition = "==1.0.3"
XlsxWriter = "==0.8.5"
nose_ittr = "==0.0.4"
Explorer = {ref = "7373383598ce9bba2c6e43e3d32342fef11035ba",git = "https://github.com/groveco/django-sql-explorer",editable = true}

[requires]
python_version = "2.7"

VCS Type Vendored Dependencies
Source
picture al-the-x
👍5

Most helpful comment

@amureki Adding the dependency on which the KeyError happens on to our dependency list worked for us. (A temporary fix, of course.)

picture GautamGupta on 11 Apr 2019
👍4

All 14 comments

I'm also seeing this issue and trying to get more info to help with debugging here. The KeyError happens on line 693 here in on the current master branch:
https://github.com/pypa/pipenv/blob/3b3ebb47526f6b15ff61294452e1323e309fee8f/pipenv/utils.py#L692-L693
The condition before even reads ... or k not in lockfile[lockfile_section], i.e. it seems that it's somewhat expected that k, which is the name of a dependency here, might not be in lockfile[lockfile_section] yet, but then the next line tries to read it anyway? Even though we just checked that it's not in there? 🤔

If I'm interpreting this code correctly, vcs_lockfile contains the dependencies of the latest versions of my vcs dependencies, so it seems to be stumbling over one of my git dependencies having added a dependency in a later version than I have installed.

teeberg picture teeberg on 30 Dec 2018

I think you're right @teeberg. Here's the commit where that was intro'd:

https://github.com/pypa/pipenv/commit/17a9c3fae4991c468891b0354d035fc070ebd847#diff-4b9d1c4c6c693a89e0cd2e6d9ca27631R678

It seems like that predicate is wrong. I edited my locally installed pipenv/utils.py to invert the predicate, and pipenv install and pipenv lock --clear both produced a consumable Pipfile.lock with the VCS deps properly checked out as editable packages as confirmed by pipenv graph and pip freeze within the pipenv shell. I'll annotate the commit message and link it to this issue.

al-the-x picture al-the-x on 2 Jan 2019

Per the commit comment linked above:

-         if k in getattr(project, vcs_section, {}) or k not in lockfile[lockfile_section]:
+         if k in getattr(project, vcs_section, {}) and k in lockfile[lockfile_section]:

Does resolve the problem with pipenv lock for me. I'll look around for a corresponding test...

al-the-x picture al-the-x on 2 Jan 2019

That fixed it for me too! :)

teeberg picture teeberg on 2 Jan 2019

In the case of your change, the second predicate after the or would never be tested, I think. The vcs_lockfile is generated from project[vcs_section], if I'm reading that correctly, so each k in vcs_lockfile.items() _must_ exist in project[vcs_section]... Maybe the update line isn't even necessary? Or maybe it should be a conditional update if the key exists and an assignment if it does not? Like:

    for k, v in vcs_lockfile.items():
        if k in getattr(project, vcs_section, {}):
            if k in lockfile[lockfile_section] lockfile[lockfile_section][k].update(v)
            else lockfile[lockfile_section][k] = v

Someone might be able to code-golf or Pythonify that for me. It would also be helpful to get @techalchemy's take on the above...

al-the-x picture al-the-x on 2 Jan 2019

I wrote @techalchemy via the email address that he lists on GitHub. Hopefully he can weigh in.

al-the-x picture al-the-x on 11 Jan 2019

Thanks for checking in on my line note @techalchemy. Happy to submit a PR for those changes, but they seem pretty small if your prefer to tackle them.

al-the-x picture al-the-x on 21 Jan 2019

Most likely I’ll catch them in my current net of changes. I have two rounds of fixes to get working before I start crossing out the small wins and open PRs. I took a long break so my dream of handling it in a weekend wasn’t to be

techalchemy picture techalchemy on 22 Jan 2019

No worries. Best of luck.

al-the-x picture al-the-x on 22 Jan 2019

I have the same error, the only difference is that I simply install a private package with pipenv install -e [email protected]/
It fails in Finding secondary dependencies: after Result of round 8: stable, done

I patched utils.py as was suggested in https://github.com/pypa/pipenv/commit/17a9c3fae4991c468891b0354d035fc070ebd847#r32012178

manycoding picture manycoding on 7 Feb 2019

Ping @techalchemy?

al-the-x picture al-the-x on 1 Mar 2019

All changes have been on hold pending #3298 which will be merged today assuming tests are passing

The release of pip 19.0 and implementation of PEP 518 and 517 have caused significant issues and required substantial rework of pipenv's resolver, specifically in https://github.com/sarugaku/requirementslib which is now functionally acting as a dependency resolver for any VCS or editable dependencies.

As a result of those changes, I've introduced additional logic to feed the dependency graph of any VCS/File/local requirements back into the resolver _before_ we begin the resolution process so that we can actually perform an accurate resolution.

What this should mean for this specific issue is that it will be resolved by the merge of #3298 because it completely eliminates this section of the code.

techalchemy picture techalchemy on 5 Mar 2019
🎉1 👍1

Hey fellows and @techalchemy,

thank you for solving this issue!
As far as I understand it was not released yet.

Do we have a temporary solution how to avoid it meanwhile? Dependencies in several of our systems are blocked because of it.

Thanks,
Rustem

amureki picture amureki on 9 Apr 2019
👍2

@amureki Adding the dependency on which the KeyError happens on to our dependency list worked for us. (A temporary fix, of course.)

GautamGupta picture GautamGupta on 11 Apr 2019
👍4
Was this page helpful?
0 / 5 - 0 ratings

Related issues

AkiraSama picture AkiraSama  Â·  3Comments
jacebrowning picture jacebrowning  Â·  3Comments
FooBarQuaxx picture FooBarQuaxx  Â·  3Comments
xi picture xi  Â·  3Comments
hynek picture hynek  Â·  3Comments