Pipenv: Pipenv lock still takes way too long to complete

Created on 17 Aug 2020 · 10Comments · Source: pypa/pipenv

pipenv lock is still taking up to 30 minutes to complete for my medium-sized project.

While #4403 was closed as fixed and indeed no full tree copies are made any more, pipenv lock is still unworkably slow. What used to take 50 seconds under 2018.11.26, now takes 30 minutes to figure out that nothing changed:

$ export PIPENV_INSTALL_TIMEOUT=10000
$ cp Pipfile.lock Pipfile.lock.before
$ time pipenv lock
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (0e1198)!

real    29m19.701s
user    28m39.300s
sys 4m8.222s
$ diff -su Pipfile.lock.old Pipfile.lock
Files Pipfile.lock.old and Pipfile.lock are identical

This is making it untenable to keep using Pipenv.

Full pipenv lock -v output in a gist.

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*"
flake8-bugbear = "*"
black = "==19.10b0"
pre-commit = "*"
pytest = "*"
pytest-cov = "*"

[packages]
project-name = {editable = true,path = "."}
"zc.buildout" = "*"

[requires]
python_version = "3.6"

where project-name (redacted name) lists the following install-requires lines:

install_requires =
    apache-airflow[aws,celery,postgres,redis] >= 1.10.11
    airflow_multi_dagrun
    airflow-prometheus-exporter

$ pipenv --support

Pipenv version: '2020.8.13'

Pipenv location: '/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2020.8.13/libexec/bin/python3.8'

Python installations found:

3.9.0: /Users/mj/bin/python3.9
3.8.5: /usr/local/bin/python3
3.8.5: /usr/local/bin/python3.8
3.8.3: /Users/mj/bin/python3.8
3.7.7: /Users/mj/bin/python3.7
3.7.6: /opt/miniconda3/bin/python3
3.7.6: /opt/miniconda3/bin/python3.7m
3.7.6: /opt/miniconda3/bin/python3.7
3.7.3: /Users/mj/.pyenv/versions/3.7.3/bin/python3
3.6.10: /Users/mj/bin/python3.6
3.6.1: /Users/mj/bin/pypy3
3.5.9: /Users/mj/bin/python3.5
3.4.10: /Users/mj/bin/python3.4
3.3.7: /Users/mj/bin/python3.3
3.2.6: /Users/mj/bin/python3.2
2.7.18: /Users/mj/bin/python2.7
2.7.16: /usr/bin/python2.7
2.6.9: /Users/mj/bin/python2.6
2.5.6: /Users/mj/bin/python2.5
2.4.6: /Users/mj/bin/python2.4

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Wed May 27 17:00:02 PDT '
                     '2020; root:xnu-4570.71.80.1~1/RELEASE_X86_64',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

PATH
AUTOJUMP_ERROR_PATH
NVM_INC
TERM_PROGRAM
GIT_PROMPT_END
rvm_bin_path
AUTOJUMP_SOURCED
GEM_HOME
NVM_CD_FLAGS
SHELL
TERM
HISTSIZE
TMPDIR
COPYFILE_DISABLE
IRBRC
Apple_PubSub_Socket_Render
CONDA_SHLVL
PERL5LIB
TERM_PROGRAM_VERSION
CONDA_PROMPT_MODIFIER
TERM_SESSION_ID
MY_RUBY_HOME
PERL_MB_OPT
LC_ALL
USER
HISTFILESIZE
NVM_DIR
COMMAND_MODE
CONDA_EXE
rvm_path
SSH_AUTH_SOCK
__CF_USER_TEXT_ENCODING
PTPIMG_API_KEY
_CE_CONDA
rvm_prefix
_
CONDA_PREFIX
PWD
HOMEBREW_GITHUB_API_TOKEN
EDITOR
LANG
ITERM_PROFILE
PYTHONSTARTUP
XPC_FLAGS
ITERM_ORIG_PS1
XPC_SERVICE_NAME
HISTCONTROL
_CE_M
FIGNORE
rvm_version
COLORFGBG
HOME
SHLVL
LC_TERMINAL_VERSION
ITERM_PREV_PS1
ITERM_SESSION_ID
PERL_LOCAL_LIB_ROOT
LOGNAME
CONDA_PYTHON_EXE
LC_CTYPE
GEM_PATH
GIT_PROMPT_ONLY_IN_REPO
CONDA_DEFAULT_ENV
NVM_BIN
DISPLAY
LC_TERMINAL
HGSRC
SECURITYSESSIONID
RUBY_VERSION
PERL_MM_OPT
COLORTERM
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/local/Cellar/pipenv/2020.8.13/libexec/tools:/usr/local/google-sdks/google-cloud-sdk/bin:/Users/mj/perl5/bin:/Users/mj/.nvm/versions/node/v12.14.1/bin:/Users/mj/.rvm/gems/ruby-2.3.0/bin:/Users/mj/.rvm/gems/ruby-2.3.0@global/bin:/Users/mj/.rvm/rubies/ruby-2.3.0/bin:/opt/miniconda3/bin:/opt/miniconda3/condabin:/Users/mj/bin:/Users/mj/Development/Library/zopatista_setup/bin:/Users/mj/Development/Library/zopatista_setup/audio:/usr/local/Cellar/ruby/2.7.1_2/bin:/Users/mj/.cargo/bin:/Users/mj/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/sbin:/opt/X11/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Users/mj/.local/bin:/Users/mj/.rvm/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
SHELL: /bin/bash
EDITOR: vim
LANG: en_GB.UTF-8
PWD: /Users/mj

Dependency Resolution Performance

Source

mjpieters

👍21 👀9

Most helpful comment

I removed all other dependencies except the editable one. It turns out no significant difference is seen in the time costs of 2018.11.26 and 2020.8.13. It will be great if you can narrow down which package brings the performance degrading.

I created a demo repo to showcase the issue, which you can clone and adjust to reproduce the performance issue:
https://github.com/frostming/pipenv-lock-performance

$ docker build . --build-arg PIPENV_VER=2020.8.13 -t pipenv-bug
$ docker build . --build-arg PIPENV_VER=2018.11.26 -t pipenv-bug

frostming on 7 Sep 2020

❤4 🚀3

All 10 comments

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

conradg on 19 Aug 2020

👍3

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

Yes, +1! Thanks for looking into this, Martijn! We had exactly the same issue (lots of big tmp directories), but sorta gave up for the time being… I couldn't really argue much when pipenv lock took 10+ minutes for simply adding another package.

(and hi, btw 👋)

witsch on 20 Aug 2020

👍5

A "workaround" that may work (and what we have used from time to time) is to remove the lock file and rerun the lock. This may also hint at the underlying problem, which may have something to do with how pipenv is using the existing lock file during the lock process.

mgmarino on 27 Aug 2020

👍2 👀1

Deleting the Pipfile.lock speeds things up for sure. I just install all the packages from scratch and run pipenv lock without changing anything and it gets stuck eventhough there is no change at all

Can-Sahin on 27 Aug 2020

Can you have a try on the master branch? I recently disabled the copying of the local source tree.

frostming on 1 Sep 2020

Can you have a try on the master branch? I recently disabled the copying of the local source tree.

Do you mean #4421? That's been part of the 2020.8.13 release that I tested with for this issue. I did note that the tree copying no longer takes place.

I also don't see any changes on master since 2020.8.13 was released that look relevant. Did I miss something?

mjpieters on 1 Sep 2020

If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.

frostming on 1 Sep 2020

If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.

I see no differences that look problematic. With 2020.8.13.dev0 (master branch) I see markers entries added. The dependencies I see added are ones that were due to other dependencies having been upgraded (I deleted the lock file to see if there was much speed difference).

With the lock file deleted and using 2020.8.13.dev0, a pipenv lock took 10 minutes.

mjpieters on 1 Sep 2020

I created a demo repo to showcase the issue, which you can clone and adjust to reproduce the performance issue:
https://github.com/frostming/pipenv-lock-performance

$ docker build . --build-arg PIPENV_VER=2020.8.13 -t pipenv-bug
$ docker build . --build-arg PIPENV_VER=2018.11.26 -t pipenv-bug

frostming on 7 Sep 2020

❤4 🚀3

Sorry, I meant to get back to this but current project work is keeping me over-utilised. I do still find Pipenv to be excruciatingly slow, and regularly have to delete the lock file just to get a solution, but that's not really very actionable feedback.

mjpieters on 28 Oct 2020

Was this page helpful?

0 / 5 - 0 ratings