pipenv install is very slow

Created on 15 May 2017  Ā·  107Comments  Ā·  Source: pypa/pipenv

Running pipenv install after changing one dependency takes about ~5 minutes for me, on a windows 10 machine with an SSD.

The vast majority of that time is spent inside Locking [packages] dependencies...

It seems like there might be some quadratic-or-worse complexity in this step?

I've included most of our pipfile below, but I had to remove some of our private repo dependencies:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
alembic = "==0.8.4"
amqp = "==1.4.7"
analytics-python = "==1.2.5"
anyjson = "==0.3.3"
billiard = "==3.3.0.20"
braintree = "==3.20.0"
celery = "==3.1.18"
coverage = "==4.0.3"
docopt = "==0.4.0"
eventlet = "==0.19.0"
flake8 = "==3.0.4"
Flask-Cors = "==2.1.2"
Flask-Login = "==0.3.2"
Flask = "==0.12.1"
funcsigs = "==0.4"
fuzzywuzzy = "==0.12.0"
gcloud = "==0.14.0"
html2text = "==2016.9.19"
itsdangerous = "==0.24"
Jinja2 = "==2.8"
jsonpatch = "==1.15"
jsonschema = "==2.5.1"
PyJWT = "==1.4.2"
kombu = "==3.0.30"
LayerClient = "==0.1.9"
MarkupSafe = "==0.23"
mixpanel = "==4.3.0"
mock = "==1.3.0"
nose-exclude = "==0.4.1"
nose = "==1.3.7"
numpy = "==1.12.1"
pdfrw = "==0.3"
Pillow = "==4.1.0"
pusher = "==1.6"
pycountry = "==1.20"
pycryptodome = "==3.4.5"
pymongo = "==3.2"
PyMySQL = "==0.7.4"
python-dateutil = "<=2.5.1"
python-Levenshtein = "==0.12.0"
python-magic = "==0.4.6"
python-coveralls = "==2.9.0"
pytz = "==2015.6"
raygun4py = "==3.1.2"
"repoze.retry" = "==1.3"
requests = "==2.8.1"
sendgrid = "==2.2.1"
slacker = "==0.7.3"
SQLAlchemy-Enum34 = "==1.0.1"
SQLAlchemy-Utils = "==0.31.6"
SQLAlchemy = "==1.1.9"
typing = "==3.5.2.2"
twilio = "==5.6.0"
Unidecode = "==0.4.19"
voluptuous = "==0.8.11"
Wand = "==0.4.4"
watchdog = "==0.8.3"
Werkzeug = "==0.12.1"
wheel = "==0.24.0"
WTForms = "==2.0.2"
xmltodict = "==0.9.2"
zeep = "==0.24.0"
picture Diggsey
👍66

Most helpful comment

Why are all the issues to this topic closed? I can't pipenv install a single thing due to the lock-step hang.

picture black-snow on 8 Jan 2019
👍55

All 107 comments

Hey again @Diggsey, this is due to the way we're writing changes right now. I have these changes ready to merge but they're breaking for the projects.py API so we're going to hold off until the next major release. Hopefully we'll have this up and ready in the next few weeks. I'll leave this open to track the issue for now.

nateprewitt picture nateprewitt on 15 May 2017

We sprinted on this together at PyCon. It'll be faster soon.

kennethreitz42 picture kennethreitz42 on 25 May 2017
👍11

Right now for me it is not slow, it is freezing...

A pipenv install my_package or a simple pipenv install does not give me any output, after 20 minutes.

EDIT: Confirmation, still nothing after a few hours. Is it the same problem? Usually it was slow, but it ended after 5 to 10 minutes.

NicolasWebDev picture NicolasWebDev on 27 May 2017
👍45

Hey @NicolasWebDev, what version of pipenv are you using? Also do you have delegator.py installed on your system separately? If so, what version is that at? This was an issue that should have been resolved in v3.6.0.

If everything above is up to date, could you please provide the contents of your Pipfile? Thanks!

nateprewitt picture nateprewitt on 27 May 2017

Hi @nateprewitt, you were right, I was on v3.5.x. Updating to 4.1.1 resolved the freezing issue. Now it is still taking 5 minutes, but at least it is usable!

Sorry for the noise, I always forget that packages installed through pip are not automatically updated.
Thanks!

NicolasWebDev picture NicolasWebDev on 29 May 2017

Glad you got things resolve @NicolasWebDev! We're working on getting this sped up more, hopefully #373 will be a step closer in the next release.

nateprewitt picture nateprewitt on 29 May 2017
👍1

@Diggsey @NicolasWebDev, I've just released 4.1.2 which should have these speed improvements added. There's still some work to do here, but this will at least expedite the initial bootstrap time for pipenv.

nateprewitt picture nateprewitt on 30 May 2017

@nateprewitt Thanks for the update, pipenv does seem faster for me now, but it still takes several minutes just to run pipenv lock, even when nothing has changed - it still waits in Locking [packages] dependencies... for the vast majority of that time.

Diggsey picture Diggsey on 31 May 2017

@Diggsey, a lot of that time is because you're downloading a huge number of files in that Pipfile. It seems like you're pinning all of your dependencies too. Are you directly importing all of these into your project or are some dependency requirements of the others?

nateprewitt picture nateprewitt on 31 May 2017

@nateprewitt We could remove some of them, but the majority are direct dependencies - why does it need to download all of the dependencies every time it generates the lock file?

Diggsey picture Diggsey on 31 May 2017

We need to determine everything that it installs as dependencies. In order to get that, we download each package and determine what an installation looks like at lock time. This allows us to appropriately pin everything in the Pipfile.lock. Without downloading, there isn't a reliable way of checking sub-dependencies and resolving range dependency declarations.

nateprewitt picture nateprewitt on 31 May 2017

Given that most of the packages are going to stay the same over time, would it be possible to cache the downloaded packages?

Diggsey picture Diggsey on 31 May 2017
👍28

@Diggsey want to look into that for us?

kennethreitz42 picture kennethreitz42 on 4 Sep 2017

This might be a silly question, but doesnā€™t Pip already do package caching?

uranusjr picture uranusjr on 5 Sep 2017

I'm under the impression that it does.

kennethreitz42 picture kennethreitz42 on 5 Sep 2017

Can pipenv use the pip cache system or it has to be implemented from the scratch?

marctc picture marctc on 5 Sep 2017

Pipenv just runs pip, so the cache should automatically be used.

kennethreitz42 picture kennethreitz42 on 5 Sep 2017

fixed! lock is wicked fast now.

kennethreitz42 picture kennethreitz42 on 6 Sep 2017
❤1

Oh, thank you. I think that was the only thing holding my back from pushing everyone over to pipenv at work.

tilgovi picture tilgovi on 6 Sep 2017
🎉3

Wow, nice, that was literally more than a 100x speedup for me, and it also caught a dependency conflict that the previous version didn't catch!

What would be useful is a verbose flag for pipenv lock - I was only able to diagnose the dependency conflict by editing piptools/logging.py to enable verbose logging, but once I did that it gave a very clear indication of what was going on.

Diggsey picture Diggsey on 6 Sep 2017
❤1 🎉1

I'm probably missing something :) Where is it fixed? Latest release is 4 days ago, so I installed latest version from master. However, pipenv install is still slow.

I tried:

  • install pipenv the fancy way āš”ļø šŸ° āš”ļø
  • use both latest published version of pipenv and latest version from master
  • install single package

Using latest stable version (5.3.5.), it takes 3:40 to install one package:

āˆ™ time pipenv install --dev raven
Installing raven...
Collecting raven
  Using cached raven-6.1.0-py2.py3-none-any.whl
Collecting contextlib2 (from raven)
  Using cached contextlib2-0.5.5-py2.py3-none-any.whl
Installing collected packages: contextlib2, raven
Successfully installed contextlib2-0.5.5 raven-6.1.0

Adding raven to Pipfile's [dev-packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
pipenv install --dev raven  10,11s user 2,77s system 5% cpu 3:40,04 total

Using version from master, it still hangs (one package, +10 minutes)

EDIT: It has just finished:

pipenv install graphene_django  8,03s user 1,28s system 1% cpu 11:23,11 total

Any ideas? Thanks a lot!

tricoder42 picture tricoder42 on 6 Sep 2017

Sometimes dependencies take a while to install, especially if they have c compilations. Want to share your Pipfile?

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

I understand sometimes it takes a while, but it was slow from the beginning. Just curious if it's a problem at my side.

Here's my Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"
pytest-django = "*"
pytest-testmon = "*"
pytest-watch = "*"
django-debug-toolbar = "*"
raven = "*"

[packages]
dj-database-url = "*"
Django = "*"
djangorestframework = "*"
gunicorn = "*"
newrelic = "*"
psycopg2 = "*"
requests = "*"
whitenoise = "*"
graphene-django = "*"
tricoder42 picture tricoder42 on 6 Sep 2017

psycopg2 might be taking a while, as it may be compiling from source. Everything else should be nice and fast. Try removing it and see how much your speed increases.

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

$ pipenv install raven just took like 1s for me.

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

$ pipenv install raven just took like 1s for me.

That's what I would expect! Can I turn on verbose output somehow?

I tried remove psycopg2, but it doesn't affect much. Running pipenv install raven hangs for a while.

I have:

  • Python 3.6.2
  • macOS 10.12.6
tricoder42 picture tricoder42 on 6 Sep 2017

i see no reason why raven shouldn't be instantaneous.

do $ pip install raven inside of $ pipenv shell and tell me if it's slow there too.

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

All pipenv is doing is running pip, so that's effectively the "verbose mode"

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

That's instant:

āˆ™ time pip install raven                                                                                                                                 19:38  tricoder@issac
Requirement already satisfied: raven in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages
Requirement already satisfied: contextlib2 in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages (from raven)
noglob pip install raven  0,54s user 0,15s system 76% cpu 0,900 total

Running pipenv hangs about 2-3 minutes (inside/outside pipenv shell).

āˆ™ time pipenv install raven                                                                                                                              19:39  tricoder@issac
Installing raven...
Requirement already satisfied: raven in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages
Requirement already satisfied: contextlib2 in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages (from raven)

Adding raven to Pipfile's [packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
pipenv install raven  4,49s user 0,46s system 2% cpu 3:21,17 total
tricoder42 picture tricoder42 on 6 Sep 2017

@Diggsey can you open a new issue about --verbose and provide an example of what you'd like it to look like?

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

@tricoder42 is the slow part the locking step or the install step? Locking was vastly improved in the latest releases.

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

```shell
$ time pipenv install raven
Installing raven...
Collecting raven
Using cached raven-6.1.0-py2.py3-none-any.whl
Requirement already satisfied: contextlib2 in /Users/kennethreitz/.local/share/virtualenvs/pipenv-u9yqWeFK/lib/python3.6/site-packages (from raven)
Installing collected packages: raven
Successfully installed raven-6.1.0

Adding raven to Pipfile's [packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
9.30 real 5.49 user 0.42 sys

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

It's like 50:50 installing:locking

tricoder42 picture tricoder42 on 6 Sep 2017

@tricoder42 and you're using latest?

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

Let me replicate with your exact pipfile.

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

I guess so:

āˆ™ pipenv --version                                                                                                                                       19:42  tricoder@issac
pipenv, version 5.3.5

āˆ™ pipsi upgrade git+https://github.com/kennethreitz/pipenv.git#egg=pipenv                                                                                19:45  tricoder@issac
Collecting pipenv from git+https://github.com/kennethreitz/pipenv.git#egg=pipenv
  Cloning https://github.com/kennethreitz/pipenv.git to /private/var/folders/g9/1wbckv154mbby3tm411z_m340000gn/T/pip-build-se4ao5/pipenv
...
Installing collected packages: pipenv
  Found existing installation: pipenv 5.3.5
    Uninstalling pipenv-5.3.5:
      Successfully uninstalled pipenv-5.3.5
  Running setup.py install for pipenv ... done
Successfully installed pipenv-5.3.5
tricoder42 picture tricoder42 on 6 Sep 2017

```shell
$ time pipenv install
No package provided, installing all dependencies.
Pipfile found at /Users/kennethreitz/pipenv/testapp/Pipfile. Considering this to be the project home.
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock...
[================================] 22/22 - 00:00:37
58.94 real 40.51 user 8.62 sys

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

It does even when I'm installing first package into fresh new pipenv. I'll try to create pipenv --three instead of pipenv --python python3.6

tricoder42 picture tricoder42 on 6 Sep 2017

@tricoder42 want to hop on a google hangouts in like an hour?

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

Or we can use Apple screen sharing if you use Messages.app.

Add me! I'm [email protected].

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

I'm about to join a meeting, but I'll be available after that.

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

Cool! I'll try clean & reinstall everything from the scratch and we'll see. I'm available in one hour

tricoder42 picture tricoder42 on 6 Sep 2017

Awesome ā€” we'll figure this out then. Add me on messages.app :)

kennethreitz42 picture kennethreitz42 on 6 Sep 2017

If anyone is experiencing extremely slow Locking behavior with v11.9.0, i found that downgrading to v9.0.0 takes a 5m30s install down to 1m36s.

ryantuck picture ryantuck on 11 Apr 2018

@ryantuck I'd recommend if you're going to pin an old version to pin 9.0.3 but you lose a significant amount of added security for the speed, you might as well just use --skip-lock at that point

techalchemy picture techalchemy on 11 Apr 2018

--skip-lock definitely sped things up. I went down this path because pipenv install --system --python=3.6 wasn't actually installing to the correct system python, and i was under the assumption that i needed to generate the Pipfile.lock before attempting the system install. It still didn't work, so I went back to using regular old pip, but will comment back on this thread if I ever make any progress along these lines.

ryantuck picture ryantuck on 11 Apr 2018

ā€”system and ā€”python are mutually exclusive ā€” the latter option always needs a virtualenv

techalchemy picture techalchemy on 12 Apr 2018

yep, locking takes a while for me too. v11.10.0. Ubuntu on WSL.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
babel = "==2.5.3"
"boto3" = "==1.7.3"
colorama = "==0.3.9"
coreapi = "==2.3.3"
dj-database-url = "==0.5.0"
djangorestframework = "==3.7.7"
django-axes = "==4.0.2"
django-clever-selects = "==0.8.2"
django-crispy-forms = "==1.7.2"
django-choices = "==1.6.0"
django-extra-views = "==0.10.0"
django-filter = "==1.1.0"
django-hijack = "==2.1.7"
django-hijack-admin = "==2.1.7"
django-js-reverse = "==0.8.1"
django-model-utils = "==3.1.1"
django-phonenumber-field = "==2.0.0"
django-polymorphic = "==2.0.2"
django-redis-cache = "==1.7.1"
django-role-permissions = "==2.2.0"
"django-s3direct" = "==1.0.4"
django-static-precompiler = {extras = ["libsass"], version = "==1.8.2"}
django-storages = "==1.6.6"
"django-tables2" = "==1.21.2"
django-webpack-loader = "==0.6.0"
django-widget-tweaks = "==1.4.2"
facebookads = "==2.11.4"
googleads = "==11.0.1"
markdown = "==2.6.11"
phonenumbers = "==8.9.3"
pillow = "==5.1.0"
"psycopg2-binary" = "==2.7.4"
pygments = "==2.2.0"
pyssim = "==0.4"
python-dotenv = "==0.8.2"
pytz = "==2018.4"
raven = "==6.6.0"
sendgrid-django = "==4.2.0"
slacker = "==0.9.65"
termcolor = "==1.1.0"
tqdm = "==4.21.0"
twitter-ads = "==3.0.0"
brotlipy = "==0.7.0"
waitress = "==1.1.0"
whitenoise = "==3.3.1"
Django = "==2.0.4"

[dev-packages]
coverage = "==4.5.1"
selenium = "==3.11.0"
tblib = "==1.3.2"
"flake8" = "==3.5.0"
django-debug-toolbar = "==1.9.1"
django-extensions = "==2.0.6"

[requires]
python_version = "3.6"

Pipfile.lock not found, creatingā€¦
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
Updated Pipfile.lock (7a535c)!
Installing dependencies from Pipfile.lock (7a535c)ā€¦
  šŸ   ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ 92/92 ā€” 00:01:01

real    8m1.993s
user    5m32.406s
sys     7m15.203s
kavdev picture kavdev on 12 Apr 2018

It should be a little bit faster the second or third time because of
caching tho. Do you see an improvement?
On Thu, Apr 12, 2018 at 10:23 AM Alexander Kavanaugh <
[email protected]> wrote:

yep, locking takes a while for me too. v11.10.0. Ubuntu on WSL.

[[source]]url = "https://pypi.python.org/simple"verify_ssl = truename = "pypi"
[packages]babel = "==2.5.3""boto3" = "==1.7.3"colorama = "==0.3.9"coreapi = "==2.3.3"dj-database-url = "==0.5.0"djangorestframework = "==3.7.7"django-axes = "==4.0.2"django-clever-selects = "==0.8.2"django-crispy-forms = "==1.7.2"django-choices = "==1.6.0"django-extra-views = "==0.10.0"django-filter = "==1.1.0"django-hijack = "==2.1.7"django-hijack-admin = "==2.1.7"django-js-reverse = "==0.8.1"django-model-utils = "==3.1.1"django-phonenumber-field = "==2.0.0"django-polymorphic = "==2.0.2"django-redis-cache = "==1.7.1"django-role-permissions = "==2.2.0""django-s3direct" = "==1.0.4"django-static-precompiler = {extras = ["libsass"], version = "==1.8.2"}django-storages = "==1.6.6""django-tables2" = "==1.21.2"django-webpack-loader = "==0.6.0"django-widget-tweaks = "==1.4.2"facebookads = "==2.11.4"googleads = "==11.0.1"markdown = "==2.6.11"phonenumbers = "==8.9.3"pillow = "==5.1.0""psycopg2-binary" = "==2.7.4"pygments = "==2.2.0"pyssim = "==0.4"python-dotenv = "==0.8.2"pytz = "==2018.4"raven = "==6.6.0"sendgrid-django = "==4.2.0"slacker = "==0.9.65"termcolor = "==1.1.0"tqdm = "==4.21.0"twitter-ads = "==3.0.0"brotlipy = "==0.7.0"waitress = "==1.1.0"whitenoise = "==3.3.1"Django = "==2.0.4"
[dev-packages]coverage = "==4.5.1"selenium = "==3.11.0"tblib = "==1.3.2""flake8" = "==3.5.0"django-debug-toolbar = "==1.9.1"django-extensions = "==2.0.6"
[requires]python_version = "3.6"

Pipfile.lock not found, creatingā€¦
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
Updated Pipfile.lock (7a535c)!
Installing dependencies from Pipfile.lock (7a535c)ā€¦
šŸ ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ 92/92 ā€” 00:01:01

real 8m1.993s
user 5m32.406s
sys 7m15.203s

ā€”
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pipenv/issues/356#issuecomment-380882203, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABhjqwIPyHtX0NTVoV1UPYR7HcwYm-2kks5tn42SgaJpZM4NbeoN
.

jtratner picture jtratner on 12 Apr 2018

I had already installed the packages before that; I just removed the lockfile and ran install again. I'll do it again to get another data point

kavdev picture kavdev on 12 Apr 2018

@jtratner Woah. Super interesting... what causes the caching to kick in only after the third+ time?

Pipfile.lock not found, creatingā€¦
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
Updated Pipfile.lock (7a535c)!
Installing dependencies from Pipfile.lock (7a535c)ā€¦
An error occurred while installing coreapi==2.3.3! Will try again.
  šŸ   ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ 92/92 ā€” 00:00:58
Installing initiallyā€“failed dependenciesā€¦
Success installing coreapi==2.3.3!ā–‰ā–‰ā–‰ 0/1 ā€” 00:00:00
  ā˜¤  ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ 1/1 ā€” 00:00:01

real    2m50.218s
user    2m19.438s
sys     5m44.797s

Pipfile.lock not found, creatingā€¦
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
Updated Pipfile.lock (7a535c)!
Installing dependencies from Pipfile.lock (7a535c)ā€¦
  šŸ   ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ 92/92 ā€” 00:00:55

real    2m32.042s
user    2m6.516s
sys     5m10.219s
kavdev picture kavdev on 12 Apr 2018

@kavdev @jtratner introduced a feature to cache the hashes as well so that should make a sizeable improvement

techalchemy picture techalchemy on 13 Apr 2018
👍1

Here I am... 15 mins later

giantas picture giantas on 22 Apr 2018
👍2 👎1

@giantas not helpful. Please provide pipfile, output, and duration with pip install. Also whether you can do invidual packages.

techalchemy picture techalchemy on 22 Apr 2018
👍1

It takes quite a lot to run on my machine. macOS 10.13.4, pipenv, version 11.10.0

Download runs almost immediately, but then it get's stuck on Locking [packages] dependenciesā€¦. Here's taking half a minute for two dependencies, and then 6 minutes for another 3 dependencies, if I throw all my project's dependencies at it, it just hangs indefinitely at the locking step

pablo@batman scanr (develop) $ time pipenv install flask flask_pymongo
Installing flaskā€¦
Looking in indexes: https://pypi.python.org/simple
Collecting flask
  Using cached https://files.pythonhosted.org/packages/77/32/e3597cb19ffffe724ad4bf0beca4153419918e7fa4ba6a34b04ee4da3371/Flask-0.12.2-py2.py3-none-any.whl
Collecting Werkzeug>=0.7 (from flask)
  Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Collecting itsdangerous>=0.21 (from flask)
Collecting Jinja2>=2.4 (from flask)
  Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting click>=2.0 (from flask)
  Using cached https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->flask)
Installing collected packages: Werkzeug, itsdangerous, MarkupSafe, Jinja2, click, flask
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 flask-0.12.2 itsdangerous-0.24

Adding flask to Pipfile's [packages]ā€¦
Installing flask_pymongoā€¦
Looking in indexes: https://pypi.python.org/simple
Collecting flask_pymongo
  Using cached https://files.pythonhosted.org/packages/fa/71/ab920741dedd605ef4adbd57d0c7d9f43a6b6fe4068604fffbc6f64b2c9c/Flask_PyMongo-0.5.1-py3-none-any.whl
Requirement already satisfied: Flask>=0.8 in /Users/pablo/.local/share/virtualenvs/scanr-2m6AW0PB/lib/python3.6/site-packages (from flask_pymongo) (0.12.2)
Collecting PyMongo>=2.5 (from flask_pymongo)
  Using cached https://files.pythonhosted.org/packages/5c/7f/1f7240883ec3fa768d7e066c9cbd42ceb42d699ba1a0fb9d231c098a542d/pymongo-3.6.1-cp36-cp36m-macosx_10_6_intel.whl
Requirement already satisfied: click>=2.0 in /Users/pablo/.local/share/virtualenvs/scanr-2m6AW0PB/lib/python3.6/site-packages (from Flask>=0.8->flask_pymongo) (6.7)
Requirement already satisfied: itsdangerous>=0.21 in /Users/pablo/.local/share/virtualenvs/scanr-2m6AW0PB/lib/python3.6/site-packages (from Flask>=0.8->flask_pymongo) (0.24)
Requirement already satisfied: Werkzeug>=0.7 in /Users/pablo/.local/share/virtualenvs/scanr-2m6AW0PB/lib/python3.6/site-packages (from Flask>=0.8->flask_pymongo) (0.14.1)
Requirement already satisfied: Jinja2>=2.4 in /Users/pablo/.local/share/virtualenvs/scanr-2m6AW0PB/lib/python3.6/site-packages (from Flask>=0.8->flask_pymongo) (2.10)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/pablo/.local/share/virtualenvs/scanr-2m6AW0PB/lib/python3.6/site-packages (from Jinja2>=2.4->Flask>=0.8->flask_pymongo) (1.0)
Installing collected packages: PyMongo, flask-pymongo
Successfully installed PyMongo-3.6.1 flask-pymongo-0.5.1

Adding flask_pymongo to Pipfile's [packages]ā€¦
Pipfile.lock (c202d3) out of date, updating to (3068be)ā€¦
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
Updated Pipfile.lock (3068be)!
Installing dependencies from Pipfile.lock (3068be)ā€¦
  šŸ   ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ 32/32 ā€” 00:00:03
To activate this project's virtualenv, run the following:
 $ pipenv shell

real    0m37.816s
user    0m34.556s
sys 0m4.517s

pablo@batman scanr (develop) $ time pipenv install gunicorn h5py joblib
Installing gunicornā€¦
Looking in indexes: https://pypi.python.org/simple
Collecting gunicorn
  Using cached https://files.pythonhosted.org/packages/64/32/becbd4089a4c06f0f9f538a76e9fe0b19a08f010bcb47dcdbfbc640cdf7d/gunicorn-19.7.1-py2.py3-none-any.whl
Installing collected packages: gunicorn
Successfully installed gunicorn-19.7.1

Adding gunicorn to Pipfile's [packages]ā€¦
Installing h5pyā€¦
Looking in indexes: https://pypi.python.org/simple
Collecting h5py
  Using cached https://files.pythonhosted.org/packages/69/d5/dff2a8f7658fd87ab3330a0ab47e4363681d8bdf734a495add65a347f5e3/h5py-2.7.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Requirement already satisfied: six in /Users/pablo/.local/share/virtualenvs/scanr-2m6AW0PB/lib/python3.6/site-packages (from h5py) (1.11.0)
Collecting numpy>=1.7 (from h5py)
  Using cached https://files.pythonhosted.org/packages/a0/df/fa637677800e6702a57ef09e1d62e42aec3f598fb235f897155d146f2f59/numpy-1.14.2-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Installing collected packages: numpy, h5py
Successfully installed h5py-2.7.1 numpy-1.14.2

Adding h5py to Pipfile's [packages]ā€¦
Installing joblibā€¦
Looking in indexes: https://pypi.python.org/simple
Collecting joblib
  Using cached https://files.pythonhosted.org/packages/4f/51/870b2ec270fc29c5d89f85353da420606a9cb39fba4747127e7c7d7eb25d/joblib-0.11-py2.py3-none-any.whl
Installing collected packages: joblib
Successfully installed joblib-0.11

Adding joblib to Pipfile's [packages]ā€¦
Pipfile.lock (0d514f) out of date, updating to (a4d15f)ā€¦
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
Updated Pipfile.lock (a4d15f)!
Installing dependencies from Pipfile.lock (a4d15f)ā€¦
  šŸ   ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ā–‰ 36/36 ā€” 00:00:03
To activate this project's virtualenv, run the following:
 $ pipenv shell

real    6m31.572s
user    1m1.986s
sys 0m11.047s
pablote picture pablote on 25 Apr 2018
👍2

@pablote holy crap that is slow! Do note that this is partly due to the installation of numpy, which I'm sure we are probalby compiling from source to lock or something stupid. would help if we provided useful output here

techalchemy picture techalchemy on 25 Apr 2018
👍1

Is there anything I can do about it? or am I just out of luck at using pipenv and having these slow lock file times? I guess I could --skip-lock but I sort of wanted that feature

pablote picture pablote on 25 Apr 2018

@pablote can you check if you rerun the same locking operation, is it still slow?

techalchemy picture techalchemy on 25 Apr 2018

If I rerun it finishes almost immediately, so I guess it's just the first time some new dependency gets added.

pablote picture pablote on 25 Apr 2018

so it's actually caching the hashes locally, so for whatever reason it's the hash generation process that is incredibly slow here...

techalchemy picture techalchemy on 25 Apr 2018

Let me know if there's any info I can provide to help diagnose. For the moment I'll just go back to pip + virtualenv + pip-tools :/

pablote picture pablote on 25 Apr 2018
👍2

@pablote once you have the hashes once, locking again won't be that slow

techalchemy picture techalchemy on 25 Apr 2018

Please, please provide useful output. I just upgraded my pipenv from 9.1.0 to 11.10.0 in order to resolve the Invalid Marker failure at package lock step as per, e.g., #1622 --- now, I have a pipfile with ipykernel, pandas, jupyter, numpy, and matplotlib in there and with my latest attempt to use pipenv install to get the lock file going, I've been sitting in locking [packages] dependenciesā€¦ for upwards of 10 minutes.

Because there's no output, I can't tell whether there is something actually going (like building numpy from source) or if it's just hanging. The best I can do is sort of squint at top and conclude that maybe it's doing something because a python process seems to be hanging around... but I'm going to have to trash this virtualenv and start fresh if something doesn't move soon.

I'm happy to contribute to some work on this if needed.

paultopia picture paultopia on 26 Apr 2018
👍9

Please, please provide useful output. I just upgraded my pipenv from 9.1.0 to 11.10.0 in order to resolve the Invalid Marker failure at package lock step as per, e.g., #1622 --- now, I have a pipfile with ipykernel, pandas, jupyter, numpy, and matplotlib in there and with my latest attempt to use pipenv install to get the lock file going, I've been sitting in locking [packages] dependenciesā€¦ for upwards of 10 minutes.

Totally understandable complaint and I've been stewing in the back of my mind trying to think about how we can provide more useful feedback to the user (because I agree it's a little bit confusing that it doesn't have any output). I'd give up after 15 minutes. As @techalchemy pointed out, each time your cache should get populated a bit more, so it should get consistently faster over time.

One question: are you using public PyPI?

jtratner picture jtratner on 26 Apr 2018

@jtratner yes, using public PyPi --- and I eventually did end up giving up, trashing the virtualenv and building a new one; I did manage to get a lockfile built by installing my dependencies one at a time. (Interestingly, matplotlib was the slowest---even worse than numpy!)

paultopia picture paultopia on 26 Apr 2018

Perhaps a message such as This may take a long time would help to reassure people until a clearer solution is decided upon.

meshy picture meshy on 26 Apr 2018
👍1

15 mins is still insanely long I doubt Iā€™d sit and wait for it. @paultopia you can run pipenv lock ā€”verbose for more output

techalchemy picture techalchemy on 26 Apr 2018

It does have the general feel of being slower than it should be, but maybe I'm underestimating the issue. If I look at the running processes in my computer, I can see the python one running the whole time pipenv is running, and it never goes above ~15%, it should probably use more of it if it's doing cpu intensive work like hashing files. Also, I've used other package managers that hash dependencies, like yarn, and they are pretty fast.

pablote picture pablote on 26 Apr 2018
👍1

We need to figure out what itā€™s doing...

techalchemy picture techalchemy on 26 Apr 2018

I've created a github project showing slowness when using locking. Please have a look at https://github.com/AndreasPresthammer/slow-pipenv . I'm not a 100% sure that this is the same problem though. Pull the repo down and run the slow.sh script, and observe the difference in execution time.

AndreasPresthammer picture AndreasPresthammer on 4 May 2018

@AndreasPresthammer so your script is just timing an uncached lock vs installing with lock. We know that itā€™s the locking but thatā€™s slow. In the case of numpy itā€™s probably because itā€™s had to use sdists for resolution in the past which meant compilation. We can use wheels now. That may speed things up

techalchemy picture techalchemy on 4 May 2018
🎉1

This is definitely still a problem (5+ mins), with latest python3.6, pip, and pipenv versions, and installing a simple package like torch. I don't think this issue should be marked as closed.

sergeyk picture sergeyk on 6 May 2018
👍13

For anyone who wants to comment on this issue being closed: provide the output of pipenv lock --verbose. We know resolution is slow for packages that get compiled from source and that take a long time to build under normal conditions. We need logs to gain insight into the cause if this is slower than normal.

techalchemy picture techalchemy on 7 May 2018

I'm also using Docker for my dev and production environment and think that the installation inside of Docker feels slow, compared to installation on the host. Maybe it's just the combination of all the packages, but I'd like to hear some feedback from people more experienced than I am.

Here is the log of pipenv lock --verbose, including the Pipfile and Pipfile.lock:

https://gist.github.com/mimischi/6270b7ece566cc571b427baaf1c331d4

mimischi picture mimischi on 9 May 2018

@mimischi The resolution results are cached in the host, but Docker cannot access them. It is slow because it needs to redo all the resolution from scratch. It would help if you mount the hostā€™s cache directory into Docker. Thereā€™s a comment in an issue mentioning the path you should mount. I donā€™t have the time to look it up now, but you can try find it. (And please contribute it to the docā€™s FAQ page when you do!)

uranusjr picture uranusjr on 9 May 2018

FWIW, locking on my dev box now takes less than 30s. Much better than before -- appreciate it.

kavdev picture kavdev on 25 May 2018

Hi, getting the same problem.
here is the verbose

pipenv lock --verbose
Locking [dev-packages] dependenciesā€¦
Using pip: -i https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  pylint

Finding the best candidates:
  found candidate pylint==1.9.1 (constraint was <any>)

Finding secondary dependencies:
  pylint==1.9.1             requires astroid<2.0,>=1.6, isort>=4.2.5, mccabe, six

New dependencies found in this round:
  adding ['astroid', '<2.0,>=1.6', '[]']
  adding ['isort', '>=4.2.5', '[]']
  adding ['mccabe', '', '[]']
  adding ['six', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  astroid<2.0,>=1.6
  isort>=4.2.5
  mccabe
  pylint
  six

Finding the best candidates:
  found candidate astroid==1.6.4 (constraint was >=1.6,<2.0)
  found candidate isort==4.3.4 (constraint was >=4.2.5)
  found candidate mccabe==0.6.1 (constraint was <any>)
  found candidate pylint==1.9.1 (constraint was <any>)
  found candidate six==1.11.0 (constraint was <any>)

Finding secondary dependencies:
  pylint==1.9.1             requires astroid<2.0,>=1.6, isort>=4.2.5, mccabe, six
  astroid==1.6.4            requires lazy-object-proxy, six, wrapt
  isort==4.3.4              requires -
  mccabe==0.6.1             requires -
  six==1.11.0               requires -

New dependencies found in this round:
  adding ['lazy-object-proxy', '', '[]']
  adding ['wrapt', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3                           
Current constraints:
  astroid<2.0,>=1.6
  isort>=4.2.5
  lazy-object-proxy
  mccabe
  pylint
  six
  wrapt

Finding the best candidates:
  found candidate astroid==1.6.4 (constraint was >=1.6,<2.0)
  found candidate isort==4.3.4 (constraint was >=4.2.5)
  found candidate lazy-object-proxy==1.3.1 (constraint was <any>)
  found candidate mccabe==0.6.1 (constraint was <any>)
  found candidate pylint==1.9.1 (constraint was <any>)
  found candidate six==1.11.0 (constraint was <any>)
  found candidate wrapt==1.10.11 (constraint was <any>)

Finding secondary dependencies:
  astroid==1.6.4            requires lazy-object-proxy, six, wrapt
  wrapt==1.10.11            requires -
  lazy-object-proxy==1.3.1  requires -
  six==1.11.0               requires -
  pylint==1.9.1             requires astroid<2.0,>=1.6, isort>=4.2.5, mccabe, six
  mccabe==0.6.1             requires -
  isort==4.3.4              requires -
------------------------------------------------------------
Result of round 3: stable, done

Locking [packages] dependenciesā€¦

Here is the pipenv --version: pipenv, version 2018.05.18

Also i don't know why this is happening no specific reason/ error is occurring.
In my case when I do pipenv lock it starts but it never ends as far as I know. I have been waiting for 2 hrs now still no sign of completion. And has given me ReadTimeoutError twice this is the third time I am doing this.
Using Python 3.6.4

Any help would be of great help as my projects due date is closer.

rajkumarGosavi picture rajkumarGosavi on 30 May 2018
👍2

Also vote for this issue to be reopened. It is far from solved....take somewhere between 10-20 minutes to lock my project within a Docker container. It also uses an insane amount of memory - such that I had to increase the allocation to Docker to stop it from killing the process.

jkp picture jkp on 5 Jun 2018
👍8

Please provide example Pipfilesmand details about your environments if you want help with issues with resolution speed. We are cutting a release this week and donā€™t have time to track individual one off comments on closed threads but can test against Pipfiles if you provide one

techalchemy picture techalchemy on 5 Jun 2018

@jkp Allow me to assure you that each and every core developer of this project (not that there are many of us to begin with) is very well aware of this issue, and is as troubled by it as you are, if not more. This is, however, by no means an easy problem, and we have already throw everything we could at it to make it as usable as possible, without having to tear everything down in Python packaging. We do also have a lot on our plate at the moment, and need to also focus on those other issues. The inevitable decision me must make, then, is to prioritise issues we actually know how to solve, and only start thinking about our next steps after those are done, to maximise the effect of our effort.

Now, I fully acknowledge that your priority may be different from ours. This performace issue can be the single most largest problem in your workflow, and you want to put it up as the most important thing in this project. Please bear in mind, however, that you are not the only user of this tool, and we need to prioritise the need of all, even the need of our own, in front of yours. I acknowledge that. I urge, therefore, anyone sharing the situation to join minds on this issue, and try to think of a way to solve it. Once we know what to actually do, we can do it.

The issue is kept closed because we do not know what we can do, and would serve only as noise in the issue tracker when we try to manage it. There is no point, as least in our workflow, to have an issue that nobody can work on.

uranusjr picture uranusjr on 5 Jun 2018
👍9 ❤7

Appreciate your workflow so if that is how you manage issues that's fine. I'll try to add any information I can to help track down the problem.

I did some debugging by installing mitmproxy between pipenv and the net to trace the requests being made. I found a couple of interesting things.

  1. We are using a private pypi index that doesn't support the json-api yet. This slows things down significantly since it looks like the fallback is to bruteforce and download everything listed in http index in order to extract metadata etc. One suggestion here would be just to add some simple logging to warn that this fallback method is being used - it might save someone else having to dig deeper to figure this out.

  2. Using the brute-force method it seems that the code downloads packages that aren't relevant to the architecture in use. For example on a linux machine it was downloading win32 or osx specific wheel packages. This feels like something that could be detected and avoided since it's clear that binary packages built for other architectures are not going to be of any use.

I will continue to debug and report back any useful info as I find it.

jkp picture jkp on 5 Jun 2018
👍3

It seems like even with the json interface in use pipenv is making unnecessary requests for wheel files that relate to different architectures. The implementation is currently pretty naive in that it checks all files listed against a given release, regardless of the target platform / arch.

Minimal test-case:

On a Linux host: pipenv install grpcio

Produced the following requests (captured using mitmproxy):

$ mitmdump -w dump.out
Proxy server listening at http://*:8080
127.0.0.1:62303: clientconnect
127.0.0.1:62303: GET https://pypi.org/simple/setuptools/
              << 200 OK 79.82k
127.0.0.1:62305: clientconnect
127.0.0.1:62305: GET https://files.pythonhosted.org/packages/7f/e1/820d941153923aac1d49d7fc37e17b6e73bfbd2904959fffbad77900cf92/setuptools-39.2.0-py2.py3-none-any.whl
              << 200 OK 554.25k
127.0.0.1:62303: GET https://pypi.org/simple/pip/
              << 200 OK 9.56k
127.0.0.1:62303: GET https://pypi.org/simple/wheel/
              << 200 OK 6.91k
127.0.0.1:62303: clientdisconnect
127.0.0.1:62305: clientdisconnect
127.0.0.1:62307: clientconnect
127.0.0.1:62307: GET https://pypi.org/simple/grpcio/
              << 200 OK 112.03k
127.0.0.1:62309: clientconnect
127.0.0.1:62309: GET https://files.pythonhosted.org/packages/1f/ea/664c589ec41b9e9ac6e20cc1fe9016f3913332d0dc5498a5d7771e2835af/grpcio-1.12.1-cp36-cp36m-manylinux1_x86_64.whl
              << 200 OK 8.57m
127.0.0.1:62307: GET https://pypi.org/simple/six/
              << 200 OK 3.34k
127.0.0.1:62309: GET https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
              << 200 OK 10.45k
127.0.0.1:62309: clientdisconnect
127.0.0.1:62307: clientdisconnect
127.0.0.1:62311: clientconnect
127.0.0.1:62311: GET https://pypi.org/simple/grpcio/
              << 200 OK 112.03k
127.0.0.1:62313: clientconnect
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/1f/ea/664c589ec41b9e9ac6e20cc1fe9016f3913332d0dc5498a5d7771e2835af/grpcio-1.12.1-cp36-cp36m-manylinux1_x86_64.whl
              << 200 OK 8.57m
127.0.0.1:62311: GET https://pypi.org/simple/six/
              << 200 OK 3.34k
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
              << 200 OK 10.45k
127.0.0.1:62315: clientconnect
127.0.0.1:62315: GET https://pypi.org/pypi/six/json
              << 200 OK 5.94k
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz
              << 200 OK 29.16k
127.0.0.1:62315: GET https://pypi.org/pypi/grpcio/json
              << 200 OK 164.26k
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/5c/73/5e65b81301956bdd32c5e8da691fde3fbd6e61283b65d2bac590b8f43765/grpcio-1.12.1-cp27-cp27m-win32.whl
              << 200 OK 1.35m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/e1/c3/bcce8247da4e6f95a900489b6f7ff3d14d93df40d69875fe4164c1b9544a/grpcio-1.12.1-cp27-cp27mu-manylinux1_i686.whl
              << 200 OK 8.01m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/ed/89/03924c56e9044b0842a014fcc0a81f55975028d1caa9cdd14234a230bc70/grpcio-1.12.1-cp27-cp27m-win_amd64.whl
              << 200 OK 1.35m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/d7/f6/ddeab13c25b8451f05875587801ad87e4e0fc23c4e3eb328c4bd1a80a415/grpcio-1.12.1-cp36-cp36m-linux_armv7l.whl
              << 200 OK 7.77m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/2d/a4/4d1d73c0339e987ea173f44cf62ec6b40fb91e0336c09c960c4a44137552/grpcio-1.12.1-cp35-cp35m-linux_armv7l.whl
              << 200 OK 7.76m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/76/27/b03ec8fc96745cde68d6ec29115f9a444945a6acc45209c5772378cc4d66/grpcio-1.12.1-cp35-cp35m-macosx_10_7_intel.whl
              << 200 OK 1.83m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/30/24/8e247548321e52c266a639b51a838ec19b41fb6bfd27e3bbef018496752e/grpcio-1.12.1-cp27-cp27m-manylinux1_x86_64.whl
              << 200 OK 8.47m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/80/c9/e582b962a4a3aa2684666ff67fc994a042b1b0e444eb6672eb9740f7b59a/grpcio-1.12.1-cp34-cp34m-macosx_10_7_intel.whl
              << 200 OK 1.84m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/a2/25/6d910070a4a07c32633c2376075d5dc03e90f69f855d700e3f73c1affebb/grpcio-1.12.1-cp27-cp27m-macosx_10_12_x86_64.whl
              << 200 OK 1.57m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/33/38/58f3e8d133de1f2e911206ead03799621205079c303ae5b27e7350051f4a/grpcio-1.12.1.tar.gz
              << 200 OK 13.56m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/68/57/da122cbfc1b7815381480b23044fff06b90f58c1be9310e68c2d6b1d623c/grpcio-1.12.1-cp36-cp36m-macosx_10_7_intel.whl
              << 200 OK 1.82m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/c6/b8/47468178ba19143e89b2da778eed660b84136c0a877224e79cc3c1c3fd32/grpcio-1.12.1-cp35-cp35m-manylinux1_x86_64.whl
              << 200 OK 8.55m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/5d/8b/104918993129d6c919a16826e6adcfa4a106c791da79fb9655c5b22ad9ff/grpcio-1.12.1-cp36-cp36m-win_amd64.whl
              << 200 OK 1.37m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/94/6c/02e9cb803cd7b9608c9c1768d86d31c61b088f5b9513a203c10fa7e905d8/grpcio-1.12.1-cp36-cp36m-win32.whl
              << 200 OK 1.12m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/2a/ed/71169dccb7f9250d17031068579832371a72891d8e64891265370ca6e264/grpcio-1.12.1-cp27-cp27mu-linux_armv7l.whl
              << 200 OK 7.68m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/63/38/d73bf5b1ef950dbab8203122b9681137b35012492ecfec56719be109e343/grpcio-1.12.1-cp27-cp27m-manylinux1_i686.whl
              << 200 OK 8.01m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/13/71/87628a8edec5bffc86c5443d2cb9a569c3b65c7ff0ad05d5e6ee68042297/grpcio-1.12.1-cp36-cp36m-manylinux1_i686.whl
              << 200 OK 8.11m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/1d/0d/146582f71161a0074dda2378617ae5f7e2c3d6cf62d4588eb586c1d6b675/grpcio-1.12.1-cp27-cp27mu-manylinux1_x86_64.whl
              << 200 OK 8.47m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/9e/3a/6aceb4fccacf6d2d7d087190c221a90f14b2bdcb56cbee5af24b7050278b/grpcio-1.12.1-cp34-cp34m-win_amd64.whl
              << 200 OK 1.35m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/f9/fa/a0187d220544b744dd3bb0d8b8ec716d130159160bf627415b2880ae599a/grpcio-1.12.1-cp34-cp34m-win32.whl
              << 200 OK 1.35m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/dd/aa/ac8e3c6badf1744f04be7d35fa95dae56df12b956f861285c8cced2a22cb/grpcio-1.12.1-cp34-cp34m-linux_armv7l.whl
              << 200 OK 7.76m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/38/2a/94665daafbcf0214adcf77ad8f5aed8b9dfcbfa871115c7890d88b1b8f3c/grpcio-1.12.1-cp34-cp34m-manylinux1_x86_64.whl
              << 200 OK 8.58m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/0d/33/22ad4a9dcefe330180cdb2d24fdd980af2a7a2dc03af208a408fd48195e0/grpcio-1.12.1-cp35-cp35m-win_amd64.whl
              << 200 OK 1.36m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/b5/13/9e8e5d68a15c51b251e512955a971214fd8425b237e6d6a04f0257c5d090/grpcio-1.12.1-cp34-cp34m-manylinux1_i686.whl
              << 200 OK 8.11m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/21/41/66ab386c65be68b4e907f2cd35223965aea2a086bcd0bd6825999e0bda7c/grpcio-1.12.1-cp35-cp35m-win32.whl
              << 200 OK 1.12m
127.0.0.1:62313: GET https://files.pythonhosted.org/packages/f7/db/fc084f59804a32a8d6efb467896a505f4dc93ea89ec44da856b91f05a5cb/grpcio-1.12.1-cp35-cp35m-manylinux1_i686.whl
              << 200 OK 8.09m
127.0.0.1:62313: clientdisconnect
127.0.0.1:62311: clientdisconnect
127.0.0.1:62315: clientdisconnect

Counting up some of the unnecessary requests:

  • 4 x win32
  • 4 x arm
  • 4 x macosx

Etc. Seems like a quick win to do some simple filtering based on host os and arch?

jkp picture jkp on 5 Jun 2018
😄1

You want things filtered by host and architecture, most others want a lockfile that includes hashes that will allow them to install on those hosts and architectures (we have a significant number of hacks in the pip resolver codebase which specifically enable this, so it is not news to us)

Regarding the JSON api, it's not actually enabled to hit directly in the current release, and I am planning to disable it in the codebase again before we release. I have done extensive profiling and I know that failed calls to packaging.version.parse() account for something like 20-50% of pipenv's runtime.

Pip 10 uses the JSON API by default as its main resolver however, so unless you want to stop using pip there isn't much to do on that front.

techalchemy picture techalchemy on 6 Jun 2018

I feel like we must be downloading the same things multiple times, right?

techalchemy picture techalchemy on 6 Jun 2018

We should probably move the discussion to #2284. It is actually the locking part that is slow (install is essentially TOML manipulation + lock + sync), not installing.

uranusjr picture uranusjr on 6 Jun 2018
👍2

Regarding locking down to a single-arch. I understand the design choice; perhaps there could be an option to pass a flag to allow a user to lock only for the host architecture? This would be a pretty significant optimisation both in terms of time, and bandwidth for some use-cases.

jkp picture jkp on 6 Jun 2018

@techalchemy thanks for your response. The packaging.version.parse() find seems like a good lead. Could you put some more colour on this statement:

Regarding the JSON api, it's not actually enabled to hit directly in the current release, and I am planning to disable it in the codebase again before we release.

I didn't understand why you are planning to disable it?

jkp picture jkp on 6 Jun 2018

@jkp Regarding the JSON API, letā€™s just say it is not the best designed thing out there. The simple API is much more suitable for us. By disabling it weā€™re using the simple API, not using no APIs altogether.

uranusjr picture uranusjr on 6 Jun 2018

It's taking too long for me to install Pyspark.
My Pipfile -

[[source]]
name = "pypi"
verify_ssl = true
url = "https://pypi.org/simple"

[dev-packages]
pylint = "*"
pyspark = "*"

[requires]
python_version = "3.5"

shell output - 

Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...

Process is stuck on the last line above.
Finishes after 15-20 mins

pipenv, version 2018.7.1

keshavkaul picture keshavkaul on 4 Jul 2018

@keshavkaul PySpark is very large, and can take quite some time just to download. Give it some time, it will be much better afterwards (because Pipenv caches the result).

(Or you can urge the developers to release a wheel distribution. That would help a bit.)

uranusjr picture uranusjr on 4 Jul 2018
👍1

Note for future visitors: Please refrain from posting your slow installation result. We know it can be slow. We know why it is slow. Your result does not add anything to this topic.

uranusjr picture uranusjr on 4 Jul 2018
👎16 👍11 ❤3

Would it be possible to have some information or progress bar like apt-get or wget (download speed, size downloaded, total size) during the libraries downloads?
I guess this is the issue here, pipenv seemed slow for me but it was just the library download, I had to open a system monitor to understand that pipenv was downloading the files and how much was already downloaded, what speed etc

alexandre-paroissien picture alexandre-paroissien on 10 Jul 2018
👍16

hava same issue: Locking [packages] dependencies... hang forever
my environment:

  • macOS High Sierra 10.13.6
  • Python: Python 3.6.4
  • pipenv: version 2018.7.1
crifan picture crifan on 23 Jul 2018
👍7 👎4

@crifan there is no need to post the same message on every issue open or closed which mentions locking speed. We will see your comment no matter how many times you say the same thing. If you want to be helpful, you will need to provide a reproducible example case. Chiming in to say ā€œme tooā€ simply doesnā€™t add anything besides extra traffic on the issue tracker. Please be mindful of that.

techalchemy picture techalchemy on 23 Jul 2018
👎4 👍3

Same here. Pipenv very slow.
Takes an hour to lock and install.

ScarletMcLearn picture ScarletMcLearn on 20 Sep 2018
👍15 👎7

I think this issue has been answered well here: https://github.com/pypa/pipenv/issues/1914#issuecomment-378846926

Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass --skip-lock.

  • by @techalchemy
whyboris picture whyboris on 25 Sep 2018
👍2

Would it be possible to get the list of hashes from the PyPI API, rather than compute them ourselves?

meshy picture meshy on 25 Sep 2018
👍1

pipenv is awesome, but this issue seems to be still existing. will be glad to see any progress. --skip-lock did not work.

imhazige picture imhazige on 22 Oct 2018
👍17

pipenv is awesome, but this issue seems to be still existing. will be glad to see any progress. --skip-lock did not work.

Worked for me

slyang-git picture slyang-git on 1 Nov 2018

I found that using Git Bash on Windows was very slow compared to Powershell. I don't have any statistics or data on this, but PS seemed faster. So if you are using Git Bash you may want try the native PS for pipenv'ing.

BramVanroy picture BramVanroy on 18 Nov 2018

I know this issue is closed, But for me while installing pandas taking lot of time. The verbose output is this 

pipenv install pandas --verbose
Installing pandasā€¦
ā ‹ Installing...Installing 'pandas'
$ ['/Users/sinscary/.local/share/virtualenvs/signzy-MSzur11z/bin/pip', 'install', '--verbose', '--upgrade', 'pandas', '-i', 'https://pypi.org/simple']
Adding pandas to Pipfile's [packages]ā€¦
āœ” Installation Succeeded
Pipfile.lock not found, creatingā€¦
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
ā ¦ Locking...

It's stuck at Locking for more than 30 minutes. I am using python 3.7.0, macos mojave. Any help with that.

sinscary picture sinscary on 6 Jan 2019
👍18

Why are all the issues to this topic closed? I can't pipenv install a single thing due to the lock-step hang.

black-snow picture black-snow on 8 Jan 2019
👍55

The following docker image takes more than 30 minutes to build on my laptop (i7/16Gb), the pipenv install ... command runs for ages...

Dockerfile

FROM python:3.7-alpine

# Update package list.
RUN set -ex && apk update

# Install apk dependencies.
RUN set -ex && apk add --no-cache musl-dev gcc libffi-dev openssl-dev make

# Install Pipenv.
RUN set -ex && pip install pipenv --upgrade

# Copy Pipfiles.
RUN mkdir /website
COPY Pipfile /website

# Install Pipenv dependencies.
WORKDIR /website
RUN set -ex && pipenv install --system --skip-lock --verbose

Pipfile

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[requires]
python_version = "3.7"

[packages]
sanic = "*"
jinja2 = "*"
asyncpg = "*"
uvloop = "*"
munch = "*"

[dev-packages]

[pipenv]
allow_prereleases = true

Is this normal? Can someone reproduce?

Update: Be careful with Alpine Linux

I realized that the issue is not on pipenv's side...

I replaced the Alpine base docker-image with one that is built on Debian-Slim and now pipenv install finishes within seconds.

The issue in my example is that Alpine Linux will always try to build packages which contain cython-extension or c-extensions from source, which can take forever in a Docker container, whereas Debian Linux installs them using the wheel format, which happens A LOT faster (within seconds).

More on this: https://stackoverflow.com/questions/49037742/why-does-it-take-ages-to-install-pandas-on-alpine-linux

feluxe picture feluxe on 2 Oct 2019
👍5

I have long left pipenv and whenever i need to create virtual env in use "venv" or other options.

rajkumarGosavi picture rajkumarGosavi on 4 Oct 2019

Having a weird slowness issue as well, only 2 modules for some scripting I am doing. 

click

Took 15/20 min to install, internet tests at over 60Mbps down and running on an up to date MacBook Pro 2019 (not my choice of hardware).

Akin2Silver picture Akin2Silver on 8 Oct 2019

Please use virtualenv for time being. until better solution available

rajkumarGosavi picture rajkumarGosavi on 8 Oct 2019

99% of the time I do this, the dependencies will resolve to the same one in my lock file, this is because it's part of my dev pipeline.

In the case where there are no new upstream packages since the last run, surely the process could be skipped ?

stuaxo picture stuaxo on 7 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ansrivas picture ansrivas  Ā·  3Comments
jerzyk picture jerzyk  Ā·  3Comments
AkiraSama picture AkiraSama  Ā·  3Comments
xi picture xi  Ā·  3Comments
jakul picture jakul  Ā·  3Comments