Updating a lockfile can be very slow. A standard pipenv lock easily takes well over a minute for me:
$ time pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (abef76)!
real 1m56.988s
user 0m21.805s
sys 0m2.417s
This means every time I need to install, uninstall or upgrade a package I need to take a 2-minute break to wait for pipenv to finish updating its lockfile. I'm not sure why that is; yarn and npm seem to perform a similar task but only take seconds, even for projects that have many many more dependencies.
wichert
All 47 comments
We are aware and have many issues tracking this topic. See #1785 #1886 #1891 and PR #1896
npm and yarn have the advantage of not having to fully download and execute each prospective package to determine their dependency graph because the dependencies are specified in plaintext. Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass --skip-lock.
Closing to track in the other issues.
techalchemy
on 5 Apr 2018
Same here im trying it out right now and installing django is already at 10 minutes and going
djangorobert
on 4 Jul 2018
seems that if you try and install a couple at a time its slow but if you do one at a time works a bit faster
djangorobert
on 4 Jul 2018
If you can provide a Pipfile that reproduces the slow behavior it would be helpful -- thanks
techalchemy
on 4 Jul 2018
Python dependencies require us to fully download and execute the setup files of each package to resolve and compute
Is this still the case when the package has its own Pipfile.lock, or will pipenv use that when possible to determine dependencies?
dideler
on 17 Jul 2018
will pipenv use that when possible to determine dependencies?
No. Pipenv is an application depedency resolution tool. When a dependency is used as a Python package by another application, it is no longer an application. Its dependencies are determined by setup.py (or setup.cfg or whatever you use to upload to PyPI). Loading dependencies from a lock file is a sure route to dependency hell, we will likely never ever do that.
uranusjr
on 17 Jul 2018
It's still super slow
iddan
on 9 Aug 2018
@iddan Thanks for the reminder, Captain Obvious!
uranusjr
on 9 Aug 2018
Sorry, as an OSS maintainer I know how sometimes issues can get dismissed because of age. Just wanted to state that even the issue received discussion it’s still relevant
iddan
on 9 Aug 2018
@techalchemy note of --skip-lock above is wonderful. This should be a more accessible or publicized option. Can we set it as a default somewhere?
statueofmike
on 13 Aug 2018
@techalchemy what if it takes 20 minutes with my brand new mac pro?
CauchyPolymer
on 1 Oct 2018
@techalchemy note of
--skip-lockabove is wonderful. This should be a more accessible or publicized option. Can we set it as a default somewhere?
As far as I gather, the overwhelming benefit of pipenv is the assurance dependencies will play nicely together — not just for you, but for anyone later dealing in your code. The product of that assurance, the lock file, absolutely takes more time than anyone expects or desires, _including the devs_ — see #2200.
However, I think you can also understand the opportunity pipenv has to shepherd well-meaning devs in the Python community at large toward a workflow imposing less head-scratching on future contributors — who might've only been visitors had they given up during the "figure out how to setup the dev environment" stage; and less hands thrown up by future maintainers — who might've only been drive-by PR authors had they given up during the "seriously screwing around with deep project internals" stage.
If --skip-lock were to become a permanent flag in a Pipfile or a setting in a pipenv config, pipenv's perception would slowly slide toward "better pip", and just another stepping stone fading into the horizon as the community eventually landed on a less compromising spiritual successor.
Better to leave it available only as an env var, or some other method whose application rests squarely in the _"your user-specific local config, your fault"_ territory, allowing pipenv to overcome the passing phase of lockfile generation slowness without giving up the truly beneficial cultural shift toward _explicitness over implicitness_ in package management.
Python's incredibly vast standard library is an enormous asset, whose history has undergone many eras of imposing consistency. That most standard packages play nicely together is an enormous feat involving consideration over many years by many people. One day, that play-nice-ability will extend to most Python projects encountered on the web — far, far from the stdlib, and with far, far fewer PEPs required (_and far, far fewer BDFLs vacating in frustration_). The impact of such a unilaterally buttery experience is hard to measure, but some current languages _did_ refuse to compromise conceptual integrity for immediate convenience... and oh, the places they'll Go.
So _yes_, generating the lockfile is slow, and _yes_, it's frustrating when you only wanted pip install --save. But it's only because we've been sweeping an elephant into the closet for years — believing we didn't have a tangled mess of expectations and intentions from external dependencies, because _"it installed fine on my machine"_.
Lockfile generation is slow _only_ because it's making explicit what we've all taken for granted. But _because_ it hurts, we will adjust things so it doesn't. We broke our arm because we pushed ourselves doing things we believed in. Sure, we can avoid the pain by never using that arm again— or we can put it in a cast while it heals.
I'd be the last person to tell you not to make pipenv convenient for yourself today (otherwise I'd be a shitty developer — _though, the jury's still out_), but I implore you to see the frustration of lockfile generation as a growing pain while the Python community develops into a strong, healthy body with more fully-functioning limbs than one really expected when removing a cast. We made it to Python 3. Let's make it to dependency management in the stdlib.
theY4Kman
on 13 Oct 2018
This took 38 minutes on my machine to create the lock file. Running Windows 10 and using Python 3.7
sjjpo2002
on 5 Nov 2018
Only numpy and pillow were already installed and it took <1 seconds to install numba and 25 minutes to lock it. Does pipenv forcibly compile every lib on lock or how does this work?
vanyakosmos
on 11 Nov 2018
FYI persistent skip lock is just waiting on someone to flip the switch for auto_envvar_prefix which is a click setting. I've been 100% focused on core functionality so I haven't had a chance to look at this yet but I suspect it isn't that difficult
techalchemy
on 12 Nov 2018
TLDR; Typical pipenv install invocation: Time: 144.82 real 33.68 user 5.77 sys. With --skip-lock: Time: 4.54 real 5.33 user 0.87 sys.
Pandas-datareader install fails on first attempt, possibly cause of lock hanging. Is this an issue anyone else is seeing with other packages?
Using version 2018.11.26
$ pipenv --version
pipenv, version 2018.11.26
Contents of requirements.txt
sklearn
pandas
numpy
matplotlib
statsmodels
Typical pipenv install invocation. Timed execution using time (BSD).
Results: 144.82 real 33.68 user 5.77 sys
$ time pipenv install -r requirements.txt
Requirements file provided! Importing into Pipfile…
Pipfile.lock (0fdb67) out of date, updating to (a65489)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
âś” Success!
Updated Pipfile.lock (0fdb67)!
Installing dependencies from Pipfile.lock (0fdb67)…
🎅 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 15/15 — 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
144.82 real 33.68 user 5.77 sys
Invoking w\ --skip-lock
Results: 4.54 real 5.33 user 0.87 sys
$ time pipenv install -r requirements.txt --skip-lock
Requirements file provided! Importing into Pipfile…
Installing dependencies from Pipfile…
🎅 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
4.54 real 5.33 user 0.87 sys
zazazack
on 25 Dec 2018
I think https://github.com/pandas-dev/pandas/ may be a problem? It's a common point with the time for me too.
Although pytest may also be an issue :\
jufemaiz
on 8 Jan 2019
It won't even finish on my machine:
Installing pandas…
Adding pandas to Pipfile's [packages]…
Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Traceback (most recent call last):
File "c:\python36\lib\site-packages\pipenv\vendor\pexpect\expect.py", line 109, in expect_loop
return self.timeout()
File "c:\python36\lib\site-packages\pipenv\vendor\pexpect\expect.py", line 82, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x00000292ADCCCDD8>
searcher: searcher_re:
0: re.compile('\n')
black-snow
on 8 Jan 2019
@black-snow I'd recommend trying it in a different shell. Without diving too deep into things, it seems like pexpect (a library for programmatically interfacing with interactive CLI programs) is used to detect the shell pipenv is being executed from, and that might be stalling. pexpect kinda seems overkill for such a thing, but I'm not aware of the whole context.
theY4Kman
on 12 Jan 2019
@theY4Kman thanks for the advice. pipenv is working fine on another pc with same ubuntu and bash version ...
black-snow
on 12 Jan 2019
Read this while waiting for pipfile to lock...:) Would be great if there was a solution.
kstohr
on 16 Jan 2019
Seems like we need some kind of API to parse and cache Python packages deps and distribute it in a machine-friendly format. So we will no longer need to download entire packages and parse them.
uetchy
on 17 Jan 2019
I believe bundler and ruby gems maintains and uses somthing like that.
jufemaiz
on 17 Jan 2019
Java also has POM files (XML) which contain package dependencies and other information about the package. They are uploaded separately from the compiled JARs.
danielcompton
on 17 Jan 2019
Every newer package manager has some separate meta files (npm/yarn, composer, gradle/maven, cargo, ruby gems/bundler, ...).
black-snow
on 18 Jan 2019
related issue:
https://github.com/pypa/warehouse/issues/474
uetchy
on 24 Jan 2019
You can get dependency information from PyPi without downloading the entire bundle (see PEP 566, which superseded PEP 426).
package_name = 'Django'
PYPI_API_URL = 'https://pypi.python.org/pypi/{package_name}/json'
package_details_url = PYPI_API_URL.format(package_name=package_name)
response = requests.get(package_details_url)
data = json.loads(response.content)
data['info'].get('requires_dist')
data['info'].get('requires')
data['info'].get('setup_requires')
data['info'].get('test_requires')
data['info'].get('install_requires')
brandonrobertz
on 28 Jan 2019
@techalchemy did you see the comment above?
jourdanrodrigues
on 1 Mar 2019
This is pretty consistently happening, essentially pipenv going away around town while "locking" something: why is this issue closed?
Understand the --skip-lock is the way to go, but it's not clear at all why "installing" takes a few seconds, and "locking" takes forever: it would be great if at least the lock command would emit some progress/update logs: as things stand, it's not even clear if it just got stuck in some sort of while True forever...
I'd at least like to know if it's me doing something wrong, or just a "feature" of pipenv.
massenz
on 3 Apr 2019
In our project pipenv lock is so so so slow. It definitely impacted our normal use. Adding a new package becomes a real pain for us now. Is there anyway we can debug this behavior?
upwell
on 29 May 2019
I'm trying to install PyTorch and it took 20 minutes to lock and then it pulls the following error
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]: File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]: skip_lock=skip_lock,
[pipenv.exceptions.InstallError]: File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]: pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]: File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]: retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]: File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]: _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]: File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]: raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting pytorch==1.0.2 (from -r /tmp/pipenv-pb00kf8t-requirements/pipenv-vae35p2d-requirement.txt (line 1))', ' Using cached https://files.pythonhosted.org/packages/ee/67/f403d4ae6e9cd74b546ee88cccdb29b8415a9c1b3d80aebeb20c9ea91d96/pytorch-1.0.2.tar.gz', 'Building wheels for collected packages: pytorch', ' Building wheel for pytorch (setup.py): started', " Building wheel for pytorch (setup.py): finished with status 'error'", ' Running setup.py clean for pytorch', 'Failed to build pytorch', 'Installing collected packages: pytorch', ' Running setup.py install for pytorch: started', " Running setup.py install for pytorch: finished with status 'error'"]
[pipenv.exceptions.InstallError]: ['ERROR: Complete output from command /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/bin/python3 -u -c \'import setuptools, tokenize;__file__=\'"\'"\'/tmp/pip-install-hix3yz6v/pytorch/setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' bdist_wheel -d /tmp/pip-wheel-f_h8w1pz --python-tag cp36:', ' ERROR: Traceback (most recent call last):', ' File "<string>", line 1, in <module>', ' File "/tmp/pip-install-hix3yz6v/pytorch/setup.py", line 15, in <module>', ' raise Exception(message)', ' Exception: You tried to install "pytorch". The package named for PyTorch is "torch"', ' ----------------------------------------', ' ERROR: Failed building wheel for pytorch', ' ERROR: Complete output from command /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/bin/python3 -u -c \'import setuptools, tokenize;__file__=\'"\'"\'/tmp/pip-install-hix3yz6v/pytorch/setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' install --record /tmp/pip-record-xr7o93_5/install-record.txt --single-version-externally-managed --compile --install-headers /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/include/site/python3.6/pytorch:', ' ERROR: Traceback (most recent call last):', ' File "<string>", line 1, in <module>', ' File "/tmp/pip-install-hix3yz6v/pytorch/setup.py", line 11, in <module>', ' raise Exception(message)', ' Exception: You tried to install "pytorch". The package named for PyTorch is "torch"', ' ----------------------------------------', 'ERROR: Command "/home/alex/.local/share/virtualenvs/pytorch-umelu-tG/bin/python3 -u -c \'import setuptools, tokenize;__file__=\'"\'"\'/tmp/pip-install-hix3yz6v/pytorch/setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' install --record /tmp/pip-record-xr7o93_5/install-record.txt --single-version-externally-managed --compile --install-headers /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/include/site/python3.6/pytorch" failed with error code 1 in /tmp/pip-install-hix3yz6v/pytorch/']
ERROR: ERROR: Package installation failed...
The error is unreadle, no idea what went wrong. Installing with pip in the environment work fine! This is really a show stopper. Going back to requirements.txt...
awhillas
on 12 Jun 2019
This is the workaround I use for now:
export PIPENV_SKIP_LOCK=true
Then pipenv install foo won't be locking and you can lock manually when you have time by running pipenv lock.
shtratos
on 18 Jun 2019
@awhillas Pretty sure the last line says all you need:
You tried to install "pytorch". The package named for PyTorch is "torch"
BramVanroy
on 25 Jul 2019
Locking dependencies is important so I don't think "skip lock" is a lasting solution. At the same time, I simply don't buy that "locking dependencies" (whatever that could possibly entail under the hood) is maximally optimized as it is now and functionally requires many minutes or hours to complete. Indeed, my pipenv lock ran for several minutes on a Pipfile that has a laughable 5 dependencies before failing—stack attached at bottom—and during that time used only 10–15% of available CPU and a little sip of memory.
Can we at least put forth some group effort to profile and determine the bottlenecks? I have a feeling there are just some silly low hanging fruit in there just waiting to take this process into reasonable runtime.
pipenv version 2018.11.26
For Pipfile:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
[dev-packages]
keras = "*"
tensorflow = "~=1.13"
setuptools = "*"
wheel = "*"
twine = "*"
[requires]
python_version = ">=3.6"
Pipfile.lock (63b275) out of date, updating to (5e165c)…
Locking [dev-packages] dependencies…
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 109, in expect_loop
return self.timeout()
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x105a17210>
searcher: searcher_re:
0: re.compile('\n')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/bin/pipenv", line 11, in <module>
load_entry_point('pipenv==2018.11.26', 'console_scripts', 'pipenv')()
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/cli/command.py", line 254, in install
editable_packages=state.installstate.editables,
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
skip_lock=skip_lock,
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1221, in do_init
pypi_mirror=pypi_mirror,
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1068, in do_lock
lockfile=lockfile
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 649, in venv_resolve_deps
c = resolve(cmd, sp)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 517, in resolve
result = c.expect(u"\n", timeout=environments.PIPENV_INSTALL_TIMEOUT)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/delegator.py", line 215, in expect
self.subprocess.expect(pattern=pattern, timeout=timeout)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 341, in expect
timeout, searchwindowsize, async_)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 369, in expect_list
return exp.expect_loop(timeout)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 119, in expect_loop
return self.timeout(e)
File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x105a17210>
searcher: searcher_re:
0: re.compile('\n')
evictor
on 12 Aug 2019
As an addendum to previous comment, running pip lock separately took a reasonable amount of time, ~15 seconds, after first running pip install --skip-lock. So perhaps the post-install invocation of lock is out of date or otherwise faulty. :)
evictor
on 12 Aug 2019
FYI I found tensorflow is the culprit of the slow/timing out lock, if that helps profile pipenv! (Still consider this a pipenv problem...)
evictor
on 12 Aug 2019
Tensorflow is one of many packages that can cause pipenv to basically become a useless tool. I like the suggestion of group profiling, though. I think that's an excellent idea to start tacking this problem. Just to re-iterate, PEP 566 enabled enumerating dependence (via pypi) without loading the entire source, which may be helpful: https://github.com/pypa/pipenv/issues/1914#issuecomment-457965038
brandonrobertz
on 13 Aug 2019
@brandonrobertz from what I see, downloading all packages of the dependencies is where most time is spent. This also has been confirmed before.
How to verify this:
- Try creating a new
Pipfileand installingscipy(for example) in it with locking enabled - Wait until locking completes. (Takes about 5 minutes on my machine)
- remove
Pipfile.lock - run
pipenv lock- locking now will take very little time (6 seconds on my machine), because all packages are already downloaded and kept in Pipenv cache which is normally in~/.cache/pipenv
Here's the Dockerfile I used to test this:
FROM python:3.6
ENV WORKDIR=/work/
WORKDIR /work/
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install pipenv
RUN PIPENV_SKIP_LOCK=true pipenv install scipy
RUN date
RUN pipenv lock
RUN date
RUN rm Pipfile.lock
RUN pipenv lock
RUN date
@shtratos Yeah, that makes sense and others have suggested that in this issue thread. Downloading and parsing dependencies is expensive. Some of those steps seem like they can be eliminated by pulling from the pypi dependency API.
For some libraries this will probably not work, due to poor quality and bad practices (not using setup.py or requirements.txt). But since some of the major offenders seem to be very popular libraries (tensorflow, numpy), implementing this with a fallback to the super slow process might be a good path forward.
brandonrobertz
on 15 Aug 2019
Can you point me in the direction of where to find that code? I could take a stab at parallelizing it in a fork.
evictor
on 9 Sep 2019
I think https://github.com/pandas-dev/pandas/ may be a problem? It's a common point with the time for me too.
Although
pytestmay also be an issue :\
I doi not think so, it works fine on my machine, the problem seems to be more general than that
yassineameur
on 20 Feb 2020
In my case the problem seems to be pylint. It always hangs on locking when simply running pipenv install pylint see https://github.com/pypa/pipenv/issues/2284#issuecomment-569457752
I have the same problem in all of my projects.
The cause seems to be pylint.
Pipenv (pip) can install it successfully, but locking takes forever!
pipenv, version 2018.11.26Minimal working example
djbrown@DESKTOP-65P6D75:~$ mkdir test djbrown@DESKTOP-65P6D75:~$ cd test djbrown@DESKTOP-65P6D75:~/test$ pipenv install --dev pylint --verbose Creating a virtualenv for this project… Pipfile: /home/djbrown/test/Pipfile Using /usr/bin/python3 (3.6.9) to create virtualenv… ⠸ Creating virtual environment...Already using interpreter /usr/bin/python3 Using base prefix '/usr' New python executable in /home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/python3 Also creating executable in /home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/python Installing setuptools, pip, wheel...done. ✔ Successfully created virtual environment! Virtualenv location: /home/djbrown/.local/share/virtualenvs/test-PW-auWy_ Creating a Pipfile for this project… Installing pylint… ⠋ Installing...Installing 'pylint' $ ['/home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/pip', 'install', '--verbose', '--upgrade', 'pylint', '-i', 'https://pypi.org/simple'] Adding pylint to Pipfile's [dev-packages]… ✔ Installation Succeeded Pipfile.lock not found, creating… Locking [dev-packages] dependencies… ⠇ Locking...
djbrown
on 20 Feb 2020
We are aware and have many issues tracking this topic. See #1785 #1886 #1891 and PR #1896
npm and yarn have the advantage of not having to fully download and execute each prospective package to determine their dependency graph because the dependencies are specified in plaintext. Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass
--skip-lock.Closing to track in the other issues.
3 out of the 4 other issues are now closed and the remaining one hasn't seen any activity since 2018. This issue still persists, so maybe re-opening it would be a good idea?
vlsd
on 17 Aug 2020
Python dependencies require us to fully download and execute the setup files of each package to resolve and compute
I don't think that is still true for wheels, which should be the majority of packages now?
wichert
on 18 Aug 2020
I know I at least have to build the wheel for dlib every time, which is horrendous.
The process of resolving dependencies should be cached somewhere on a per-package-version basis, even if it requires another remote lookup on the client each time to get the tree (it shouldn't, you could just cache it locally too after the fact). For example, package repos could easily store resolved dep trees. The additional network hit for the cached resolved dep tree would always be faster than downloading an entire package and computing.
evictor
on 18 Aug 2020
FWIW I've removed pipenv from all my projects (this being the one main reason, there are others).
virtualenv + pip (with requirements.txt) now work pretty well, even for Prod deployments; and in any event, one deploys a fully-formed container these days; after getting really into pipenv, I no longer see the point of it.
massenz
on 7 Sep 2020
Please reopen this issue.
Otherwise pipenv will never be the reference packaging tool
dpinol
on 29 Sep 2020
Related issues
jerzyk
·
3Comments
AkiraSama
·
3Comments
Californian
·
3Comments
jakul
·
3Comments
randName
·
3Comments
Most helpful comment
Read this while waiting for pipfile to lock...:) Would be great if there was a solution.