pipenv install times out while creating virtualenv

Created on 24 Jan 2017 · 25Comments · Source: pypa/pipenv

Running pipenv --three install fails at Creating a virtualenv for this project with the output:

Creating a virtualenv for this project...
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 97, in expect_loop
    return self.timeout()
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7fe6f4e65f28>
searcher: searcher_re:
    0: EOF

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/pipenv", line 9, in <module>
    load_entry_point('pipenv==0.2.8', 'console_scripts', 'pipenv')()
  File "/usr/lib/python3.4/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.4/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 393, in install
    ensure_project(dev=dev, three=three)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 76, in ensure_project
    ensure_virtualenv(three=three)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 57, in ensure_virtualenv
    do_create_virtualenv(three=three)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 166, in do_create_virtualenv
    click.echo(crayons.blue(c.out))
  File "/usr/lib/python3.4/site-packages/delegator.py", line 78, in out
    self.__out = self._pexpect_out
  File "/usr/lib/python3.4/site-packages/delegator.py", line 66, in _pexpect_out
    result += self.subprocess.read().decode('utf-8')
  File "/usr/lib/python3.4/site-packages/pexpect/spawnbase.py", line 413, in read
    self.expect(self.delimiter)
  File "/usr/lib/python3.4/site-packages/pexpect/spawnbase.py", line 321, in expect
    timeout, searchwindowsize, async)
  File "/usr/lib/python3.4/site-packages/pexpect/spawnbase.py", line 345, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 107, in expect_loop
    return self.timeout(e)
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7fe6f4e65f28>
searcher: searcher_re:
    0: EOF
<pexpect.popen_spawn.PopenSpawn object at 0x7fe6f4e65f28>
searcher: searcher_re:
    0: EOF

The .venv directory is created and subsequent use of pipenv install [package] work correctly.

Relevant package versions:

delegator.py==0.0.7
pexpect==4.2.1
pipenv==0.2.8
virtualenv==15.1.0

reported bug

Source

jstutters

👍6

Most helpful comment

It would be good to make a reference to PIPENV_TIMEOUT environment variable in that exception.

nuno-andre on 13 May 2018

👍10

All 25 comments

interesting, cannot reproduce locally

kennethreitz42 on 24 Jan 2017

@jstutters did you by any chance have any networking problems at the moment that could potentially lead to packages needed for the initial virtualenv to take too long to download.

How to reproduce:
Modify virtualenv to add a more than 30 second sleep without outputing anything.

pipenv --three install

@kennethreitz Are you using the default timeout of 30 seconds for pexpect in delegator? Does that make sense in this case? Shouldn't it probably have a very larger timeout to accommodate slow connections or even not have a timeout at all and let the user stop it if he thinks its stuck somewhere?

spiliopoulos on 25 Jan 2017

I am, but there's no reason creating a virtualenv should take more than 30s. I suspect a setup issue is at hand.

kennethreitz42 on 25 Jan 2017

@jstutters can you time the creation of a new virtualenv by hand?

kennethreitz42 on 25 Jan 2017

@kennethreitz It's on a networked home directory which is certainly a potential source of the slowness (sorry- should have said this in the original report). Execution time for making the virtualenv manually is 49 seconds so I guess a timeout increase would help. I had a look at doing this myself but it didn't seem from the docs that delegator exposes the pexpect timeout argument.

jstutters@host ~/test % /usr/bin/time -f %E virtualenv -p /usr/bin/python3 foo
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/jstutters/test/foo/bin/python3
Also creating executable in /home/jstutters/test/foo/bin/python
Please make sure you remove any previous custom paths from your /home/jstutters/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.
0:48.59

jstutters on 25 Jan 2017

Okay, so right off the bat I do think we should leave the timeout by default at 30 seconds, because as @kennethreitz mentioned there's not many cases in which this is not enough time. However, there are use cases when it may be useful (or even necessary).

In the do_create_virtualenv function, we make the call delegator.run(cmd, block=False).

If we go take a look at delegator/delegator.py, it makes the call PopenSpawn(self._popen_args, **self._default_pexpect_kwargs). Viewing the documentation for PopenSpawn, we see the constructor takes an optional timeout parameter

def __init__(self, cmd, timeout=30, maxread=2000, searchwindowsize=None, logfile=None,  
             cwd=None, codec_errors='strict')

So if we're wanting to set a custom timeout for PopenSpawn, then we would need to modify the delegator constructor or delegator.run to take an optional timeout=30 parameter, or something to that effect. Then we would be able to pass in the value through Pipenv in whatever fashion is deemed suitable.

jessebraham on 26 Jan 2017

I think 30s is reasonable too.

kennethreitz42 on 26 Jan 2017

It would be awesome if there was a fix for this issue. For me creating a virtualenv takes 40 seconds when I have my antivirus running. Disabling it drops the time to 2 seconds and then pipenv works fine, but that's not really an acceptable solution as disabling antivirus company-wide is not an option.

Randati on 5 Mar 2017

@Randati, can you confirm for me if creating a new environment with the virtualenv command or pew new command takes the same amount of time as pipenv --three?

nateprewitt on 5 Mar 2017

Yes, they all take about the same time: pipenv --three 50 s, virtualenv 40 s, pew new 40 s. For pipenv I increased the timeout manually to be able to measure it. All of them are a couple seconds if antivirus is turned off.

Randati on 5 Mar 2017

So I'm a bit confused what it is about the virtualenv creation that's being delayed so drastically by your AV. This feels like more of an issue with virtualenv, but we could implement a "fix" in Delegator.py. That's a separate project of @kennethreitz, so I'm going to tag him in to see if he has thoughts here.

Bumping the timeout makes for a worse experience for people with a legitimate failure, but will increase tolerance for cases like this. Given that we've only had two users encounter this though, I'm not sure how common it really is.

nateprewitt on 6 Mar 2017

👍1

I am having this problem as well, also with a slow network home directory. A way of overriding the default timeout or specifying where the venvs are created would be very useful. It takes around 30s to create the venv normally so it only fails sometimes.

A hugely useful script otherwise. Thanks.

tpoliaw on 9 Mar 2017

I ran into this problem too. pipenv times out if I have an intensive process running in the backround, but works fine if I kill the background process. Would prefer an option to override the default timeout.

cdetrio on 11 Mar 2017

This happens for me on AWS t2.x instances types, and likely will continue to happen inside containers as well where limitations are exaggerated. While 30s is reasonable, not allowing configuration through something as easy as an env variable is pretty unflexible

myoung34 on 17 Apr 2017

➜ pipenv --two
Creating a virtualenv for this project...
Traceback (most recent call last):
File "/usr/local/bin/pipenv", line 11, in
sys.exit(cli())
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
return self.main(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1043, in invoke
return Command.invoke(self, ctx)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
return callback(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 756, in cli
ensure_project(three=three, python=python)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 135, in ensure_project
ensure_virtualenv(three=three, python=python)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 115, in ensure_virtualenv
do_create_virtualenv(three=three, python=python)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 377, in do_create_virtualenv
click.echo(crayons.blue(c.out), err=True)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/delegator.py", line 78, in out
self.__out = self._pexpect_out
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/delegator.py", line 66, in _pexpect_out
result += self.subprocess.read().decode('utf-8')
File "/usr/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 413, in read
self.expect(self.delimiter)
File "/usr/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 321, in expect
timeout, searchwindowsize, async)
File "/usr/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 345, in expect_list
return exp.expect_loop(timeout)
File "/usr/local/lib/python2.7/site-packages/pexpect/expect.py", line 107, in expect_loop
return self.timeout(e)
File "/usr/local/lib/python2.7/site-packages/pexpect/expect.py", line 70, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT:
searcher: searcher_re:
0: EOF

searcher: searcher_re:
0: EOF

Don't know why it takes so long normally:

➜ time virtualenv whatever
New python executable in /.../whatever/bin/python2.7
Also creating executable in /.../whatever/bin/python
Installing setuptools, pip, wheel...done.
virtualenv whatever 2.58s user 0.71s system 7% cpu 42.808 total

rhymes on 19 Apr 2017

@kennethreitz and I had previously discussed adding a timeout option, or bumping the timeout on delegator.py. If he still feels that's a good way to go, we can look into a patch for this.

nateprewitt on 19 Apr 2017

I am having this problem as well

liangshinegood on 2 Jan 2018

👍2

I'm experiencing the same issue as well but I solved it. I tried switching wifi networks because the problem is somehow related to the firewall setup. Didn't bother to ask IT as it will take time.

asumawang on 14 Feb 2018

It was a networking problem for me. VPN was blocking it somehow. Better error message would be handy

awhillas on 17 Apr 2018

👍1

Contributions on better error messages would be welcomed!

uranusjr on 19 Apr 2018

It would be good to make a reference to PIPENV_TIMEOUT environment variable in that exception.

nuno-andre on 13 May 2018

👍10

I was on my phone's internet, and when I got back on the good wifi, this error went away.

Eunoia on 28 Jul 2019

This solution solved my problem: https://github.com/pypa/pipenv/issues/2681#issuecomment-504469464 (i.e. "using PIP_NO_CACHE_DIR=off when running pipenv install")

hminooei on 10 Oct 2019

Got similar problem with timeout. It waited for around 20 minutes, then failed:

positron@ubox:~/Documents$ pipenv install mesa
Creating a virtualenv for this project…
Pipfile: /home/positron/Documents/Pipfile
Using /usr/bin/python3 (3.5.2) to create virtualenv…
⠏ Creating virtual environment...Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/positron/.local/share/virtualenvs/Documents-rjeAWyrt/bin/python3
Also creating executable in /home/positron/.local/share/virtualenvs/Documents-rjeAWyrt/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment! 
Virtualenv location: /home/positron/.local/share/virtualenvs/Documents-rjeAWyrt
Creating a Pipfile for this project…
Installing mesa…
Adding mesa to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
⠇ Locking...
⠙ Locking...
⠸ Locking...
Traceback (most recent call last):
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 109, in expect_loop
    return self.timeout()
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7f870d98f1d0>
searcher: searcher_re:
    0: re.compile('\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/positron/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/cli/command.py", line 254, in install
    editable_packages=state.installstate.editables,
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/core.py", line 1992, in do_install
    skip_lock=skip_lock,
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/core.py", line 1244, in do_init
    pypi_mirror=pypi_mirror,
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/core.py", line 1068, in do_lock
    lockfile=lockfile
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/utils.py", line 649, in venv_resolve_deps
    c = resolve(cmd, sp)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/utils.py", line 517, in resolve
    result = c.expect(u"\n", timeout=environments.PIPENV_INSTALL_TIMEOUT)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/delegator.py", line 215, in expect
    self.subprocess.expect(pattern=pattern, timeout=timeout)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 119, in expect_loop
    return self.timeout(e)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7f870d98f1d0>
searcher: searcher_re:
    0: re.compile('\n')
<pexpect.popen_spawn.PopenSpawn object at 0x7f870d98f1d0>
searcher: searcher_re:
    0: re.compile('\n')

Pipenv is 2018.11.26
OS is xubuntu 16.04.3 x64