pipenv only works in an activated shell

[mmerickel]

Reproducible example:

$ python3 -m venv env
$ env/bin/pip install pipenv
<snip>
$ env/bin/pipenv install pyramid
Creating a Pipfile for this project...
Creating a virtualenv for this project...
Traceback (most recent call last):
  File "env/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==4.1.4', 'console_scripts', 'pipenv')()
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/cli.py", line 801, in install
    ensure_project(three=three, python=python)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/cli.py", line 155, in ensure_project
    ensure_virtualenv(three=three, python=python)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/cli.py", line 135, in ensure_virtualenv
    do_create_virtualenv(three=three, python=python)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/cli.py", line 388, in do_create_virtualenv
    c = delegator.run(cmd, block=False)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/delegator.py", line 247, in run
    c.run(block=block, binary=binary)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/delegator.py", line 146, in run
    s = PopenSpawn(self._popen_args, **pexpect_kwargs)
  File "/Users/michael/work/oss/pipenv/env/lib/python3.6/site-packages/pipenv/vendor/pexpect/popen_spawn.py", line 45, in __init__
    self.proc = subprocess.Popen(cmd, **kwargs)
  File "/Users/michael/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/Users/michael/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'pew'

Expected: A new pipenv virtualenv is created with pyramid installed and the pipfile updated similar to how things work if I run:

$ python3 -m venv env
$ env/bin/pip install pipenv
$ source env/bin/activate
(env) $ pipenv install pyramid

This is a common problem with programs that shell out without activating the virtualenv but it does make things a bit more difficult for people who do not need to activate 99% of the time and prefer being explicit about what python they are using. There are several workarounds but one is to prepend the virtualenv's bin/scripts folder to the path before shelling out via os.path.join(os.environ('VIRTUAL_ENV'], bin_or_scripts).

[nateprewitt]

Hey @mmerickel, thanks for taking the time to open this ticket. So I'm a bit confused on what you're trying to do here. What are you hoping to accomplish by creating a virtual environment inside of a second virtual environment? I'm not saying there isn't a bug here, this is just a use of pipenv we haven't seen before.

Usually you'd have pipenv installed at a system level, like you would your pip install. Then you'd run pipenv install pyramid inside of your project directory and it establishes your virtualenv and Pipfile for you.

[mmerickel]

@nateprewitt A couple notes:

• I install all python libraries/scripts into virtualenvs. One way of doing this is with pipsi. Users should never install stuff into the system python without really good reason (maybe that is a future for pipenv but it's not quite ready for that level of trust). Pipenv also fails in this scenario because the pew script is not linked into the bin folder... and I don't believe pipenv's dependencies should pollute my system path or system python site-packages.

• Pipenv itself doesn't fully support being installed into a system location as it does not define the python it's running against (pipenv-2.7 versus pipenv-3.6). I always care which python interpreter I'm using because in general those decisions affect the defaults used by those tools. For example if I use python3 -m virtualenv versus python2 -m virtualenv I get a virtualenv coupled to python 3 versus python 2 respectively - unless I specify the exact --python binary I want. pipenv already supports --python and so it seems confusing / magical to also support --three versus --two which are harder to guess what they will do... I'd argue those options should be thrown away totally and just stick with --python to be consistent with the virtualenv package instead of relying on pipenv to find the "right" one for me.

• Assuming pipenv was properly isolated how could I be sure that it's using the right pew that it was tested against? The only way is for pipenv to try harder to execute the script from the same environment... pew is a library dependency after all, not a system dependency - meaning pipenv can control which versions it's compatible with.

Sorry for the long response but the TLDR is that if pipenv wants to be installed globally on a system with multiple pythons it needs to be a bit more careful about how it interacts with its dependencies.

As a concrete example, if you compare this to yarn - I've never been concerned that yarn would run a command installed from a different version of node... it's isolated to the node it's installed into. Even if it wanted to run a shell command exported from another dependency I'd be reasonably confident it'd run the right one independent of my NODE_PATH or whatever else.

[nateprewitt]

I install all python libraries/scripts into virtualenvs. One way of doing this is with pipsi. Users should never install stuff into the system python without really good reason (maybe that is a future for pipenv but it's not quite ready for that level of trust). Pipenv also fails in this scenario because the pew script is not linked into the bin folder... and I don't believe pipenv's dependencies should pollute my system path or system python site-packages.

pipenv has historically been intended to be a global tool by default, rather than a library, and pew expects to work in that paradigm. This is to allow all of your virtualenvs to exist in a single location. I personally would be alright with going back to pew as an optional dependency, but that is an ongoing discussion.

As noted in the docs, you can set PIPENV_VENV_IN_PROJECT to use virtualenv to create a venv inside the project instead of using pew. That will prevent the error you're receiving.

Pipenv itself doesn't fully support being installed into a system location as it does not define the python it's running against (pipenv-2.7 versus pipenv-3.6).

So I'm not sure I understand this. pipenv will be installed with whatever version of python you have bound to pip. Once pipenv is installed, it shouldn't matter which version of python it's bound to unless you're actively modifying your system default. The tool is written to be version agnostic, so it can be used for Python 2.6-2.7 and 3.3-3.6.

I always care which python interpreter I'm using because in general those decisions affect the defaults used by those tools. For example if I use python3 -m virtualenv versus python2 -m virtualenv I get a virtualenv coupled to python 3 versus python 2 respectively - unless I specify the exact --python binary I want. pipenv already supports --python and so it seems confusing / magical to also support --three versus --two which are harder to guess what they will do... I'd argue those options should be thrown away totally and just stick with --python to be consistent with the virtualenv package instead of relying on pipenv to find the "right" one for me.

--two/--three are directly analogous to your examples of python2 -m virtualenv and python3 -m virtualenv. I don't know if it's any more magic than the symlink for python2 or python3.

Assuming pipenv was properly isolated how could I be sure that it's using the right pew that it was tested against? The only way is for pipenv to try harder to execute the script from the same environment... pew is a library dependency after all, not a system dependency - meaning pipenv can control which versions it's compatible with.

That's true. The issue here is you're executing pipenv from an arbitrary location. It's looking for pew in your environment to install the virtual environment as a globally accessible directory. However, you're quarantining it to a directory off the path, defeating the purpose of pew. Unless you modify your path, activate your top-level venv, or disable pew, this will be an issue. This same problem exists if you were using an equivalent tool like virtualenv though, so it's not pew specific.

[mmerickel]

This issue has actually gotten worse since pipenv 5.x was released with auto-detection of virtualenvs. Now if you activate the virtualenv then pipenv will install dependencies into it whereas env/bin/pipenv install pyramid will try to create a new virtualenv. This is just completely unexpected.

I feel like I've already made my points clear in previous posts - they are common sense arguments about isolation of dependencies and conforming to standard workflows similar to pip itself. If you choose to ignore them then things will remain inconsistent. I do not know what pew is and I do not care... I care that pipenv requires it to be on my path and since it is not pipenv no longer works.

[bertjwregeer]

I built a quick tool to solve for this problem of having an unactivated virtualenv from were you'd like to run pipenv:

$ python3 -mvenv env
$ ./env/bin/pip install pipenv vrun
$ ./env/bin/vrun pipenv install pyramid

Source here: https://github.com/bertjwregeer/vrun
PyPI: https://pypi.python.org/pypi/vrun/0.2

This will install everything inside the existing virtualenv: ./env:

Output from an example run

alexandra:pipenvtest xistence$ ~/.pyenv/versions/3.6.1/bin/python -mvenv env
alexandra:pipenvtest xistence$ ./env/bin/pip install pipenv vrun
Collecting pipenv
  Using cached pipenv-5.0.0.tar.gz
Collecting vrun
  Downloading vrun-0.2-py2.py3-none-any.whl
Collecting virtualenv (from pipenv)
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Collecting pew>=0.1.26 (from pipenv)
  Using cached pew-0.1.26-py2.py3-none-any.whl
Requirement already satisfied: pip in ./env/lib/python3.6/site-packages (from pipenv)
Collecting pythonz-bd>=1.10.2 (from pew>=0.1.26->pipenv)
  Using cached pythonz_bd-1.11.4-py3-none-any.whl
Requirement already satisfied: setuptools>=17.1 in ./env/lib/python3.6/site-packages (from pew>=0.1.26->pipenv)
Collecting virtualenv-clone>=0.2.5 (from pew>=0.1.26->pipenv)
  Using cached virtualenv-clone-0.2.6.tar.gz
Collecting resumable-urlretrieve; python_version == "3.6" (from pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Using cached resumable_urlretrieve-0.1.5-py2.py3-none-any.whl
Collecting requests (from resumable-urlretrieve; python_version == "3.6"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Using cached requests-2.17.3-py2.py3-none-any.whl
Collecting idna<2.6,>=2.5 (from requests->resumable-urlretrieve; python_version == "3.6"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Using cached idna-2.5-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->resumable-urlretrieve; python_version == "3.6"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.22,>=1.21.1 (from requests->resumable-urlretrieve; python_version == "3.6"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Using cached urllib3-1.21.1-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->resumable-urlretrieve; python_version == "3.6"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Using cached certifi-2017.4.17-py2.py3-none-any.whl
Installing collected packages: virtualenv, idna, chardet, urllib3, certifi, requests, resumable-urlretrieve, pythonz-bd, virtualenv-clone, pew, pipenv, vrun
  Running setup.py install for virtualenv-clone ... done
  Running setup.py install for pipenv ... done
Successfully installed certifi-2017.4.17 chardet-3.0.4 idna-2.5 pew-0.1.26 pipenv-5.0.0 pythonz-bd-1.11.4 requests-2.17.3 resumable-urlretrieve-0.1.5 urllib3-1.21.1 virtualenv-15.1.0 virtualenv-clone-0.2.6 vrun-0.2
alexandra:pipenvtest xistence$ ./env/bin/vrun pipenv install pyramid
Creating a Pipfile for this project...
Installing pyramid...
Collecting pyramid
  Using cached pyramid-1.8.3-py2.py3-none-any.whl
Collecting venusian>=1.0a3 (from pyramid)
  Using cached venusian-1.1.0-py2.py3-none-any.whl
Collecting translationstring>=0.4 (from pyramid)
  Using cached translationstring-1.3-py2.py3-none-any.whl
Collecting WebOb>=1.7.0rc2 (from pyramid)
  Using cached WebOb-1.7.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from pyramid)
Collecting repoze.lru>=0.4 (from pyramid)
Collecting hupper (from pyramid)
  Using cached hupper-1.0-py2.py3-none-any.whl
Collecting zope.interface>=3.8.0 (from pyramid)
  Using cached zope.interface-4.4.1-cp36-cp36m-macosx_10_6_intel.whl
Collecting zope.deprecation>=3.5.0 (from pyramid)
  Using cached zope.deprecation-4.2.0-py2.py3-none-any.whl
Collecting PasteDeploy>=1.5.0 (from pyramid)
  Using cached PasteDeploy-1.5.2-py2.py3-none-any.whl
Installing collected packages: venusian, translationstring, WebOb, repoze.lru, hupper, zope.interface, zope.deprecation, PasteDeploy, pyramid
Successfully installed PasteDeploy-1.5.2 WebOb-1.7.2 hupper-1.0 pyramid-1.8.3 repoze.lru-0.6 translationstring-1.3 venusian-1.1.0 zope.deprecation-4.2.0 zope.interface-4.4.1

Adding pyramid to Pipfile's [packages]...

[mmerickel]

@kennethreitz this issue hasn't been solved yet. pipenv should do something similar to vrun to ensure that the virtualenv's path (if available) is prepended to the path of the subprocess when executing pew.

[kennethreitz42]

this is done in master.

[bertjwregeer]

https://github.com/kennethreitz/pipenv/commit/4e5f59a5122801bac0f184a6fdc28e41c9b98b6e ?

[kennethreitz42]

yep!

[kennethreitz42]

released.

[mmerickel]

Thanks @kennethreitz I didn't see this fix. Unfortunately it is not working for me and I'm able to reproduce it with the same steps as the original ticket using pipenv 5.4.0.

[kennethreitz42]

Just install it properly: http://docs.pipenv.org/en/latest/advanced.html#fancy-installation-of-pipenv

[mmerickel]

As someone who has contributed to pipsi and watches its repo and uses / maintains a fork I can tell you that it's almost entirely unmaintained... I'm not sure that's a great installation method to recommend to people. It hasn't had a release in over 2 years and the get-pipsi.py script installs the version from pypi, not the one from the repository. On top of that those installation instructions are currently semi-broken if you don't have the virtualenv command on your path. It's of course installed into the python that I'm using to run pipenv, but the virtualenv command should again be found relative to the python that's running, not the system command (similar issue as this one with pew).

Note below the usage of .local/bin/pew from pipsi. I know it's hard to get this all isolated when using subprocesses but I maintain that it's an important thing to do to make a tool that's easily installable without a bunch of assumptions on the installed environment. pipsi is able to find virtualenv but later pipenv/pew cannot

$ pipsi install pew
$ pipsi install pipenv
$ pipenv install
Creating a virtualenv for this project...
⠋Traceback (most recent call last):
  File "/Users/michael/.local/bin/pew", line 11, in <module>
    sys.exit(pew())
  File "/Users/michael/.local/venvs/pew/lib/python3.6/site-packages/pew/pew.py", line 737, in pew
    return command(sys.argv[2:])
  File "/Users/michael/.local/venvs/pew/lib/python3.6/site-packages/pew/pew.py", line 253, in new_cmd
    args.requirements, rest)
  File "/Users/michael/.local/venvs/pew/lib/python3.6/site-packages/pew/pew.py", line 215, in mkvirtualenv
    check_call(["virtualenv", envname] + rest, cwd=str(workon_home))
  File "/Users/michael/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 286, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/Users/michael/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/Users/michael/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/Users/michael/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'virtualenv'

Virtualenv location:
Traceback (most recent call last):
  File "/Users/michael/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/michael/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/michael/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/michael/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/michael/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/michael/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/michael/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/cli.py", line 1057, in shell
    with open(activate_this) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/bin/activate_this.py'

[kennethreitz42]

@mmerickel patches welcome!

[drcongo]

I'm not sure how this is closed - pipenv still behaves differently in an activated venv compared to calling the pipenv installed in that venv directly. I've been trying for months now to get pipenv to play nicely with our Ansible provisioning - with Ansible the recommended way of running a command inside a venv is to call it directly - /path/to/myvenv/pipenv - but this creates a new virtualenv every time.

$ source venv3/bin/activate
$ pipenv install --sequential

Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
Installing dependencies from Pipfile.lock (0d75ff)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 30/30 — 00:00:19

$ deactivate
$ venv3/bin/pipenv install --sequential

Creating a virtualenv for this project…
⠋Using base prefix '/usr'
New python executable in /srv/www/.venv/bin/python3
Also creating executable in /srv/www/.venv/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /srv/www/.venv
Installing dependencies from Pipfile.lock (0d75ff)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 30/30 — 00:00:35
To activate this project's virtualenv, run the following:
    $ pipenv shell

Given that the docs now say that if pipenv finds itself running inside a virtual env it will install to that virtual env - and endless other threads asking for the ability to specify the virtual env location have been closed, pointing to this feature as the solution - I'm lost as to how this issue is closed. It's still not possible to pipenv install via Ansible without changing every single other thing that relies on the venv being called venv3 inside my project folder on every single server.

I kinda love everything else about pipenv, it's amazing. It's just a shame I can't actually use it on real work.

[techalchemy]

@drcongo calling a virtualenv installed version of pipenv will always do one of two things:

1. Create a new virtualenv _using python from the system path_, so the only value to having pipenv in a virtualenv in that case is to isolate it from the system
2. Install packages to the calling virtualenv _if activated_

There is also a third option of activating the virtualenv and setting the environment variable PIPENV_IGNORE_VIRTUALENVS which will create virtualenvs even when one is activated. Alternatively you can use —system and put the relevant pip and python on your path. Does that help at all?

[drcongo]

Thanks @techalchemy, I'm not entirely sure any of that meets our needs. I'm going to experiment with pipenv lock --requirements and then just letting Ansible pip install. I think this gives us all the joy of pipenv for local development and then just fall back to pip when provisioning.