Pipenv: $PATH for shell and run is different

[goern]

Why is $PATH different when using pipenv shell or pipenv run?

[goern@tBook p (develop)]$ pipenv run python
Python 3.6.2 (default, Jul 17 2017, 16:44:47)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ["PATH"]
'/Users/goern/.rvm/gems/ruby-2.3.0/bin:/Users/goern/.rvm/gems/ruby-2.3.0@global/bin:/Users/goern/.rvm/rubies/ruby-2.3.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/usr/local/MacGPG2/bin:/usr/local/opt/go/libexec/bin:/Users/goern/.rvm/bin'
>>>
[goern@tBook piler-dev (develop)]$ pipenv shell
Launching subshell in virtual environment. Type 'exit' or 'Ctrl+D' to return.
[goern@tBook p (develop)]$ echo $PATH
/Users/goern/.local/share/virtualenvs/piler-dev-VN2jPVB_/bin:/Users/goern/.rvm/gems/ruby-2.3.0/bin:/Users/goern/.rvm/gems/ruby-2.3.0@global/bin:/Users/goern/.rvm/rubies/ruby-2.3.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/usr/local/MacGPG2/bin:/usr/local/opt/go/libexec/bin:/Users/goern/.rvm/bin
[goern@tBook p (develop)]$

[techalchemy]

Because only one of them actually modifies your path. Pipenv is a dependency manager, when you use it it does not _activate_ your virtualenv.

pipenv run does not touch your path, it simply uses your virtualenv to look for an executable. In this case, pipenv run python runs the python executable pipenv located in your project virtualenv.

pipenv shell however, promises a subshell with access to the environment. By nature this requires a modification to the path. As you can see yourself, that means it prepends the virtualenv bin to your path to ensure that you will look first in your virtualenv for executables.

So, why is your path different with these commands? Because only one of them modifies your path.

Is this causing you any problems?

[jacebrowning]

I think this is a regression from when this functionality was intentionally added in 5.4.2 ("Compatibility improvement with run"):

$ pipenv --version
pipenv, version 5.4.2

$ pipenv run honcho start
14:44:57 system    | web.1 started (pid=7296)
14:44:57 web.1     | [2017-09-12 14:44:57 -0400] [7296] [INFO] Starting gunicorn 19.7.1
14:44:57 web.1     | [2017-09-12 14:44:57 -0400] [7296] [INFO] Listening at: http://0.0.0.0:5000 (7296)
14:44:57 web.1     | [2017-09-12 14:44:57 -0400] [7296] [INFO] Using worker: sync
14:44:57 web.1     | [2017-09-12 14:44:57 -0400] [7300] [INFO] Booting worker with pid: 7300

vs now:

$ pipenv --version
pipenv, version 6.2.6

$ pipenv run honcho start
14:43:48 system    | web.1 started (pid=7131)
14:43:48 web.1     | /bin/sh: gunicorn: command not found

[kennethreitz42]

It was added but then removed, as it is not necessary.

[kennethreitz42]

We can always add it back in, but would require some thinking.

[jacebrowning]

It's necessary if I want to run any command that might call another command in the virtual environment. https://github.com/kennethreitz/pipenv/issues/378#issuecomment-306855801 outlines another use case: test runners.

---

The workaround is messy and not easy to type:

pipenv shell -c "honcho start; exit $?"

[kennethreitz42]

It was removed because it was causing problems for some users, so the code will need to be carefully re-evaluated. It's commented out, currently.

[kennethreitz42]

oh i spoke too soon!

[kennethreitz42]

pipenv run --system now exists, which changes the path for you.

[kennethreitz42]

$ pipenv run --system honcho start

[jacebrowning]

OK. That seems to work.

The name of the option is a bit misleading to me for the behavior described on this issue:

--system : Fallback to system–available executables.

[kennethreitz42]

That's what it does.

[jacebrowning]

But in this case, I want to use the executables in the virtual environment, not the system.

[techalchemy]

@kennethreitz we should update the argument and documentation to indicate that first it looks in the virtualenv, then it falls back to the system executables

I can understand the confusion, we describe a fallback behavior but don't indicate how that is performed (it is impossible to fallback to system executables unless we are looking for things on the path in the first place, which is why we modify it and check there)

[kennethreitz42]

@jacebrowning you're a special case :)

[goern]

I see, so the trick for https://github.com/kennethreitz/pipenv/blob/master/pipenv/cli.py#L1425-L1427 is really to which(python), where I thought that it is obvious which python to use and skipped it...

[jacebrowning]

It was removed because it was causing problems for some users

Do you recall what those cases where?

To me, having this display anything other than the virtual environment's Python seems strange:

pipenv run python -c "import os; os.system('which python')"

[kennethreitz42]

run venv/bin/python directly, you'll get the same result.

[kennethreitz42]

virtualenv's primary job is to give you a working python installation, in addition to the help it provides you with your shell. pipenv run is also only doing that one thing — giving you a working installation, with minimal interference, and that's why --system was added, for people like you.

[goern]

maybe its a RFE to delegator to add the pipenv managed virtualenv to the path automagically?!

[kennethreitz42]

P.S. if you use full paths, --system is used automatically

[jacebrowning]

I'm hoping that with pipenv I will no longer need to worry about managing virtual environments because pipenv will activate them automatically when needed.

But, if pipenv run python myscript.py _sometimes_ behaves differently than running the script in the manually activated virtual environment, then I'll always have to do the latter and the utility of pipenv run is lost on me. Having pipenv run and pipenv shell produce different results seems to violate the principle of least astonishment.

I get that I might be in the minority here, but I also wonder what harm it would cause if pipenv run added the virtual environment to the path automatically.

[kennethreitz42]

I'd like to have --system be default, but I need more people to try it out first. We had someone report an exception, and that's enough reason to pull it out of default for now.

[goern]

So what is the test case?

pipenv install deptree
pipenv shell deptree

This should put a binary in the virtualenv that should be executable via pipenv shell. Or is that to trivial?

[kennethreitz42]

the test case is $ pipenv run --system deptree.

[techalchemy]

@goern @jacebrowning ideally as Kenneth mentioned the default behavior of pipenv run would be that of pipenv run --system but there was an issue with this in the past. I don't have time to dig it up right now and I suspect he doesn't either but I'd be curious to revisit it if someone can find it

[kennethreitz42]

The code just needs to be rigified.

[kennethreitz42]

e.g. if there's any failure, fallback to our current behavior.

[kennethreitz42]

got this in master

[kennethreitz42]

rigified

[kennethreitz42]

released!

[jacebrowning]

Tested with pipenv==6.2.7. Works great! Thanks @kennethreitz! 🎉

[kennethreitz42]

thanks for your persistience @jacebrowning :)