pip install --user misbehaves inside venv

[raffomania]

Environment

• pip version: 18.0
• Python version: 3.7
• OS: Arch Linux, updated on 12. August 2018

Description
When using pip install --user inside a venv (created using the venv module coming with python), packages get installed in my ~/.local folder instead of the venv.
My pip.conf usually sets this flag per default, since I don't install packages globally. This bug makes that config option unusable.

Expected behavior
I would expect venv/bin/pip list to include the ipdb package. Instead, the package is installed outside of the venv.

How to Reproduce

python -m venv venv
venv/bin/pip install --user ipdb
venv/bin/pip list

Output

venv/bin/pip install --user ipdb
Collecting ipdb
  Using cached https://files.pythonhosted.org/packages/80/fe/4564de08f174f3846364b3add8426d14cebee228f741c27e702b2877e85b/ipdb-0.11.tar.gz
Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages (from ipdb) (39.0.1)
Collecting ipython>=5.0.0 (from ipdb)
  Using cached https://files.pythonhosted.org/packages/f7/62/2fef7db3a7b75e8099c3d9db2630ae5ba0b9eefefd91f7497862393d90e8/ipython-6.5.0-py3-none-any.whl
Collecting pickleshare (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/9f/17/daa142fc9be6b76f26f24eeeb9a138940671490b91cb5587393f297c8317/pickleshare-0.7.4-py2.py3-none-any.whl
Collecting traitlets>=4.2 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl
Collecting backcall (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/84/71/c8ca4f5bb1e08401b916c68003acf0a0655df935d74d93bf3f3364b310e0/backcall-0.1.0.tar.gz
Collecting pexpect; sys_platform != "win32" (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl
Collecting simplegeneric>0.8 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip
Collecting jedi>=0.10 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/3d/68/8bbf0ef969095a13ba0d4c77c1945bd86e9811960d052510551d29a2f23b/jedi-0.12.1-py2.py3-none-any.whl
Collecting decorator (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Collecting pygments (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl
Collecting prompt-toolkit<2.0.0,>=1.0.15 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/04/d1/c6616dd03701e7e2073f06d5c3b41b012256e42b72561f16a7bd86dd7b43/prompt_toolkit-1.0.15-py3-none-any.whl
Collecting ipython-genutils (from traitlets>=4.2->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl
Collecting six (from traitlets>=4.2->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting ptyprocess>=0.5 (from pexpect; sys_platform != "win32"->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl
Collecting parso>=0.3.0 (from jedi>=0.10->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/09/51/9c48a46334be50c13d25a3afe55fa05c445699304c5ad32619de953a2305/parso-0.3.1-py2.py3-none-any.whl
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.15->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
Installing collected packages: pickleshare, ipython-genutils, six, decorator, traitlets, backcall, ptyprocess, pexpect, simplegeneric, parso, jedi, pygments, wcwidth, prompt-toolkit, ipython, ipdb
  Running setup.py install for backcall ... done
  Running setup.py install for simplegeneric ... done
  Running setup.py install for ipdb ... done
Successfully installed backcall-0.1.0 decorator-4.3.0 ipdb-0.11 ipython-6.5.0 ipython-genutils-0.2.0 jedi-0.12.1 parso-0.3.1 pexpect-4.6.0 pickleshare-0.7.4 prompt-toolkit-1.0.15 ptyprocess-0.6.0 pygments-2.2.0 simplegeneric-0.8.1 six-1.11.0 traitlets-4.3.2 wcwidth-0.1.7

venv/bin/pip list
Package    Version
---------- -------
pip        18.0   
setuptools 39.0.1 

[bulletmark]

Geepers, I didn't know what the heck was going on until I just found this bug! Seems a serious bug. Also on Arch with Python 3.7.

[pradyunsg]

Given the confirmation from another user and the obvious severity of the bug, I've marked this as a blocker for 18.1; even though it's ~2 months away. :P

[bulletmark]

BTW, the bug happens with both pip versions 10.0.1 and 18.0 for me.

[pradyunsg]

Great; thanks for checking that @bulletmark! :)

[lorencarvalho]

@raffomania @bulletmark am I understanding correctly that you _expect_ pip to _ignore_ the --user flag if it's being ran inside a virtualenv? Just want to clarify.

[pradyunsg]

pip should refuse to install:

[lorencarvalho]

@pradyunsg yeah that's what I would expect

[piotr-dobrogost]

Erroring out is the plan described in issue #4575 "Adopting "working" scheme for every run":

Only packages in the same scheme as the working scheme can be modified. By modifying, I mean installing or uninstalling a package. Trying to modify a package in a different scheme is not allowed and pip would print a message and error out.

[bulletmark]

The issue of this bug is not to do with explicit specification of --user on the command line. The issue is that a user having user mode set in his ~/.config/pip/pip.conf currently can NEVER use pip inside a virtualenv, because it just ignores the virtualenv and installs to ~/.local. That is why I described the bug above as "serious". It seems to me that pip should ignore the user mode setting in ~/.config/pip/pip.conf when it is run inside a virtualenv.

Whether pip errors out on an explicit --user when run within a virtualenv, or just installs to ~/.local as the user has explicitly asked, is another matter and another debate. Personally I think pip in that case should respect the added --user.

It is arguably a design error that by default pip installs globally (see the long running and still open debate in issue #1668) so many users just set the user default in their config and forget about it, like I did a month or so ago then scratched my head for ages why my next virtualenv was not getting set up correctly.

[piotr-dobrogost]

This is duplicate of issue #4141 (Behavior of --user in virtualenv), which mentions error message shown also by @pradyunsg in his comment. It's strange OP did not get this error.

@bulletmark

Different behavior based on where an option was specified (configuration file vs command line) would be very odd.

[pfmoore]

@piotr-dobrogost Agreed. I'd be against having behaviour differ based on where the user specifies --user. What about an environment variable? Should a persistent environment variable be treated differently than PIP_USER=yes pip install foo? I don't think we want to start down this route...

I understand that people see setting --user in a config file as being somehow different in intention, as more of a choice to default to user than a request to always force user, but that's just not what it is.

many users just set the user default in their config

And that's the problem - setting user=true in the pip config file is not "setting the user default", rather it's "asking pip to use --user for all installs", which is subtly (but significantly) different. I guess there may be some value in adding a warning note to the docs explaining that point.

Having user installs be the default is what #1668 is about, and discussions about that should take place over there.

[bulletmark]

@pfmoore, so you are happy with the current situation that if you set user in your config you can NEVER pip install in a virtualenv?! That is the important problem here. What happens if you specify --user on the command line when running in a virtualenv is another less important issue.

[bulletmark]

@piotr-dobrogost, as I said (my paragraph 2) that is a less important issue. I'd also be happy if that case just reported an error to the user. The important issue is my paragraphs 1 and 3.

[piotr-dobrogost]

@pfmoore

I would argue issue #1668 is orthogonal to this one. The problem here (and in many issues related to user scheme) is pip not establishing current working scheme, which is the problem raised in issue #4575.

[pfmoore]

@piotr-dobrogost Good point. See my comment on #4575 - there's a need for a "local,user" option for the working scheme, which says "local if possible, otherwise user". But apart from that detail, yes this is essentially #4575.

@bulletmark No, I'm not happy with the current situation[1], but I don't agree that the correct fix is to break the current behaviour that specifying options in the config file works exactly the same as specifying them on the command line.

[1] Actually, on a personal basis, I am perfectly happy with the current situation, as I don't specify user in my pip config, because it doesn't do the right thing :smile: (and anyway, I avoid global installs anyway, both system and user...) But I agree, my working practices aren't the point here.

[pradyunsg]

For the non-usability in a venv, I think it's reasonable to modify the message to say that "if you've modified configuration files to include user = true, put user = false in your virtualenv config file". That's a workaround at best but I think it's the least effort change.

IMO the bug here is that isn't pip printing the error message, i.e. treating the value set in a configuration file differently from a CLI option. That's a severe bug, as it's (silently) breaking the isolation of the virtualenv.

Other than that, the rest of the discussion here is either related to user-by-default (#1668) or related to mixing of schemes (#4575). Let's keep have those discussions there.

[pradyunsg]

Could someone provide clearer instructions to reproduce this issue?

[bulletmark]

Pip should not be used globally but since user mode is not the default (as per https://github.com/pypa/pip/issues/1668) people want to set the user option in their ~/.config/pip/pip.conf. However if you do this you can not use pip with virtual environments because the packages get installed into your ~/.local instead of the venv. Thus as the OP says here, the "user" config file option is useless, and very confusing to users. Pretty simple and shocking bug.

[pradyunsg]

Dropping this from 18.1, since this is not something introduced in 18.0 and there's no clear plan on how to fix this within pip anyway.

[nkrot]

I have run into this issue with pip 19.0.3 and 19.2.2. Spent the whole day trying to figure out why packages do not get installed while in the virtual environment. It is a pity that such important issues are not fixed. I think we need another PEP for this situation... :(

[pradyunsg]

Could someone provide clearer instructions to reproduce this issue?

Reiterating what I said 11 months ago.

PEP

We don’t need a PEP here. We need someone to explain how to reproduce this issue, so that someone can debug it.

Packaging PEPs are needed for issues that affect multiple tools, and a broader design discussion is needed about interoperability.

[nkrot]

Does the following serve as a demonstration?

1. Install Ubuntu 18.04, install systems pythons (2.7 and 3.6.8), install system pip. system pip is v.9.0.1

2. install/upgrade pip in the user space. If i am not mistaken, i did it with the following command:

$ pip install --user --upgrade pip
$ pip --version
=> 19.2.3

1. scenario that works (you can skip it)

$ mkdir bigbang; cd bigbang
$ python3 -m venv pip-no-user

$ . pip-no-user/bin/activate
$ which pip
=> /path/to/bigbang/pip-no-user/bin/pip

$ pip --version
=> 9.0.1 # this is system pip in ubuntu 18.04

$ pip install --upgrade pip
# upgrades the version inside virtual environment

$ pip --version
=> 19.2.3

$ pip list | grep tabulate
=> nothing, as expected

$ pip install tabulate
$ pip list | grep tabulate
=> tabulate      0.8.3 

# lets do a closer test by loading the library
$ python
>>> import tabulate
=> OK

# type ctrl+d to exit repl

$ deactivate

conclusion: as expected, the package tabulate was installed in the virtual environment

1. configure pip to always perform user install. this is accomplished by adding the following to the configuration file $HOME/.pip/pip.conf

[install]
user = true

Lets do some testing

$ pip list | grep tabulate
=> nothing

$ pip install tabulate
$ pip install tabulate
=> Requirement already satisfied: tabulate in HOME/.local/lib/python3.6/site-packages (0.8.3)

$ pip list | grep tabulate
=> tabulate      0.8.3 

$ python3
>>> import tabulate
=>OK

Conclusion: the package was successfully installed into user space without explicitly giving --user option to pip

1. Based on the setup outlined in 4). This is the failing scenario: when in virtual environment, the libraries are not installed in the environment but rather in user space (HOME/.local).

$ python3 -m venv pip-user-true
$ . pip-user-true/bin/activate
$ pip --version
=> pip 9.0.1

At this point one could run

$ pip install --upgrade pip

but i am not doing it, because it fucks up HOME/.local/bin/pip by changing shebang to be `

!/path/to/bigbang/pip-user-true/bin/python3`

The question is why HOME/.local/bin/pip was updated if environment-specific pip should have been updated, namely this one: /path/to/bigbang/pip-user-true/bin/pip.

Next we check if tabulate (installed in step 4 in user space) is available:

$ pip list |grep tabulate
=> nothing

$ python
>>> import tabulate
=> ModuleNotFoundError: No module named 'tabulate'

Conclusion: as expected, virtual environment isolates current project from user space.

And finally the painful part.

$ pip install tabulate
=> Collecting tabulate
=> Cache entry deserialization failed, entry ignored
=> Installing collected packages: tabulate
=> Successfully installed tabulate-0.8.3

Weird message 'Cache entry deserialization failed, entry ignored'. Heh.

$ pip list | grep tabulate
=> nothing... weird

$ python
>>> import tabulate
=> ModuleNotFoundError: No module named 'tabulate'

It smells bad. Lets continue the journey. Installing tabulate library also drops a command line tool. It should be in the environment bin /path/to/bigbang/pip-user-true/bin/tabulate but in reality it is found in user bin:

$ ls HOME/.local/bin/tabulate
=> ok

Lets look at the script more closely:

$ head -n1 HOME/.local/bin/tabulate
=> #!/path/to/bigbang/pip-user-true/bin/python3

This is weird! Shebang is environment-specific but the script is in user space. Something is wrong here.

And the last chance. Try loading the library we (think we) have just installed.

$ python
>>> import tabulate
ModuleNotFoundError: No module named 'tabulate'

hmmm. It is not available in the environment.

1. Scratching the head... Undoing modifications in .pip/pip.conf for the benefit of everything.

Sorry for the long read.

[pfmoore]

Thanks, that makes things much clearer.

To summarise, --user doesn't work properly when used within a virtual environment. Normally, the obvious workaround is "don't do that", but when you have --user forced on by a config file (or environment variable) it's much less straightforward to "not do that". Is that a reasonable summary?

Your example didn't explicitly demonstrate using an explicit --user command line option from within an activated virtual environment, but I assume it would work like case (5).

The fix for the underlying problem is under discussion in #4575. The fact that there's no --no-user option to override user=true in a config file is an omission, but it's only really needed because people put user=true in their config, and in reality that doesn't do what they want it to do (which is to make pip work as if we'd implemented #1668 (user as default)) so --no-user is a case of piling workaround on top of workaround.

[nkrot]

To summarise, --user doesn't work properly when used within a virtual environment. Normally, the obvious workaround is "don't do that", but when you have --user forced on by a config file (or environment variable) it's much less straightforward to "not do that". Is that a reasonable summary?

yes. this is what happens. I would be almost happy if the documentation of pip and/or venv at least makes a note in bold/red that --user scheme does not play well with venv. Even better if invocation of python -m venv envname throws a warning when it detects that the user scheme is activated (either --user or by a configuration option). I think informing the user would already reduce the amount of wasted time.. Before there is a final solution for this issue.

[pradyunsg]

Thank you for that! It helps a lot!

--user is supposed to abort when installing from within an activated virtualenv. That seems to not be happening.

I'll look into why that's happening and print an error message instead of doing the wrong thing.

[pradyunsg]

Is this venv specific or does this also happen with virtualenv?

[pradyunsg]

Is this venv specific or does this also happen with virtualenv?

Okay, this is venv specific.

Basically, pip was never updated to also look at PEP 405's system-site-packages mechanism. I'll file a PR to fix this in a bit.

FWIW, it's a matter of updating the function virtualenv_no_global() in pip, to actually look do the checks for PEP 405-based virtual environments.

[pradyunsg]

^ #7155 filed for fixing this issue.

If someone could try it out and let me know if that works well for them, that'll be great!

I'd like to avoid fighting our test suite if that change isn't sufficient.

[jonathonf]

If someone could try it out and let me know if that works well for them, that'll be great!

With #7155, pip install --user inside a venv now throws the expected error "Can not perform a --user install. ..."

[pradyunsg]

Awesome! Thanks for the confirmation @jonathonf!

Now, I need to figure out how to fix our build isolation test. :)

[pradyunsg]

pip install --user inside a venv

Expect a release in Jan 2020 with this fix.

Thanks everyone for your patience around this. This fix was "deep in the weeds" and hits a bunch of edge cases w.r.t. how the development workflow of pip works. :)