Pip: Installing system-wide ignores existing user installations

[Jean-Abou-Samra]

• pip version: 20.0.2
• Python version: 3.8.2
• Operating system: Ubuntu (Focal Fossa); confirmed on Mac OS

Using sudo pip to install a package will ignore a previously existing user installation. The package will therefore be duplicated.

What's worse, if the package defines a command, then after a sudo pip install, the command found will still be that provided by the user installation. This can be problematic on Mac OS because ~/.local/bin is not in the PATH by default (as I could understand; at least it was not the case for the Mac OS user we were helping).

Here is an example with the python-ly package, which provides a command named ly. Note how uninstalling once will reveal the duplicate.

$ # Install some package with a command, locally.
$ pip install --user python-ly
Collecting python-ly
  Using cached python-ly-0.9.6.tar.gz (179 kB)
Installing collected packages: python-ly
    Running setup.py install for python-ly ... done
Successfully installed python-ly-0.9.6

$ pip show python-ly
Name: python-ly
[...]
Location: /home/jean/.local/lib/python3.8/site-packages

$ which ly 
/home/jean/.local/bin/ly

$ # Reinstall the same package system-wide.
$ # The existing local installation is not detected.
$ sudo pip install python-ly
[sudo] Mot de passe de jean : 
Collecting python-ly
  Using cached python-ly-0.9.6.tar.gz (179 kB)
Installing collected packages: python-ly
    Running setup.py install for python-ly ... done
Successfully installed python-ly-0.9.6

$ # pip show still tells it's installed locally...
$ pip show python-ly
Name: python-ly
[...]
Location: /home/jean/.local/lib/python3.8/site-packages

$ # ... and the command is still in a local bin !
$ which ly
/home/jean/.local/bin/ly

$ # Let's uninstall it. This removes the local install and reveals the system-wide one.
$ pip uninstall python-ly
Found existing installation: python-ly 0.9.6
Uninstalling python-ly-0.9.6:
  Would remove:
    /home/jean/.local/bin/ly
    /home/jean/.local/bin/ly-server
    /home/jean/.local/lib/python3.8/site-packages/ly/*
    /home/jean/.local/lib/python3.8/site-packages/python_ly-0.9.6-py3.8.egg-info
Proceed (y/n)? y
  Successfully uninstalled python-ly-0.9.6

$ pip show python-ly
Name: python-ly
[...]
Location: /usr/local/lib/python3.8/site-packages

$ which ly
/usr/local/bin/ly

[uranusjr]

I believe this is intentional. System-site installations are meant to be shared by all users using the Python installation, and a user install is only usable by that user. So a user installation should not satisfy a system installation, otherwise it would result in a broken environment for other users on the system.

[Jean-Abou-Samra]

Correct, but I mean that in my humble opinion, the user install should be overridden (uninstalled) when a system installation is performed, or at least a warning should be issued.

This can result in confusing situations. For example, suppose that you install package X with --user, and package X requires package Y==2.1. So you get package Y version 2.3 in your local site-packages. Then sudo pip install Y. It will install the latest version of Y, say 3.1, but if you import it, you will unconsciously be using 2.3.

I do understand that there is a hierarchy between site-packages directories, and this behavior is of course desirable for virtual environments. But in the specific case of system and user site-packages, given that the two different approaches are coexisting, should there be a special case? It is indeed easy to mess up between user and system installs.

[uranusjr]

Sorry for not understanding your problem in the initial comment. I think this is a valid request, maybe what we ought to do is to uninstall a matching user site install ok a system site install, with a message to tell the user why. The user can install the package back into user site if they want to (pip won’t uninstall the matching system site package on --user. There are likely complications in virtual environment situations though, so hopefully others can chime in and fill in things I missed.

[gutsytechster]

Also, I think using sudo with pip is not the recommended way of using pip.

BTW, when you say user-site location, do you mean /home/jean/.local/lib/python3.8/site-packages and when you say system-site location, do you mean /usr/local/lib/python3.8/site-packages?

[McSinyx]

what we ought to do is to uninstall a matching user site install ok a system site install, with a message to tell the user why

I'm not sure if this is possible though:

$ sudo whoami
root

Alternatively, I think we can provide a tool to investigate all current installations. Another similar situation I usually run into is that eggs installed directly by setuptools are not uninstalled when the wheel is installed (even on the same site), so when the wheel is removed the module is still there. This suggestion does not solve this issue though, but provide more insight for users.

[pfmoore]

IMO, pip should focus on managing one installation at a time (system, user, virtualenv, whatever). Managing the interactions between two installations is likely to get very complex, very quickly, and is probably impossible in any but the simplest of cases. Rather than get sucked into that, I'd say that pip's responsibility should end with the targeted environment, and it's the user's responsibility to co-ordinate across environments.

[SanketDG]

From what I can see the step of commands in the original post, I think the problem is the user not being able to understand what is affected and what is not. Even though when a package is installed/uninstalled, the full path is printed, the user needs to read through all the lines to understand that the package is being added/removed into system/user.

Just a random throw out there, but can we atleast label system and user during installing/uninstalling of packages? I am not sure how this will extend to virtualenv and other python environments, but it would act as sort of a visual aid to users to klnow what level of the system is being affected.

Successfully installed python-ly-0.9.6 in system

user is just an example here and not helpful here.

[uranusjr]

@McSinyx sudo is weird:

$ sudo whoami
root
$ sudo python3 -c 'import sysconfig; print(sysconfig.get_path("purelib", scheme="posix_user"))'
/home/uranusjr/.local/lib/python3.6/site-packages

Even if it makes you root, various configurations still “correctly” point to the non-root user. So there is definitely a way (whether it is worthwhile though, I can’t say without digging into the implementation).

[pfmoore]

can we at least label system and user during installing/uninstalling of packages

We may have to give the directory and leave it to the user to infer what environment that is (as you say, virtual environments and the like make a simple system/user distinction hard) but yes, that seems like a useful idea. We probably already report that in the verbose output, but maybe something more visible would be helpful.