Pip: "Detected a distutils installed project" is fatal in pip 8

[rbtcollins]

However distributions still ship a bunch of distutils projects - e.g. six, urllib3, jsonpointer, chardet, libvirt-python, pyOpenSSL, PyYAML, netaddr

So this is likely to affect many users.

A variation of this, where a virtualenv is being used also occurs - this shouldn't have been warning ever, since the virtualenv means we won't uninstall, we'll just shadow it. See #3404 for the bug for that.

We could:

• suck it up and point folk at the distros for a while (5 years, plus or minus) until the current binary packages without the file record are going
• advise folk to use --ignore-installed (which will cause lots of possibly undesired upgrades / network traffic), but _may_ work
• revert it and release 8.1
• provide an option to make it a warning

[dstufft]

Just a bit of background here:

The fundamental problem is that "pure" distutils projects (ones that import distutils rather than setuptools, and which were not installed in a way to force setuptools to be injected, as pip does) when installed do not include all of the relevant metadata. They include information like "foobar v1.0" was installed but they do not include /usr/lib/python3.5/site-packages/foobar/__init__.py belongs to "foobar v1.0". This means that whenever we uninstall these, we don't actually uninstall any files we simply remove the record that something has been installed and then overwrite files.

This generally "works", however it can lead to really weird, strange errors where you end if with foobar.py _AND_ foobar/__init__.py (if for example, foobar switched form a module to a package) or files are left behind that should have been deleted, confusing import mechanisms. So, previously we lied and said we uninstalled something when we actually didn't, then we at least told you that were were lying to you, now finally, we've stopped lying to you and started to refuse to do something we don't have the information to actually do.

Long term projects should stop importing from distutils and switch instead to just unconditionally importing from setuptools. This is the recommended way to function and in 2016 you're almost always guaranteed to have setuptools available on an end users's machine. We may elect to work around it and push the date for this back further, but realistically things need to stop relying on plain distutils.

[petecheslock]

Hit this issue with installing elasticsearch-curator with this error

Detected a distutils installed project ('urllib3') which we cannot uninstall. The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall.

Worked around it by pip install -I pip==7.1.2

[Ivoz]

+1 on not lying to people that we know how to uninstall something when we really don't - literally the reason everyone stopped using easy_install, right?

[dstufft]

Adding to 8.0.1 milestone, we should at least figure out what our answer is going to be before we release 8.0.1.

[pfmoore]

Can we not as a minimum add something to the message saying "If you wish to uninstall FOO you need to manually identify and delete the files associated with the project from site-packages."? That seems like a better workaround than having people downgrade pip...

[ianw]

So I have no doubt everyone understands the issue; but I am seeing something that's easier to reproduce than nebulous devstack jobs. On Ubuntu Trusty the problem is I think coming out of #1570 and that argparse is kind of in the virtualenv and kind of not (it's filtered, i think?)

ubuntu@trusty:/tmp$ virtualenv test
New python executable in test/bin/python
Installing setuptools, pip...done.
ubuntu@trusty:/tmp$ . test/bin/activate
(test)ubuntu@trusty:/tmp$ pip install argparse
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.7
Cleaning up...
(test)ubuntu@trusty:/tmp$ pip install --upgrade argparse
Downloading/unpacking argparse from https://pypi.python.org/packages/2.7/a/argparse/argparse-1.4.0-py2.py3-none-any.whl#md5=c37216a954c8669054e2b2c54853dd49
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: argparse
  Found existing installation: argparse 1.2.1
    Not uninstalling argparse at /usr/lib/python2.7, outside environment /tmp/test
Successfully installed argparse
Cleaning up...

this has now turned into an error

(test)ubuntu@trusty:/tmp$ pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-8.0.0-py2.py3-none-any.whl#md5=7b1da5eba510e1631791dcf300657916
  Downloading pip-8.0.0-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
(test)ubuntu@trusty:/tmp$ pip install -U argparse
Collecting argparse
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: argparse
  Found existing installation: argparse 1.2.1
Detected a distutils installed project ('argparse') which we cannot uninstall. The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall.

So it seems "outside environment" has been caught along with "can't know what files to delete"

[xavfernandez]

I was hit by the same argparse issue. I guess the dist_is_local step should happen before.

[dstufft]

The argparse issue seems like a legit bug in that it shouldn't be getting caught by this error yea?

[daviddyball]

@dstufft, is it not possible to add a CLI argument called --IGNORE-DISTUTILS-ERRORS-YES-I-UNDERSTAND-THIS-IS-NOT-SUPPORTED until everyone switched to setuptools in their installation scripts?

[dstufft]

@daviddyball Yes, and that's a possibility for what we'll do here. Today I'll be poking at the issue and we'll ultimately end up with a decision of some kind for how we proceed.

[wichert]

We hit the exact same issue as @ianw, only with requests instead of argparse.

Isn't the issue here pip trying to uninstall a non-local package? Even if that had all the necessary metadata pip would never be able to do that, so the only sensible thing it can do is suck it up and not error out.

[sigmavirus24]

@wichert is requests already installed by the distro when you try to install requests again?

[wichert]

Because our requirements.txt pins a specific version so we have the same version everywhere.

[dstufft]

Also, folks should file issues with the relevant projects to unconditionally use setuptools in their setup.py. It's 2016 and distutils isn't a reasonable thing to use by itself anymore.

[pfmoore]

so the only sensible thing it can do is suck it up and not error out.

But if pip can't uninstall the old version, when it does the install it could create a corrupt installation. Is it really any better to ignore the error and (potentially) leave the user with an unusable package?

IMO, there isn't really any good answer here.

As far as I can see, this only affects people with packages installed that were not installed using pip. I guess that mostly, these are distro-installed packages, and as such, shouldn't people be using their distro package manager to update these, rather than pip?

[wichert]

The chances of being able to use the distro package manager to get a version of pip, virtualenv, requests, Pillow, etc. to anything more recent than 2 years old is effectively zero. The target audience here are people using something like a Debian or Ubuntu LTS release: they will contain software that is a few years old, and will only receive patches for security fixes.

Now your next question is probably going to be: why don't use use a virtualenv that does not expose system packages? The answer to that is: because there are packages that are not installable by pip. For example Python modules for complex C++ libraries that can't use setuptools for compilation (our situation), or because compilation requires more memory then a machine might have available (happens very often with numpy/scipy on VMs).

[dstufft]

Specifically, it's packages installed that were installed using distutils rather than setuptools. Which means all of the following were true:

• Was not installed using pip.
• Was not installed using easy_install.
• Project either conditionally or unconditionally imports from distutils in setup.py, preferring distutils or setuptools was not installed.

My estimation is that the vast bulk of these are going to be OS packages where either downstream unconditionally uses distutils, or supports both setuptools and distutils and the OS packager chose to use distutils over setuptools.

[pfmoore]

OK, so those scenarios are not ones I'm likely to be affected by, so I don't really have a way of evaluating the options. Are they that common (given that many people will just use a virtualenv)? And are none of the following viable approaches?

• User manually goes in and deletes the offending package's files (before you say "how do I know which they are?", that's the reason pip can't do it, so you're on your own there...)
• You use --ignore-installed to force an install of the specific package (and you then take responsibility for ensuring that the resulting installation is OK)

I'm basically -1 on letting pip do an install that we know is risky, without the user being told about the risks and consequences. It'd be too easy for us to end up dealing with "pip broke my system" tickets. Presumably people have been seeing the warning about this since it was introduced early last year? And presumably the behaviour was deprecated because it had caused issues prior to that?

[wichert]

As I explained we do use a virtualenv, so using virtualenv is not a possible solution :)

The user can't manually delete the offending package, since that would require root access, and is likely to break the OS.

At least as far as we are concerned we just want pip to not try to uninstall something that is not local to the virtualenv. I don't particular mind if that is controlled by a flag, although from a user perspective I probably prefer that pip gives a very clear warning but defaults to skipping the uninstall attempt.

[pfmoore]

I'm clearly missing something here. My apologies.

You're using a virtualenv. OK. Presumably with use_system_site_packages, as otherwise how come pip's seeing the system package? These are packages that can't be installed with pip, I think you said? If so, why is pip trying to install (upgrade) them? If the user's asking pip to install something that you've confirmed can't be installed with pip, then not being able to uninstall the old version is surely the least of your worries?

[wichert]

@pfmoore We have packages that can't be installed using pip, but those are not the ones pip is trying to upgrade.

We use a virtualenv with system site packages so we can use packages that can not be installed using pip. That also pulls in a set of other packages that are installed by the OS such as requests, pillow and psycopg2. As far as we are concerted that is a side-effect if virtualenv had an option to only selectively expose system packages we would use that to exclude those.

We also use exact version pins for all packages in requirements.txt to guarantee that we use the exact same version of all dependencies everywhere. That will include packages that are provided by the OS on some machines for various reasons – for example Ubuntu ships with a bunch of Python packages installed that OSX/MacPorts does not install. When we point pip to that requirements.txt it can try to upgrade packages installed by the OS. That has always worked, but results in a fatal error with pip 8.

I suspect the easiest way to reproduce this is to grab an Ubuntu 14.04 LTS box and do this:

$ sudo apt-get install python-requests  # This will install requests 0.4 or 2.2.1-1ubuntu0.3 (if you run trusty-updates)
$ virtualenv --system-site-packages tst
$ cd tst
$ bin/pip install --upgrade requests==2.9.1

This works fine with pip 7. On pip 8 it will abort with an error.

[pfmoore]

Ah, OK I see now. Sorry for taking so long to understand. Agreed that the "real" solution would be for virtualenv to have an "expose only the following system packages" option.

Maybe what we should do is downgrade the error to a warning if pip is planning to install the new version to a different location than the detected one. (We can't ignore it totally as (a) we don't know which one will take precedence, and (b) there's still the risk of partial shadowing).

Installing a package like this, shadowing another installation, would still be an unsupported arrangement, but at least people who understand the risks and want to (or have to) take them, can do so.

[dstufft]

I think @wichert is hitting something different than the underlying issue here, it just so happened that promoting this to an error exposed it. Right now our uninstallation path looks like:

1. Figure out what kind of installation it is (distutils (aka broken), setuptools, egg, etc).
   
   
   
   • If this is a distutils installation, this is the point we raise an error because we can't exactly do the next step without the metadata that is missing from these steps.
   
   
2. Locate all of the paths associated with a particular project, and add them to a list of paths to remove.
3. Determine if we're running inside of a virtual environment, and if we are if the item we're attempting to uninstall is "local" to that virtual environment, or if it comes from the system and remove files based on that.
   
   
   
   • If it comes from the system, we simply skip out here and pretend we uninstalled it. This means pip uninstall is broken, but upgrades will work because pip will shadow the system installed item with one installed inside of the virtual environment.
   
   
   • If it is inside of the virtual environment, we queue up a bunch of paths to remove, delete them, and then continue on with whatever the next task is.
   
   

This should be easy to support by splitting the third step up so that the first thing we do is check if the item we're attempting to install is "local" to any currently running virtual environments and simply skipping out earlier than we are currently. The goal here was to fix a slightly different problem, where you're not shadowing via sys.path, but you're literally overwriting files to the same disk location, or you're removing the .egg-info but leaving the package otherwise installed. Those cases don't happen in the specific one that @wichert is having.

[pfmoore]

Agreed. What we do in step 3a doesn't care if the existing install is broken, so we can (actual technical details permitting!) do that up front.

[dstufft]

Argparse is another thing that people are running into this issue on, because argparse in the standard library includes a .egg-info file. However the standard library argparse is always going to shadow any installed argparse (because of sys.path ordering) so installing argparse on Python 2.7+ and 3.3+ is pointless unless you munge sys.path to put it first.

What do we think about making installing argparse a no-op on 2.7+, 3.3+ and just hard failing if someone attempts to "install" a version specifier of it that doesn't include what is shipped in the standard library?

[rbtcollins]

Catching up - so there are these classes of things that might have distutils metdata:

• now-stdlib things with disutils metadata - argparse, wsgiref
• distutils metadata for non-stdlib things - requests etc from system packages

And they can turn up in the following places:

• present outside the stdlib (argparse on python 2.6, argparse installed on python 2.7, requests in any python version)
• in the stdlib - argparse/wsigref
• in system-site-packages

With the following targets for install:

• not on sys.path
• on sys.path after purelib
• on sys.path before purelib

I think there is consensus that we shouldn't complain or error on installs where the problem thing is both in an immutable place, and we're installing into a place that will shadow it.

Where its a bit messy is where we're installing into a place that won't shadow an existing install like argparse in the stdlib, or just poorly configured sys.paths. There if we permit the install to complete, it will have no impact. If we don't, then we're breaking already published packages that trigger this situation. So I think we have to warn, indefinitely - its not a deprecatable thing in the same way that getting distros to use setuptools injection when building packages is.

Where we're installing into something not on on sys.path, I think we should totally disable uninstalls and also not warn.

Where the interacting is with something in a different environment, it should be covered by the first point - the problem is in an immutable environment, and we're going to shadow it.

Assuming that this makes sense then, we need to do something along the lines of @dstufft's patch to not attempt to uninstall local items.

This still doesn't address the impact of distro packages without file records and non-virtualenv uses.

I'd like to rollback the warning->fatal change. Here's my reasoning:

• we haven't made a consistent visible effort to get distros to change
• we haven't had a deluge of folk with the corner cases turning up - the status quo is tolerable
• without the distros being fixed, end users have no way to meaningfully address this - asking folk to retool to virtualenvs is nontrivial

[SpamapS]

Have to agree with Robert. I understand 7->8 _can_ break things because of the major version bump. I just don't think it should break things that aren't in need of breaking. It seems like this one can just be left alone and the usual slow Warning bubble-up process will have to suffice.

[coderanger]

Checking in, standard Ubuntu 14.04 image includes gross versions of both requests and six, so attempting to install either system-wide will fail.

[dstufft]

This should be fixed now in develop.