Pip: wonky pth file breaks upgrade

[rbtcollins]

nstalling collected packages: pbr, pip
  Found existing installation: pbr 0.10.0
Cleaning up...
  Removing temporary dir /tmp/pip_build_root...
Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 555, in uninstall
    paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1785, in add_pth
    self.pth[pth_file] = UninstallPthEntries(pth_file)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1868, in __init__
    raise UninstallationError("Cannot remove entries from nonexistent file %s" % pth_file)
UninstallationError: Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth

[rbtcollins]

pip 1.5.4, trying with newer pip.

[rbtcollins]

still present in 6.1.1

[rbtcollins]

Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run
    root=options.root_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 687, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 671, in uninstall
    paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 67, in add_pth
    self.pth[pth_file] = UninstallPthEntries(pth_file)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 162, in __init__
    "Cannot remove entries from nonexistent file %s" % pth_file
UninstallationError: Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth

[rbtcollins]

this is trying to install pbr, which appears to be registered but isn't really there: its not in pip list.

[rbtcollins]

There is a /usr/local/lib/python2.7/dist-packages/oslo.config.egg-link file with one line in it /home/robertc/work/openstack/oslo.config

[rbtcollins]

And

cat /usr/local/lib/python2.7/dist-packages/easy-install.pth

import sys; sys.__plen = len(sys.path)
/home/robertc/work/openstack/oslo.config
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

[rbtcollins]

removing both of those files allowed the install to proceed. I suspect ignoring the missing file would have been better?

[cdeil]

I think I just ran into the same issue here with pip 7.1.2:
https://travis-ci.org/astropy/astroplan/jobs/82742624#L613
https://github.com/astropy/astroplan/blob/master/.continuous-integration/travis/setup_dependencies_common.sh#L58

Am I doing something wrong?
Is there a workaround?

[xavfernandez]

@rbtcollins It would be interesting to know how the initial package was installed ?

@cdeil Sorry I'm not familiar with conda and even less on how it behaves/conflicts with pip but since the issue comes upgrading sphinx which comes from pip install sphinx_rtd_theme you could switch it to pip install sphinx_rtd_theme==0.1.7to avoid the need to upgrade sphinx :)

[cdeil]

@asmeurer from conda – Maybe you know why conda and pip run into the issue mentioned in my last comment.
Also: sphinx_rtd has sphinx declared as a dependency for for pip, but not for conda. Presumably this should be added for conda?

[asmeurer]

@ilanschnell

[stxlvt]

I tried filing this as an anaconda bug, but @ilanschnell closed it as a setuptools/pip issue. I'm including a summary of the discussion here:

Setup

D:\workspace>conda info
Current conda install:

             platform : win-32
        conda version : 3.18.5
  conda-build version : 1.14.1
       python version : 2.7.10.final.0
     requests version : 2.8.1
     root environment : D:\anaconda32  (writable)
  default environment : D:\anaconda32
     envs directories : D:\anaconda32\envs
        package cache : D:\anaconda32\pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/win-32/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/win-32/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

D:\workspace>conda create --name tst python
Fetching package metadata: ....
Solving package specifications: ............
Package plan for installation in environment D:\anaconda32\envs\tst:

The following NEW packages will be INSTALLED:

    msvc_runtime: 1.0.1-vc9_0   [vc9]
    pip:          7.1.2-py27_0
    python:       2.7.10-4
    setuptools:   18.4-py27_0
    wheel:        0.26.0-py27_1

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################| 100%
#
# To activate this environment, use:
# > activate tst
#

D:\workspace>activate tst
Activating environment "D:\anaconda32\envs\tst"...

The bug

[tst] D:\workspace>pip install -U setuptools
Collecting setuptools
  Downloading setuptools-18.5-py2.py3-none-any.whl (462kB)
    100% |################################| 462kB 787kB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 18.4
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

Trying to install -U any package depending on setuptools also fails with the same error.

Known workarounds

Adding the --ignore-installed option will sucessfully upgrade setuptools and the problem will be solved for the lifetime of that particular conda environment. Downloading and running ez_setup.py also fixes the problem for that conda environment. The second workaround will create the missing pth file, while the first will not, but in both cases everything seems to work afterwards.

While using conda to upgrade setuptools is also a workaround, it is not a good solution, since a lot of packages on pypi (1822 in 2013) depend on setuptools and therefore try to upgrade setuptools too, when upgraded:

[tst] D:\workspace>pip install -U zc.buildout
Collecting zc.buildout
  Downloading http://python/root/pypi/+f/739/582d22e3ddd5e/zc.buildout-2.5.0-py2
.py3-none-any.whl (261kB)
    100% |################################| 262kB 6.8MB/s
Collecting setuptools>=8.0 (from zc.buildout)
  Downloading http://python/root/pypi/+f/d40/182384798e286/setuptools-18.8.1-py2
.py3-none-any.whl (463kB)
    100% |################################| 466kB 10.2MB/s
Installing collected packages: setuptools, zc.buildout
  Found existing installation: setuptools 18.5
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

A lot of pypi packages are not installable using conda install, and so using conda to upgrade these is not an option.

None of these workarounds are good permanent solutions to this bug because:

1. They will trip a lot of users who do not know that adding -I fixes the problem and will have to spend a considerable amount of time searching for some solution.
2. They break automated testing (e.g. tox)
3. They do not fix the bug permanently on that PC, but just work for the lifetime of the environment, which is no good if the environment is a throwaway environment.

Cause

The error message is correct, there is no easy-install.pth file in site-packages. The error occurs when pip tries to uninstall the old setuptools package, in pip/req/req_uninstall.py:160. The file pip is trying to remove (easy-install.pth) does not exist in a fresh Anaconda/Miniconda installation because no conda packages are able to include this file.

This is a _periodically sleeping bug_, since it rears its head whenever the conda setuptools version (currently 18.8.1) lags behind the pip setuptools version (currently 19.1.1).

[ilanschnell]

Thanks for bringing this to my attention. I'm not sure what to do about it (from an Anaconda) perspective. The easy-install.pth cannot really be part of any conda package (as it would overlap with others). When building conda packages, we either install them:

• without setuptools
• with --single-version-externally-managed
• create a .pth file for the specific package <package name>.pth

I think that pip should not assume that easy-install.pth always exists.

[rbtcollins]

The assumption of easy-install.pth file is guarded by a check for .egg:

        elif dist.location.endswith('.egg'):
            # package installed by easy_install
            # We cannot match on dist.egg_name because it can slightly vary
            # i.e. setuptools-0.6c11-py2.6.egg vs setuptools-0.6rc11-py2.6.egg
            paths_to_remove.add(dist.location)
            easy_install_egg = os.path.split(dist.location)[1]
            easy_install_pth = os.path.join(os.path.dirname(dist.location),
                                            'easy-install.pth')
            paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)

So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well. Perhaps don't do that?

[ilanschnell]

Correct, conda packages usually include .egg-info (as it is installed by setuptools during the build process). If we removed all .egg-info files from conda packages, pip wouldn't be able to see what's installed at all (and try to overwrite packages). Not including .egg-info files wouldn't be a problem for conda itself. For conda, .egg-info files are just like any other files that are part of a conda package. Many conda packages (the ones which are not Python specific, such as libxml2, hdf5, r, etc.) have no .egg-info files either.

[rbtcollins]

No, you've misunderstood. .egg-info is not the same as .egg. The test I pasted is for "SOMETHING.egg", not "SOMETHING.egg-info". What you are doing with .egg-info files is probably correct (though I haven't read your code to authoritatively say so).

[stxlvt]

@ilanschnell, are you still of the opinion that this is a setuptools/pip issue?

[stxlvt]

FYI, there's activity over at the corresponding conda issue at https://github.com/ContinuumIO/anaconda-issues/issues/542, but this is the general consensus over there:

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

[pradyunsg]

If someone could provide steps to reproduce this issue, that'd be awesome! :)

[stxlvt]

@pradyunsg I present a reliable way to reproduce it in the OP of the anaconda issue. It assumes you have conda installed.

[chrahunt]

This seems straightforward, so I'll mark it as "awaiting PR".

If a pth file does not exist, then it does not contain any entries so we shouldn't try to edit it.

The error in question is raised here, which is invoked from add_pth, which itself is invoked from here and here.

One fix would be to simply check whether the incoming file exists in add_pth and if not do not use it to create an UninstallPthEntries. A functional test of reinstalling an editably-installed project and an egg project after removing easy-install.pth should be sufficient. The tests should be done in a separate commit and decorated with pytest.mark.xfail so they can be easily validated. I would put them in tests/functional/test_install_upgrade.py.

[deveshks]

Since as part of https://github.com/pypa/pip/pull/7891 , the UninstallationError has been converted to a logger.warning in case the pth file is not found, IMO this issue can be closed.

[uranusjr]

I’m marking this as awaiting response. Please comment if the recently-released 20.1b1 still has this problem. Otherwise the bot will auto-close this after some time.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.