Pip: list --outdated: TypeError: '>' not supported between instances of 'Version' and 'Version'

[daavve]

Environment

• pip version:
• Python version:
• OS:

Arch Linux, Testing version of pip: https://www.archlinux.org/packages/testing/any/python-pip/
Description

$ pip list -o

Exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
status = self.run(options, args)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 145, in run
packages = self.get_outdated(packages, options)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 156, in get_outdated
dist for dist in self.iter_packages_latest_infos(packages, options)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 157, in
if dist.latest_version > dist.parsed_version
TypeError: '>' not supported between instances of 'Version' and 'Version'

[daavve]

Reported downstream: https://bugs.archlinux.org/index.php?do=details&task_id=58670

[yan12125]

Looks like an issue related to deblundling. If I modify the get_outdated() function in /usr/lib/python3.6/site-packages/pip/_internal/commands/list.py to:

    def get_outdated(self, packages, options):
        ret = []
        for dist in self.iter_packages_latest_infos(packages, options):
            print(dist.latest_version.__class__)
            print(dist.parsed_version.__class__)
            if dist.latest_version > dist.parsed_version:
                ret.append(dist)
        return ret

pip list -o prints:

<class 'pip._vendor.packaging.version.Version'>
<class 'pkg_resources.extern.packaging.version.Version'>
Exception:
Traceback (most recent call last):                                                                                                      
  File "/usr/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main                                               
    status = self.run(options, args)                                                                                                    
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 145, in run                                              
    packages = self.get_outdated(packages, options)                                                                                     
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 159, in get_outdated                                     
    if dist.latest_version > dist.parsed_version:                                                                                       
TypeError: '>' not supported between instances of 'Version' and 'Version'                                                               

pip._vendor.packaging.version.Version and pkg_resources.extern.packaging.version.Version are actually the same class (both are packaging.version.Version), while Python does not think so.

[pradyunsg]

Thanks for looking into this @yan12125! :)

Indeed, this is a downstream packaging issue.

[pradyunsg]

Adding to my above comment, I don't see anything actionable on this from pip's on this -- it's due to how Arch Linux is debundling pip.

[daavve]

I think you are right. This looks like an Arch issue.

[yan12125]

I managed to patch pkg_resources locally to make pip list -o work with Arch Linux packages. Hopefully that helps other distributions using debundling. This patch is created against python-setuptools 1:39.2.0-1.

--- pkg_resources/__init__.py.orig  2018-05-22 00:21:05.592049050 +0800
+++ pkg_resources/__init__.py   2018-05-22 00:21:19.378315213 +0800
@@ -71,11 +71,11 @@

 from . import py31compat
 from pkg_resources.extern import appdirs
-from pkg_resources.extern import packaging
-__import__('pkg_resources.extern.packaging.version')
-__import__('pkg_resources.extern.packaging.specifiers')
-__import__('pkg_resources.extern.packaging.requirements')
-__import__('pkg_resources.extern.packaging.markers')
+import packaging
+__import__('packaging.version')
+__import__('packaging.specifiers')
+__import__('packaging.requirements')
+__import__('packaging.markers')


 if (3, 0) < sys.version_info < (3, 3):

[eli-schwartz]

Adding to my above comment, I don't see anything actionable on this from pip's on this -- it's due to how Arch Linux is debundling pip.

We're debundling according to the method nominally supported by pip itself, so I think it should be actionable here. :)

pip._internal.index.PackageFinder attempts to use pip._vendor.packaging.version.parse (which is dist.latest_version in the traceback), then compares it to pip._internal.utils.misc.get_installed_distributions which uses pip._vendor.pkg_resources to return a set of pkg_resources packages -- even though they're the same files, they appear in different locations on the import path, and python complains.

This somehow works for pip when using vendored everything, but only because the vendored pkg_resources is then additionally modified from stock pkg_resources to import everything from the top-level _vendor.packaging

...

I don't fully grok the magic used by pip's vendored() and pkg_resource's VendorImporter(), but it looks like they might be incompatible, and pip is papering over this logic by completely ignoring the latter in its own vendoring code.

[pradyunsg]

Hey @eli-schwartz!

Could you take a look at https://github.com/pypa/pip/issues/5346#issuecomment-388046352 and provide information about the people at Arch Linux to ping for such things? :)

(I can't respond to your comment now because I'm super short on time; I'll reopen this issue so that I remember that I need to get back to this.)

[yan12125]

I created a more general patch:

--- pkg_resources/extern/__init__.py.orig   2018-05-23 00:58:31.680640763 +0800
+++ pkg_resources/extern/__init__.py    2018-05-23 00:58:47.678203875 +0800
@@ -48,7 +48,7 @@
                 # on later Python versions to cause relative imports
                 # in the vendor package to resolve the same modules
                 # as those going through this importer.
-                if sys.version_info > (3, 3):
+                if sys.version_info > (3, 3) and prefix != '':
                     del sys.modules[extant]
                 return mod
             except ImportError:

Anyway this looks like a setuptools issue rather than a pip one.

[eli-schwartz]

@jaraco, can you confirm?

[yan12125]

The issue reappeared with pip git-master and Python 3.7.2 even with the setuptools patch in https://github.com/pypa/pip/issues/5429#issuecomment-390705580. This time the fix should go to pip IMO. I'm currently working on this.

[valtron]

Reinstall pip to fix this:

pip3 install --ignore-installed pip

[ntninja]

I'm affected as well by this on Debian unstable and neither of @yan12125's patches worked. :slightly_frowning_face:

[yan12125]

The issue still exists in pip 19.1?

[eli-schwartz]

Debian unstable still packages pip 18.1, so @alexander255's issue is definitely not confirmed with pip 19.

[bernd-wechner]

pip3 install --ignore-installed pip

Wow I just did that because I have this issue on Mint 19. Only it broke pip. Awesome.

$ pip3 install --ignore-installed pip
Collecting pip
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 814kB/s 
Installing collected packages: pip
Successfully installed pip-19.1.1
$ pip3
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
ImportError: cannot import name 'main'

Any pointer appreciated. That sux!

Addendum: fixed it my uninstalling pip3 with apt and reinstalling.

[bernd-wechner]

@yan12125

and prefix != '':

I made that patch and still had the problem. So undid it.

[eli-schwartz]

Wow I just did that because I have this issue on Mint 19. Only it broke pip. Awesome.

Don't follow bad advice from people who don't know what they are talking about and don't have context on the issue. Speaking with my "I am a developer for a linux distro" hat on, do not ever sudo pip anything. Ever. And definitely, definitely, definitely, do not try to install pip from your distro package manager and then update it using pip, because you WILL break it and as extensively discussed in #5346, and ideally pip would learn to say "you are not allowed to update this using pip, please use your package manager instead" as tracked by #5605.

You will need to either install pip via apt, or install pip via get-pip.py, and make sure you don't have bits and pieces of the wrong thing in your PYTHONPATH.

...

As for why Linux Mint's pip package has this bug, you need to discuss this with the Debian packagers of python3-pip. My guess is they are shipping a broken setuptools. See https://bugs.archlinux.org/task/58670 for more details.

[bernd-wechner]

Thanks enormously for informative advice. I uninstalled the Debian pip packages and installed get-pip.py as per the Python website and now pip works fine, and I have pip installing python3 packages (whereas Debian packages install pip2 and pip3 and link pip to pip2 still - and I like many, have near zero interest in Python 2 anymore).

And pip list -o works fine now too. Basically everything seems fine after removing the debian packages and using get-pip.py.

I do have one question though.

do not ever sudo pip anything.

I mildly agree with the principle here, but what if I want my packages installed in /usr/lib/python3.6/dist-packages say and not in my home dir? In fact with this pip I just installed it tries that and bombs suggesting I use the --user option. What is your recommendation in that space?

[pradyunsg]

what if I want my packages installed in /usr/lib/python3.6/dist-packages

What's the use case for not using a Debian-provided pip but still wanting to install to a location where that would install packages?

[pradyunsg]

What is your recommendation in that space?

As the fellow who added that to message -- I'd say use --user as the message suggests. 🙃

[bernd-wechner]

what if I want my packages installed in /usr/lib/python3.6/dist-packages

What's the use case for not using a Debian-provided pip but still wanting to install to a location where that would install packages?

Forgive me, but the obvious: to make the packages available not to the current user and Python code they run, but to other users, including system services that need access to them. The use case is not to install them where the Debian packaged pip would install them, the use case is system wide installation (vs. local user installation).

With all due respect, that is such a common and ordinary and fairly obvious use case I'd have thought that pip by default installs there (I have to use --user flag to get around it) and I forgot to state it explicitly.

[eli-schwartz]

You're installing system services without using the package manager? Personally I'd avoid that sort of thing. Then again, Arch Linux does make it very easy to empower the masses to create system packages.

...

Using sudo pip means that pip can and will overwrite files tracked by the package manager. Do you have software on your system which is written in python and distributed via apt? Python is a popular programming language for desktop software, there are DEs written in python. Do you want to break that software by overwriting system packages with incompatible components?

[pradyunsg]

With all due respect, that is such a common and ordinary and fairly obvious use case I'd have thought that pip by default installs there (I have to use --user flag to get around it) and I forgot to state it explicitly.

Yep yep - #1668. It's a bit tricker than it looks on the surface. 🙃

[bernd-wechner]

Using sudo pip means that pip can and will overwrite files tracked by the package manager. Do you have software on your system which is written in python and distributed via apt? Python is a popular programming language for desktop software, there are DEs written in python. Do you want to break that software by overwriting system packages with incompatible components?

Guess you're right. Never really thought about it. Still that does create a conundrum that is discussed in the issue predyunsg mentioned via a linked to item... namely that the OS package (Ubuntu apt repos in my case) are generally way out of date ... and so if you want the latest at a system level it's pip or conda I guess.

Of course I guess I concur, to be safe just install to home dir as a first cut.

In the end though I admit I work on a few premises:

1. That anybody installing python packages (apt, pip, conda?) also takes the trouble to ensure the python path includes the directory they use to install them.
2. That python packages on the whole don't do backward breaks, i.e. maintain backward compatibility as a priority (inside of a major Python release in any case).

Of course as you point out those premises may be a little naive, and introduce unnecessary risk if taken as truths and installing with pip to system dirs.

it does leave the question open (as per predyunsg's linked item) why on earth pip defaults to system dirs, if the wisdom you share is widely held in the community of people who develop and maintain pip! It does indeed seem sensible to default to local.

[pradyunsg]

why on earth pip defaults to system dirs

Because... no one's put in the effort to change that and it's a non-trivial change. :)

[martin3000]

In list.py I changed

    def get_outdated(self, packages, options):
        return [
            dist for dist in self.iter_packages_latest_infos(packages, options)
            if dist.latest_version > dist.parsed_version
        ]

to

    def get_outdated(self, packages, options):
        return [
            dist for dist in self.iter_packages_latest_infos(packages, options)
            if str(dist.latest_version) > str(dist.parsed_version)
        ]

and it works.

[yan12125]

@martin3000 That patch may lead to incorrect results as PEP 440 version comparisons are not plain string comparisons.

@pradyunsg Could you consider closing this issue to send a message that "this has been fixed"? Specifically, users need setuptools >= 40.0.0 and pip >= 19.1. I don't think this ticket needs more discussions about patches for pip/setuptools unless pip list -o would be broken again.

[martin3000]

Comparing the strings ist not ok, but here in Ubuntu 19.04 it was crashing.

[yan12125]

Apparently Ubuntu 19.04 ships with pip 18.1 [1]. The issue is fixed since pip 19.1 (commit 8ef3283fcf7c7c5537f305a374355edc96f40d3d, specifically). You may want to report the issue to Ubuntu or Debian developers, so that they can fix their package by either upgrading pip or backporting the relevant commit.

Alternatively, you can also uninstall the system pip package with apt and install the latest pip following https://pip.pypa.io/en/stable/installing/. Note that it's better to keep only one of Ubuntu's pip and the one from pypa.io, or there might be errors like https://github.com/pypa/pip/issues/5429#issuecomment-510337059.

[1] https://launchpad.net/ubuntu/disco/+source/python-pip

[jeremy-rifkin]

Had this issue with ubuntu 18 and apt's python3-pip package (pip 9.0.1).
Fixed by uninstalling pip 9.01 and using get-pip.py to install pip 19.2.

[pradyunsg]

Could you consider closing this issue to send a message that "this has been fixed"? Specifically, users need setuptools >= 40.0.0 and pip >= 19.1.

Sounds reasonable!

[verhovsky]

This might be unsafe, but

sudo -H pip3 install --upgrade pip

Solved it for me on Ubuntu 18.04. Make sure you do sudo, otherwise you'll get ModuleNotFoundError: No module named 'pip'.