Pip: Error checking if package with dotted name is installed (pip==20.2)

[damb]

Hi,

I've got an issue while installing namespace distribution dependencies. While it worked fine with pip==20.1.1 resolving fails with pip==20.2.

• Python: 3.8.2
• Pip: 20.2
• Platform:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:    20.04
Codename:   focal

My namespace distributions follow PEP420. However, I use a slightly different naming scheme than proposed in https://github.com/pypa/sample-namespace-packages/tree/master/native.

To reproduce the issue the following steps need to be executed:

1. Clone https://github.com/pypa/sample-namespace-packages

$ git clone https://github.com/pypa/sample-namespace-packages.git && cd sample-namespace-packages

1. Apply the following diff:

$ git diff
diff --git a/native/pkg_a/setup.py b/native/pkg_a/setup.py
index 8021a7a..1238c0c 100644
--- a/native/pkg_a/setup.py
+++ b/native/pkg_a/setup.py
@@ -16,7 +16,7 @@ from setuptools import setup


 setup(
-    name='example_pkg_a',
+    name='example_pkg.a',

     version='1',

diff --git a/native/pkg_b/setup.py b/native/pkg_b/setup.py
index 872122a..1830dcc 100644
--- a/native/pkg_b/setup.py
+++ b/native/pkg_b/setup.py
@@ -16,7 +16,7 @@ from setuptools import setup


 setup(
-    name='example_pkg_b',
+    name='example_pkg.b',

     version='1',

@@ -27,6 +27,7 @@ setup(
     author_email='[email protected]',

     license='Apache Software License',
+    install_requires=['example_pkg.a==1'],

1. Check pip version:

$ pip --version
pip 20.2 from /home/ubuntu/work/projects/sample-namespace-packages/venv/lib/python3.8/site-packages/pip (python 3.8)

1. Install pkg_a:

$ pip install native/pkg_a/
Processing ./native/pkg_a
Using legacy 'setup.py install' for example-pkg.a, since package 'wheel' is not installed.
Installing collected packages: example-pkg.a
    Running setup.py install for example-pkg.a ... done
Successfully installed example-pkg.a-1
$ pip freeze
example-pkg.a==1
pkg-resources==0.0.0

1. Try to install pkg_b which should fail:

$ pip install native/pkg_b/
Processing ./native/pkg_b
ERROR: Could not find a version that satisfies the requirement example_pkg.a==1 (from example-pkg.b==1) (from versions: none)
ERROR: No matching distribution found for example_pkg.a==1 (from example-pkg.b==1)

Though, when naming the namespace distributions as described in https://github.com/pypa/sample-namespace-packages/tree/master/native i.e. with a _ instead of a ., the distributions install correctly.

It took me a while until I noted that the issue is the naming convention, I didn't follow thoroughly.

Question: Is the change from pip==20.1.1 to pip==20.2 on purpose?

If you need any further details, let me know.

[uranusjr]

This seems like a normalisation issue. Dot, underscore, and dash are actually treated as equivalents when comparing package names, so pip is probably not doing that comparison correctly somewhere.

[damb]

You refer to the snippet below?

https://github.com/pypa/pip/blob/31299ee37058fafc84535ee3a47f0468433aeb20/src/pip/_vendor/packaging/utils.py#L16-L23

[sbidoul]

@damb that canonicalize_name should be fine. It's the way pip uses it that has a bug since 20.2.

[sbidoul]

Here is a reproducer for a variant of what is probably the same bug:

#!/bin/bash
set -ex
cd $(mktemp -d)
python3 -m venv venv
. venv/bin/activate
pip install -U "pip>=20.2" wheel py3o.formats
echo "from setuptools import setup; setup(name='myproject', install_requires=['py3o.formats'])" > setup.py
pip install --no-index -e .

[sbidoul]

So the bug is in check_if_exists and has been introduced in https://github.com/pypa/pip/pull/8054.

Apparently, passing a canonical name with dots replaced by dashes is not compatible with pkg_resources.get_distribution()

Example:

$ pip install zope.inferface
$ python -c "import pkg_resources as pr; print(pr.get_distribution('zope.interface'))"
zope.interface 5.1.0
$ python -c "import pkg_resources as pr; print(pr.get_distribution('zope-interface'))"
...
pkg_resources.DistributionNotFound: The 'zope-interface' distribution was not found and is required by the application

[uranusjr]

You beat me to it 😛 Relevent code:

https://github.com/pypa/pip/blob/31299ee37058fafc84535ee3a47f0468433aeb20/src/pip/_vendor/pkg_resources/__init__.py#L1317-L1322

We’ll probably need to ask setuptools maintainers whether this is an oversight or a delibrate decision, and whether we can patch this logic ourselves.

[uranusjr]

Or alternatively, we can rollback #8054 (the root issue it tries to resolve is exactly pkg_resources does not handle dot in names well), and migrate wholesale to importlib.metadata.

[pradyunsg]

Looks like we'd need a bugfix release.

Or alternatively, we can rollback #8054

I like this plan. Anyone wanna file a PR for this?

[sbidoul]

I wanted to verify why check_if_exists still needs to call pkg_resources.get_distribution while we have our own misc.get_distribution which seems to do the right thing. I will probably not have time for that until the week-end or early next week though.

[uranusjr]

Oh, good point! IIRC that was planned as a follow-up PR (since the refactoring would be difficult to review if committed together with the get_distribution() change), but it never happened. I’ll try rewriting that and see if that’s enough to get this issue resolved.

[deveshks]

I had filed an issue to remove the last remaining usage of pkg_resources.get_distribution (https://github.com/pypa/pip/issues/8550) as a follow-up of https://github.com/pypa/pip/pull/8054, but I was waiting on https://github.com/pypa/pip/pull/8114 to get merged first before proceeding on it further.

[uranusjr]

I experiemented a bit, and it seems like switching from calling pkg_resources.get_distribution() to pip’s own would indeed solve this particular issue.

Since this is a quite significant issue (there are many popular packages preferring to be referred by a dot in the name, e.g. coverage.py), I’m going to do the one required replacement in a separate PR, so we can get a bugfix version out as soon as possible. #8114 and the other replacements can happen afterwards.

[uranusjr]

I opened #8659 to fix check_if_exists.