Pip: move_wheel_files fails on setuptools dependencies

[jaraco]

Reported in pypa/setuptools#951, it appears that when upgrading or force-reinstalling setuptools dependencies, pip calls into pkg_resources to get package metadata, but since the package is currently uninstalled, the search for its metadata fails.

I haven't looked into the pip code, but I wanted to ask this project first: Can someone provide a brief explanation for what pip is doing here and if there's an adjustment that pip or pkg_resources could make to avoid the failure?

[dstufft]

Um, well move_wheel_file is use to install a wheel. I think that inside of it, it calls distutils_scheme() which creates a Distribution object to figure out where distutils would have installed the files so that we can install them to the same location (taking into account ~/.pydistutils.cfg files and such).

[jaraco]

Hmm. The Distribution object that's created is a distutils.dist.Distribution, which wouldn't be calling into setuptools unless that object had been monkeypatched by an import of setuptools. Searching through the pip code, it does seem that setuptools is imported at the invocation of InstallRequirement.setup_py.

That explains why the issue only occurs when a pure sdist package is first installed (such as junit-xml). It's the installation of that package that triggers the import of setuptools in the pip runtime, which triggers the monkeypatch, which triggers the enumeration of distutils.setup_keywords entry points, which forces the dependencies of setuptools to be required, but since six isn't currently installed things fall apart. I suspect this issue could also be triggered by the upgrade or downgrade of any setuptools dependency from a source dist.

My first instinct is to see if we can adapt pip not to import setuptools and thus not monkeypatch the Distribution object. I'm unsure if the monkeypatching is necessary.

Indeed, when I apply this patch, the failure goes away:

diff --git a/pip/req/req_install.py b/pip/req/req_install.py
index 1a98f377..f4716c57 100644
--- a/pip/req/req_install.py
+++ b/pip/req/req_install.py
@@ -383,17 +383,10 @@ class InstallRequirement(object):
     @property
     def setup_py(self):
         assert self.source_dir, "No source dir for %s" % self
-        try:
-            import setuptools  # noqa
-        except ImportError:
-            if get_installed_version('setuptools') is None:
-                add_msg = "Please install setuptools."
-            else:
-                add_msg = traceback.format_exc()
-            # Setuptools is not available
-            raise InstallationError(
-                "Could not import setuptools which is required to "
-                "install from a source distribution.\n%s" % add_msg
+        if get_installed_version('setuptools') is None:
+            raise InstallationError("Setuptools is required to install "
+                "from a source distribution, but none was found. Please "
+                "install setuptools."
             )

         setup_py = os.path.join(self.setup_py_dir, 'setup.py')

I can imagine this change would cause failures in environments where setuptools is importable but not installed as pip sees it. To retain the existing behavior, one could import setuptools in a subprocess to check for the existence of an importable setuptools... or bypass a full import and call into find_module instead to see if it can be found without executing it.

I'd also be open to the possibility of removing the implicit monkeypatching of distutils, such that all setup scripts would would have to be updated to reference the setuptools objects directly or call the monkeypatching explicitly. There's a lot of unknowns and possibly intractable challenges in that scenario.

What do you think about the patch above? Do late versions of pip need to detect and pass on an importable but not installed setuptools? Do you have any other ideas?

[jaraco]

I'd also be open to the possibility of removing the implicit monkeypatching of distutils

Another less aggressive option would be for setuptools to expose a routine to unmonkeypatch distutils, and pip could call that after the import. Another option would be for pip to import the Distribution object before it has a chance to be monkeypatched and always use that.

[dstufft]

Do late versions of pip need to detect and pass on an importable but not installed setuptools?

I don't think there is any reason they need to work if setuptools isn't "installed", at least not off the top of my head.

[bsundsrud]

Is there an ETA on releasing a fix for this? This is heavily impacting our workflow.

[underyx]

cryptography depends on setuptools so if we run pip-compile, it pins the appdirs version.

If you then try to use tox to run anything, it will

1. Get the latest setuptools and appdirs with broken uninstallation
2. Try to install our requirements which have a non-latest appdirs version pinned, triggering an uninstall and breaking everything

Rather annoying. Any chance of a release soon?