Setuptools: _NamespacePath object has no attribute sort (38.5.1)

[wichert]

Using Python 3.6, pip 9.0.1 and 38.5.1 I get the following error:

$ bin/pip list
Traceback (most recent call last):
  File "bin/pip", line 7, in <module>
    from pip import main
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/__init__.py", line 26, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/utils/__init__.py", line 27, in <module>
    from pip._vendor import pkg_resources
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3018, in <module>
    @_call_aside
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3004, in _call_aside
    f(*args, **kwargs)
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3046, in _initialize_master_working_set
    dist.activate(replace=False)
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2578, in activate
    declare_namespace(pkg)
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2152, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2092, in _handle_ns
    _rebuild_mod_path(path, packageName, module)
  File "/Users/wichert/.local/share/virtualenvs/curvetips.api-ihv708oq/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2121, in _rebuild_mod_path
    orig_path.sort(key=position_in_sys_path)
AttributeError: '_NamespacePath' object has no attribute 'sort'

This is a virtualenv made yesterday using pipenv.

As requested in #885 the nspath data is:

$ for pth in $(ls lib/python3.6/site-packages/*-nspkg.pth); do echo $pth ; cat $pth ; done
lib/python3.6/site-packages/PasteDeploy-1.5.2-py2.6-nspkg.pth
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('paste',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('paste',types.ModuleType('paste')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
lib/python3.6/site-packages/curvetips.logging-1.4.2-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('curvetips',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('curvetips', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('curvetips', [os.path.dirname(p)])));m = m or sys.modules.setdefault('curvetips', types.ModuleType('curvetips'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/curvetips.model-1.0.3-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('curvetips',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('curvetips', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('curvetips', [os.path.dirname(p)])));m = m or sys.modules.setdefault('curvetips', types.ModuleType('curvetips'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/curvetips.utils-1.1.0-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('curvetips',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('curvetips', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('curvetips', [os.path.dirname(p)])));m = m or sys.modules.setdefault('curvetips', types.ModuleType('curvetips'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/google_auth-1.4.1-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/repoze.lru-0.7-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('repoze',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('repoze', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('repoze', [os.path.dirname(p)])));m = m or sys.modules.setdefault('repoze', types.ModuleType('repoze'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/zope.component-4.4.1-py3.4-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('zope', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)])));m = m or sys.modules.setdefault('zope', types.ModuleType('zope'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/zope.deprecation-4.3.0-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('zope', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)])));m = m or sys.modules.setdefault('zope', types.ModuleType('zope'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/zope.event-4.3.0-py3.5-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('zope', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)])));m = m or sys.modules.setdefault('zope', types.ModuleType('zope'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/zope.interface-4.4.3-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('zope', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)])));m = m or sys.modules.setdefault('zope', types.ModuleType('zope'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
lib/python3.6/site-packages/zope.sqlalchemy-1.0-py3.6-nspkg.pth
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('zope', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)])));m = m or sys.modules.setdefault('zope', types.ModuleType('zope'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)

[haroldsheep]

The __path__ attribute should always be treated as an interable, _not_ a list (the Python docs describe __path__ as an iterable). The line in pkg_resources/__init__.py that reads:

orig_path.sort(key=position_in_sys_path)

Should be:

orig_path = sorted(orig_path, key=position_in_sys_path)

As a general rule (if practical), in-place sorting should not be used.

[jaraco]

I've updated the comment in 885 to indicate more clearly that the -nspkg.pth info is only helpful if the issue is happening outside of pip. If it's happening in pip, the issue is the vendored version that doesn't yet have the fix (pypa/pip#4216). The issue has been addressed in setuptools since https://github.com/pypa/setuptools/commit/7c0c39ef1f60571709e5a8e6680f2147c38cd4ff.

As a general rule (if practical), in-place sorting should not be used.

That seems reasonable to me; I retained the mutating behavior as I couldn't be sure what interfaces might be relying on it. We could certainly try your suggestion. Guaranteed it won't fix the reported issue, though.

[wichert]

@jaraco pip 9.0.2, released 2 days ago includes the fix from setuptools, but still shows the exact same problem.

[wichert]

I was looking at the wrong tree - pip 9.0.2 is indeed missing this fix.

[arutz12]

@haroldsheep

Your solution did the trick, thanks a lot!

[oberstet]

I can't believe this is not a Prio 1 ticket, but actually closed! The fix still appear to be unshipped, and I am getting tired of applying the edit manually after some new py install;) please guys, could someone just merge that? for the fix, see: https://github.com/pypa/setuptools/issues/1282#issuecomment-372422080

[rizplate]

@oberstet I upgraded pip to 9.0.3 and it works there.

[chrysm]

sudo pip3 install -U setuptools
did the trick. pip was already up-to-date.

[Hskye]

After I have remove files related to pip3 and apt-get remove python3-pip, I refer to https://pip.pypa.io/en/stable/installing/
and solve the broken pip3 problem