Virtualenv: Error during environment creation in Python 3.9.0a1

[bskinn]

I compiled Python 3.9.0a1 (no optimizations) on my Debian stretch box just now, and am getting errors when trying to create a virtual environment.

Based on the traceback, it looks like the problem stems (at least in part) from virtualenv using the vendored pip v19.1.1. (pip added an import cascade for collections.abc.Mapping in v19.2.) I see pip v19.3.1 in my Python 3.9 install, but it's apparently going with v19.1.1 instead:

~/python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support$ ls
__init__.py                      __pycache__
pip-19.1.1-py2.py3-none-any.whl  setuptools-41.4.0-py2.py3-none-any.whl
pip-19.3.1-py2.py3-none-any.whl  wheel-0.33.6-py2.py3-none-any.whl

Commands and output:

$ python3.9 -m virtualenv env39test
Using base prefix '.../python/3.9.0a1'
New python executable in .../py/env39test/bin/python3.9
Also creating executable in .../py/env39test/bin/python
Command .../py/env39test/bin/python3.9 -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command .../py/env39test/bin/python3.9 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_internal/__init__.py", line 40, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 8, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_internal/cli/main_parser.py", line 12, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_internal/commands/__init__.py", line 6, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_internal/commands/completion.py", line 6, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_internal/cli/base_command.py", line 25, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_internal/index.py", line 14, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_vendor/html5lib/__init__.py", line 25, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_vendor/html5lib/html5parser.py", line 8, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_vendor/html5lib/_tokenizer.py", line 16, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_vendor/html5lib/_trie/__init__.py", line 3, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_vendor/html5lib/_trie/py.py", line 6, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/_vendor/html5lib/_trie/_base.py", line 3, in <module>
ImportError: cannot import name 'Mapping' from 'collections' (.../py/env39test/lib/python3.9/collections/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
ImportError: cannot import name 'main' from 'pip' (.../python/3.9.0a1/lib/python3.9/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl/pip/__init__.py)
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File ".../python/3.9.0a1/lib/python3.9/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File ".../python/3.9.0a1/lib/python3.9/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv.py", line 2628, in <module>
    main()
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv.py", line 860, in main
    create_environment(
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv.py", line 1173, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv.py", line 1019, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv.py", line 1110, in _install_wheel_with_search_dir
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
  File ".../python/3.9.0a1/lib/python3.9/site-packages/virtualenv.py", line 963, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command .../py/env39test/bin/python3.9 - setuptools pip wheel failed with error code 1

Debian:

Linux ... 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64

Python version:

$ python3.9
Python 3.9.0a1 (default, Nov 20 2019, 10:14:23)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Installed packages:

$ python3.9 -m pip list
Package    Version
---------- -------
pip        19.3.1
setuptools 41.6.0
virtualenv 16.7.7
wheel      0.33.6

[nedbat]

Ditto on Mac. Same traceback.

[bskinn]

Reply from @gaborbernat on Twitter:

This will be fixed via the rewrite of virtualenv that's WIP. If anyone wants it fixed in 16.x series now, better make a pr 👍

[nedbat]

@gaborbernat What sort of timeframe are you thinking of for the rewrite? This is blocking libraries from testing Python 3.9.

[bskinn]

@nedbat Interestingly, it worked ok for two of my libraries using PEP517/pyproject.toml in the src/ project layout. The above failure is for a project still on non-src and non-PEP517.

@gaborbernat, if I read the traceback correctly, since it's a python -m pip invocation leading to this error, is the solution to rename the vendored pip 19.1.1 to, say, pip_legacy, and then use python -m pip_legacy for Python < 3.5?

[bskinn]

Ooh, no -- since all we need(?) is a short-term fix, maybe it would be enough to wrap this version search with reversed(...).

If I emulate the glob.glob search for pip wheels, I indeed get 19.1.1 first in the list:

>>> import glob
>>> sorted(glob.glob("pip-*.whl"))
['pip-19.1.1-py2.py3-none-any.whl', 'pip-19.2.3-py2.py3-none-any.whl']
>>> list(reversed(sorted(glob.glob("pip-*.whl"))))
['pip-19.2.3-py2.py3-none-any.whl', 'pip-19.1.1-py2.py3-none-any.whl']

The [0] index to the versions list would then grab 19.2.3 (or whatever latest version), and the filter for Python 3.4 should still constrain to pip 19.1.1 for 3.4.

[asottile]

here's a demo of it working -- https://github.com/asottile/azure-pipeline-templates/pull/29

thanks again for the issue and fix here @bskinn :tada:

[DahlitzFlorian]

As far as I know glob.glob is not ordered (see reference below). So different operating systems/environments may produce different results. Wouldn't it be more convenient if something like reversed(sorted(glob.glob(...))) is used?

Reference: https://pymotw.com/3/glob/

[bskinn]

@DahlitzFlorian Exactly right -- the original code used (approximately) sorted(glob.glob(...)); the fix uses (approximately) list(reversed(sorted(glob.glob(...))))

My specific above test was dangerously imprecise, but I think the fix itself should be robust (well, as long as pip doesn't move to a 3-digit major version number anytime soon).

In the interest of not causing people to use glob.glob dangerously, I'll go ahead and edit my example.