Virtualenv: Setuptools v45.0.0 breaks virtualenv on Python 2

[AndreLouisCaron]

Foreword: I am quite familiar with the whole January 1st, 2020 deadline and am in the process of dropping Python 2 across all my projects, but this surprise from setuptools is quite inconvenient regardless. If you chose not to fix this, I will not judge anyone, but please don't lecture me about Python 2 being deprecated :-)

---

All my automated virtual environment creation scripts on Python 2 started breaking today.

Here is an example:

$ py -2.7-32 -m virtualenv foo
…
UnsupportedPythonVersion: Package 'setuptools' requires a different Python: 2.7.8 not in '>=3.5'
…
OSError: Command …\Scripts\python.exe - setuptools pip wheel failed with error code 1

This seems to be caused by the release of setuptools v45.0.0 yesterday:

v45.0.0: 11 Jan 2020

1458: Drop support for Python 2. Setuptools now requires Python 3.5 or later. Install setuptools using pip >=9 or pin to Setuptools <45 to maintain 2.7 support.
1959: Fix for Python 4: replace unsafe six.PY3 with six.PY2

https://setuptools.readthedocs.io/en/latest/history.html#v45-0-0

It seems that, by default, virtualenv tries to install the latest setuptools and that pip install setuptools picks up setuptools>=45.0.0 even though these versions don't support Python 2.

As a workaround, I am now using the following commands:

$ py -2.7-32 -m virtualenv --no-setuptools foo
$ foo\Scripts\python -m pip install "setuptools<45"

I'm no export in Python packaging, but I would expect setuptools>=45 to declare it doesn't support Python 2 in its distribution metadata and that pip install setuptools would automatically detect that and avoid it, but this doesn't seem to be the case.

If that is not possible, maybe versions of virtualenv for Python 2 should avoid setuptools>=45?

[yoctozepto]

I confirm this is a major issue. In OpenStack most gates are broken atm due to this. See: http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011956.html

Thanks for reporting, /me late by half an hour - I thought virtualenv would use the pip's modern resolver but it seemingly does not.

[yoctozepto]

Already tracked in setuptools for another downstream https://github.com/pypa/setuptools/issues/1963

[gaborbernat]

Not sure why setuptools 45+ gets picked up, virtualenvs bootstrap should keep it 44 for setuptools. 🤔

[yoctozepto]

Here is the traceback: http://paste.openstack.org/show/788301/

legacy_resolve sounds like something that would discard the new rules.

[yoctozepto]

Nah, wrong avenue to explore, raw pip also ignores py version constraint (despite being advertised as working): http://paste.openstack.org/show/788302/

[gaborbernat]

I'll have a look and release a fix tomorrow. If anyone has time to make a pr before that would help.

[gaborbernat]

@yoctozepto can't seem to replicate your failure; can you give us the full creation log with triple verbosity?

[pganssle]

@gaborbernat If it's the same issue as pypa/pip#7586, then this may be a bug in one of the PyPA mirrors @yoctozepto is using.

Apparently one of them is not propagating the python_requires information correctly.

[gaborbernat]

If so, then this is a bug for that mirror; nothing to do with ourselves 🤷‍♂

[ikrambabai]

The following worked for me getting out of this issue.

pip install --upgrade 'setuptools<45.0.0'

[pamelars86]

I am working with a project that is still using python 2.7 :(, and I had the same problem.
But, using command above (by ikrambabai), it worked again!! :D
Thanks!

[ostefano]

However this is still critical when creating a new virtual environment for python 2.7, since virtualenv would download setuptools-45.0.0, and no upgrade (downgrade via pip with pip install --upgrade 'setuptools<45.0.0') would actually remove it. Only solution is to download the wheel manually.

[gaborbernat]

However this is still critical when creating a new virtual environment for python 2.7, since virtualenv would download setuptools-45.0.0, and no upgrade (downgrade via pip with pip install --upgrade 'setuptools<45.0.0') would actually remove it. Only solution is to download the wheel manually.

This should only happen if you're behind an index server that does not propogate python-requires... so I'd recommend fixing the index server.

[ostefano]

@gaborbernat not that I am aware of. Is there a way for me to check it?

I can see that pythonhosted.org is used when downloading the packages when creating the virtualenv:

Installing setuptools, pkg_resources, pip, wheel...
  Running command /opt/llenv22/bin/python2.7 - setuptools pkg_resources pip wheel
  Collecting setuptools
    Using cached https://files.pythonhosted.org/packages/af/e7/02db816dc88c598281bacebbb7ccf2c9f1a6164942e88f1a0fded8643659/setuptools-45.0.0-py2.py3-none-any.whl
  Collecting pkg_resources
  Collecting pip
    Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl

[gaborbernat]

Pass in the -v flags for pip install setuptools -vvv?

[ostefano]

That works and the correct version is installed:

Collecting setuptools
  Created temporary directory: /tmp/pip-unpack-zJMfUH
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/f9/d3/955738b20d3832dfa3cd3d9b07e29a8162edb480bf988332f5e6e48ca444/setuptools-44.0.0-py2.py3-none-any.whl HTTP/1.1" 200 583230
  Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
     |████████████████████████████████| 583 kB 3.3 MB/s
  Added setuptools from https://files.pythonhosted.org/packages/f9/d3/955738b20d3832dfa3cd3d9b07e29a8162edb480bf988332f5e6e48ca444/setuptools-44.0.0-py2.py3-none-any.whl#sha256=180081a244d0888b0065e18206950d603f6550721bd6f8c0a10221ed467dd78e to build tracker '/tmp/pip-req-tracker-OTDORt'
  Removed setuptools from https://files.pythonhosted.org/packages/f9/d3/955738b20d3832dfa3cd3d9b07e29a8162edb480bf988332f5e6e48ca444/setuptools-44.0.0-py2.py3-none-any.whl#sha256=180081a244d0888b0065e18206950d603f6550721bd6f8c0a10221ed467dd78e from build tracker '/tmp/pip-req-tracker-OTDORt'
Installing collected packages: setuptools
  Created temporary directory: /tmp/pip-unpacked-wheel-vDEYe

The problem arises only when creating the virtual environment.

[AndreLouisCaron]

@ostefano It was the same for me. I worked around this by installing setuptools explicitly:

$ py -2.7-32 -m virtualenv --no-setuptools foo
$ foo\Scripts\python -m pip install "setuptools<45"

[MichaIng]

We faced the same issue: https://github.com/mozilla-services/syncserver/issues/239
First I thought it's pythonhosted.org not respecting the Requires-Python flag but with the same mirror it works on Debian Stretch with Python 2.7.13 while it fails on Ubuntu Xenial with Python 2.7.12, otherwise pretty clean system setups, even tested in a Docker environment. The Python version cannot be the reason as the same issue was reported already with Ubuntu Bionic and Python 2.7.17.

So I am a bid puzzled where/on which level the bug needs to be searched, Python, virtualenv, pip, pythonhosted.org or is there something in the middle?
The workaround is clear but it would be great to investigate and fix the reason that setuptools 45 is pulled in the first place.