Setuptools: upgrade to version 34 with modules that depends on setuptools for build/setup.py

[fruch]

We have a CI setup for running diffrent test repos, each repo has it's own requirements.txt
we build a virtualenv on each run and use that to run the tests.

our build mechines has old virtualenv, which install old version of setuptools to begin with.
once we do
pip install -r requirements.txt -U

it's brings in lots of packages that some of them are depnded on setuptool for the installtion (i.e. selenium)
cause now setuptool is depnded on six, it's not available in the same pip install, hence failing the installation.

currentlly we'll pin to older version of setuptools

I'm trying to update with a Dockerfile that will demonstrate the problem

[fruch]

Here is a Dockerfile that partly reproduce this issue:

# in our case we have python 2.7.3
FROM python:2.7

# and old version virtualenv installed
RUN pip install setuptools==3.6
RUN pip install virtualenv==1.11.6
RUN virtualenv .venv

RUN echo 'setuptools>=5.4' > requirements.txt
RUN echo 'six>=1.10.0' >> requirements.txt
RUN echo 'selenium==2.46.0' >> requirements.txt

RUN . .venv/bin/activate && python -m pip install --process-dependency-links -r requirements.txt

it's doesn't fail on six, but on other dependcey i guess packaging.version is missing

[agroszer]

breaks buildout too:
http://winbot.zope.org/builders/z3c.tabular_py_270_win32/builds/171/steps/buildout/logs/stdio

bin\buildout.exe

Upgraded:
  setuptools version 34.0.1;
restarting.
Generated script 'c:\\buildslave\\z3c.tabular_py_270_win32\\build\\bin\\buildout'.
Develop: 'c:\\buildslave\\z3c.tabular_py_270_win32\\build\\.'
Traceback (most recent call last):
  File "c:\temp\tmpegmfar", line 6, in <module>
    import os, setuptools
  File "d:\eggs\setuptools-34.0.1-py2.7.egg\setuptools\__init__.py", line 10, in <module>
    from six.moves import filter, map
ImportError: No module named six.moves
While:
  Installing.
  Processing develop directory 'c:\\buildslave\\z3c.tabular_py_270_win32\\build\\.'.

[jwg4]

I have an almost identical problem to @fruch, except that the failure is on a different requirement:

pip install --upgrade setuptools
Downloading/unpacking setuptools from https://pypi.python.org/packages/b1/d1/3df86b914d72b18d37266f964a11576f135eadbb7d2e3c5165377e4263bc/setuptools-34.0.1.zip#md5=9e907950a94d38c0295df0da2507c280
  Running setup.py egg_info for package setuptools
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "setuptools/__init__.py", line 12, in <module>
        import setuptools.version
      File "setuptools/version.py", line 1, in <module>
        import pkg_resources
      File "pkg_resources/__init__.py", line 70, in <module>
        import packaging.version
    ImportError: No module named packaging.version
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 3, in <module>

  File "setuptools/__init__.py", line 12, in <module>

    import setuptools.version

  File "setuptools/version.py", line 1, in <module>

    import pkg_resources

  File "pkg_resources/__init__.py", line 70, in <module>

    import packaging.version

ImportError: No module named packaging.version

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /var/lib/jenkins-slave/workspace/quotes_database_deploy/venv/build/setuptools
Storing complete log in /var/lib/jenkins-slave/.pip/pip.log

[msm1975]

I have identical problem to @jwg4:
Using version 34.0.1 (newest of versions: 34.0.1, 34.0.0, 33.1.1, 33.1.0, 32.3.1, 32.3.0, 32.2.0, 32.1.3, 32.1.2, 32.1.2, 32.1.1, 32.1.0, 32.1.0, 32.0.0, 32.0.0, 31.0.1, 31.0.1, 31.0.0, 31.0.0, 30.4.0$
Downloading/unpacking setuptools from https://pypi.python.org/packages/b1/d1/3df86b914d72b18d37266f964a11576f135eadbb7d2e3c5165377e4263bc/setuptools-34.0.1.zip#md5=9e907950a94d38c0295df0da2507c280 (fr$

Downloading from URL https://pypi.python.org/packages/b1/d1/3df86b914d72b18d37266f964a11576f135eadbb7d2e3c5165377e4263bc/setuptools-34.0.1.zip#md5=9e907950a94d38c0295df0da2507c280
Running setup.py egg_info for package setuptools

Traceback (most recent call last):

  File "<string>", line 3, in <module>

  File "setuptools/__init__.py", line 12, in <module>

    import setuptools.version

  File "setuptools/version.py", line 1, in <module>

    import pkg_resources

  File "pkg_resources/__init__.py", line 70, in <module>

    import packaging.version

ImportError: No module named packaging.version

Complete output from command python setup.py egg_info:

Traceback (most recent call last):

File "", line 3, in

File "setuptools/__init__.py", line 12, in

import setuptools.version

File "setuptools/version.py", line 1, in

import pkg_resources

File "pkg_resources/__init__.py", line 70, in

import packaging.version

ImportError: No module named packaging.version

---

Please, fix this issue soon! Our customer can't use our web application when this bug occurs!

[niko-fp]

same here.. 34.0.0 and 34.0.1 both fail
33.1.1 works fine for us

[georgealton]

same issue for me

12:33:58.750   File "/srv/jenkins/workspace/CloudFormation_Deployment_and_Test/venv/local/lib/python2.7/site-packages/setuptools/version.py", line 1, in <module>
12:33:58.750 
12:33:58.750     import pkg_resources
12:33:58.750 
12:33:58.750   File "/srv/jenkins/workspace/CloudFormation_Deployment_and_Test/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 70, in <module>
12:33:58.750 
12:33:58.750     import packaging.version
12:33:58.750 
12:33:58.750 ImportError: No module named packaging.version

I 've fixed my current issue by adding

appdirs

to my requirements.txt which is allowing me to use 34.x

[deppy]

Also the same here. Since the release of 34.x we get

Collecting setuptools (from ipdb==0.10.1->-r /setup/requirements_testing.txt (line 13))
  Downloading setuptools-34.0.1.zip (617kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-f2nbk3ik/setuptools/setuptools/__init__.py", line 12, in <module>
        import setuptools.version
      File "/tmp/pip-build-f2nbk3ik/setuptools/setuptools/version.py", line 1, in <module>
        import pkg_resources
      File "/tmp/pip-build-f2nbk3ik/setuptools/pkg_resources/__init__.py", line 70, in <module>
        import packaging.version
    ImportError: No module named 'packaging'

We've temporarily rolled the version we install to 33.1.1 in the meantime :-)

[asavah]

+1
setuptools depend on packaging,
packaging depends on pyparsing,
pyparsing depends on setuptools

Good job!

[idgserpro]

We're using buildout here (Plone world), and having the following traceback:

Generated script '/home/user/plone/zinstance/bin/buildout'.
Traceback (most recent call last):
  File "bin/buildout", line 13, in <module>
    import zc.buildout.buildout
  File "/home/user/.buildout/eggs/zc.buildout-2.5.3-py2.7.egg/zc/buildout/buildout.py", line 18, in <module>
    import zc.buildout.easy_install
  File "/home/user/.buildout/eggs/zc.buildout-2.5.3-py2.7.egg/zc/buildout/easy_install.py", line 26, in <module>
    import pkg_resources
  File "/home/user/.buildout/eggs/setuptools-34.0.1-py2.7.egg/pkg_resources/__init__.py", line 72, in <module>
    import packaging.requirements
  File "/home/user/.buildout/eggs/packaging-16.8-py2.7.egg/packaging/requirements.py", line 59, in <module>
    MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker")
TypeError: __call__() takes exactly 2 arguments (1 given)

Downgrading to 33.1.1 it works. I'm glad we have --setuptools-version in https://bootstrap.pypa.io/bootstrap-buildout.py. I know buildout is another project, but I'm posting here in case someone searches for this traceback.

[jaraco]

This behavior is by design (#581) - as mentioned in the changelog. What's not (currently) mentioned is that you have to be using a recent version of pip that supports wheel installs of setuptools (#940). If you're using an old virtualenv, you'll need to upgrade your pip before installing setuptools or pin to an older version of setuptools. Otherwise, this issue is essentially a duplicate of #937.

breaks buildout too

That looks like a separate issue, but probably needs to be addressed in buildout. See #939.

@idgserpro Your traceback is suspiciously different from that of others. You did get packaging 16.8 installed (as an egg!) but apparently there's a bug in that installation. As you can see, the behavior is invoked with a simple import packaging.requirements. You may want to dig deeper and see if that error occurs in isolation with the same import, which would implicate the environment or packaging or something about the way that package was installed.

[zroadhouse-rmn]

This behavior is by design (#581) - as mentioned in the changelog. What's not (currently) mentioned is that you have to be using a recent version of pip that supports wheel installs of setuptools (#940).

Am I interpreting this correctly that sdist installation of setuptools is no longer supported in v34? That's a pretty significant breaking change.

[jaraco]

Yes and no. You can't install from sdist without having installed setuptools dependencies first, but if you ensure that the dependencies are installed, then you can install from sdist. Eventually, PEP 518 support should come to pip at which point pip will be able to install setuptools from sdist without prerequisites. It is a significant breaking change, and I appreciate the efforts some of you have to expend to help make this transition. I've done everything I can to limit the impact, and if you can think of something more that can be done, please do share.

[fruch]

First of all sorry for the noise I've did on tweeter.( I assumed a call for issues, can be a place to raise them)

I think the importent missing part here, is a big callout about this change, in all possible mediums, a week notice that no one heard is not enough.

This like notices in earlier versions (same as pip did for various things)

I've seen this type of things happens in various parts related to python packaging, and it was almost was in supprize, even that I follow lots of the blog's related to python,band a lots of other vocal people related to python, please use all of them for those types of call outs and breaking changes.

All of us are stuck in those werid situations that upgrading a part, or changing order of what needs to be done, means a lot of work for multiple people. (yes it's a sad/bad place, but we're there, and breakage doesn't make it better...)

Anyhow keep us the good work, so the python packaging world would be rock solid. (And hopefully end our suffering :))

[sla-shi]

I used this to workaround and it helped me to install the requirements
pip install setuptools==33.1.1
pip install -r requirements.txt

[domichel]

Hi folks,
Same issue on gentoo. I reported it here: https://bugs.gentoo.org/show_bug.cgi?id=617030