Setuptools: TypeError: 'output_dir' must be a string or None since setuptools 40.7.0

[hsoft]

We have a tricky regression in Gentoo with setuptools release 40.7.0. This affects many (all?) packages that involve build_ext. These errors aren't present in 40.6.3. I think I managed to have reproducible steps:

git clone https://github.com/python-pillow/Pillow.git
cd Pillow
virtualenv -p python2 env
. env/bin/activate
pip install -U setuptools
echo -e "[build]\nbuild-base = /tmp/foo" >> setup.cfg
mkdir /tmp/foo
python setup.py build
[...]
Traceback (most recent call last):
  File "setup.py", line 793, in <module>
    zip_safe=not (debug_build() or PLATFORM_MINGW), )
  File "/home/vdupras/src/Pillow/env/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "setup.py", line 695, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
    self.build_extension(ext)
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 499, in build_extension
    depends=ext.depends)
  File "/home/vdupras/src/Pillow/mp_compile.py", line 40, in _mp_compile
    output_dir, macros, include_dirs, sources, depends, extra_postargs)
  File "/usr/lib64/python2.7/distutils/ccompiler.py", line 329, in _setup_compile
    raise TypeError, "'output_dir' must be a string or None"
TypeError: 'output_dir' must be a string or None

Can you reproduce on your side?

It the fix isn't obvious, I can try myself on a PR soon.

(by the way, this is due to distutils's isinstance() call being made on str rather than basestring and only happens under py2. output_dir is unicode when it should be str)

[benoit-pierre]

This is probably due to #1180.

[reinout]

I think something similar is the cause of a zc.buildout test error. It only fails on 2.7 (pypy and python 3 are file) and it happens when we test a package with an extension.

It has to do with include_dirs instead of output_dir, but it sounds similar enough that the traceback (from travis-ci) could help:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 2316, in main
**kw
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
  File ".../python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
  File ".../python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
  File ".../python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 418, in run
self.easy_install(spec, not self.no_deps)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 660, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 705, in install_item
dists = self.install_eggs(spec, download, tmpdir)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 890, in install_eggs
return self.build_and_install(setup_script, setup_base)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 1158, in build_and_install
self.run_setup(setup_script, setup_base, args)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 1144, in run_setup
run_setup(setup_script, args)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 253, in run_setup
raise
  File ".../python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
yield
  File ".../python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 166, in save_modules
saved_exc.resume()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 141, in resume
six.reraise(type, exc, self._tb)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 154, in save_modules
yield saved
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
yield
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
  File "/tmp/tmpc2rzJKbuild/extdemo-1.5/setup.py", line 10, in <module>
ext_modules = [Extension('extdemo', ['extdemo.c'])],
  File ".../python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
  File ".../python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
  File ".../python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/bdist_egg.py", line 163, in run
self.run_command("egg_info")
  File ".../python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
  File ".../python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 296, in run
self.find_sources()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 303, in find_sources
mm.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 534, in run
self.add_defaults()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 570, in add_defaults
sdist.add_defaults(self)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
  File ".../python2.7/distutils/cmd.py", line 312, in get_finalized_command
cmd_obj.ensure_finalized()
  File ".../python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/build_ext.py", line 133, in finalize_options
_build_ext.finalize_options(self)
  File ".../python2.7/distutils/command/build_ext.py", line 159, in finalize_options
self.include_dirs.append(py_include)
AttributeError: 'unicode' object has no attribute 'append'

self.include_dirs apparently is a unicode object instead of a list.

I haven't myself contributed to that part of the buildout tests, but from looking at it, we're passing the result of an os.path.join() as include_dirs in a dict to setuptools.command.setopt.edit_config(). (I don't know what happens in there, though).

[reinout]

I've looked at my local 2.7 distutils, and in distutils/command/build_ext.py around line 150 I see something like this:

    if self.include_dirs is None:
        self.include_dirs = self.distribution.include_dirs or []
    if isinstance(self.include_dirs, str):
        self.include_dirs = self.include_dirs.split(os.pathsep)

That self.distribution.include_dirs is probably the os.path.join() output buildout puts into include_dirs.
The isinstance normally detected this as str and called "split", turning it into the (expected) list.

Now the value is a unicode instance, so this is another case where string_types should be used.

I did a bit of grepping, and there are some 25 cases where I see isinstance(something, str). #1180 didn't get all of them. Note that it will be a pain to fix them all with monkey patching...

Brainstorming: would it be an idea to stick with old-style strings on 2.7? I don't know exactly what #1180 fixes, but perhaps encoding it as byte strings again (only on 2.7) could be OK?

[hsoft]

Or revert the change, postpone it for v41 and make that release py3-only? isittimeyet?

[bivald]

I'm seeing this for pypy as well, installing packages such as grpcio

[jaraco]

I don't yet understand which part of the change in #1180 led to these failures. Reading through the code, it seems to me the only place that unicode would appear where previously str would be present is in the .cfg file parsing, and I don't see those config values set in setup.cfg... unless maybe the build tools are setting those values. Yes, that must be it...

Confirmed. As can be seen from the trace, the following is being set in setup.cfg:

[build]
build-base=/tmp/foo

And tracing the distutils code, I can see where output_dir is derived from build_temp and build_temp is derived from build_base.

[jaraco]

In the aforementioned commit, I have what I think is a suitable workaround. @hsoft or @reinout Can you possibly test with setuptools from https://m.devpi.net/jaraco/dev/setuptools/40.7.0.post20190128 and see if your builds work with that code present?

[rouge8]

Confirmed that https://m.devpi.net/jaraco/dev/setuptools/40.7.0.post20190128 fixed the building an affected package for me on OS X with Python 2.7

[gnossen]

@jaraco I can confirm that the artifact linked above fixes the regression for grpc.

[hsoft]

Yes, it fixes the problem demonstrated by the steps written in my initial comment.

[jaraco]

v40.7.1 is released now with the fix. Thanks for the report and sorry for any inconvenience.

[reinout]

Yeah, also the buildout tests run fine again.

@jaraco: nice fix!