Setuptools: Packages are not ignored on specific platforms with 38.2.0

[thijstriemstra]

We define this for our library (https://github.com/rm-hull/luma.led_matrix/blob/master/setup.py#L41):

install_requires = [
    'luma.core>=1.1.1',
    'rpi_ws281x;platform_machine=="armv7l" and platform_system=="Linux"',
    'ws2812;platform_machine=="armv7l" and platform_system=="Linux"'
]

Because these last 2 packages should only be installed on ARM systems. But it seems this broke in 38.2.0 (our travis started complaining).

$ pip install --ignore-installed --upgrade setuptools pip tox

Collecting setuptools

  Using cached setuptools-38.2.0-py2.py3-none-any.whl

Collecting pip

  Using cached pip-9.0.1-py2.py3-none-any.whl

Collecting tox

  Using cached tox-2.9.1-py2.py3-none-any.whl

Collecting virtualenv>=1.11.2; python_version != "3.2" (from tox)

  Using cached virtualenv-15.1.0-py2.py3-none-any.whl

Collecting py>=1.4.17 (from tox)

  Using cached py-1.5.2-py2.py3-none-any.whl

Collecting six (from tox)

  Using cached six-1.11.0-py2.py3-none-any.whl

Collecting pluggy<1.0,>=0.3.0 (from tox)

Installing collected packages: setuptools, pip, virtualenv, py, six, pluggy, tox

Successfully installed pip-9.0.1 pluggy-0.6.0 py-1.5.2 setuptools-38.2.0 six-1.11.0 tox-2.9.1 virtualenv-15.1.0

And then later you see they're installed anyway:

  Using cached pygame-1.9.3-cp27-cp27mu-manylinux1_x86_64.whl

Collecting ws2812 (from luma.led_matrix>=1.0.6->luma.examples==0.0.0)

Collecting rpi-ws281x (from luma.led_matrix>=1.0.6->luma.examples==0.0.0)

  Using cached rpi_ws281x-3.0.4.tar.gz

[thijstriemstra]

cc @benoit-pierre

[benoit-pierre]

I can't reproduce this locally (using tox -e py27). This kind of issue (markers in install_requires' requirements being ignored) looks like something that was fixed in 36.2.0. Note that for compatibility with older versions, using extras_require is better:

  setup.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git i/setup.py w/setup.py
index f1ef0b6..1e2ae50 100644
--- i/setup.py
+++ w/setup.py
@@ -40,8 +40,6 @@ test_deps = [

 install_deps = [
     'luma.core>=1.1.1',
-    'rpi_ws281x;platform_machine=="armv7l" and platform_system=="Linux"',
-    'ws2812;platform_machine=="armv7l" and platform_system=="Linux"'
 ]

 setup(
@@ -61,6 +59,9 @@ setup(
     setup_requires=pytest_runner,
     tests_require=["mock", "pytest", "pytest-cov"],
     extras_require={
+        ':platform_machine=="armv7l" and platform_system=="Linux"': [
+            'ws2812', 'rpi_ws281x',
+        ],
         'docs': [
             'sphinx >= 1.5.1'
         ],

[thijstriemstra]

@benoit-pierre thanks for your feedback! Just to confirm, extras_require behaves the same as install_requires (also when installing using a wheel)?

[benoit-pierre]

OK, I see the issue: starting with 38.2.0, setuptools started using wheels, and the wheel for luma.led_matrix is invalid:

> unzip -p ~/downloads/luma.led_matrix-1.0.6-py2.py3-none-any.whl '*/METADA
Requires-Dist: luma.core (>=1.1.1)
Requires-Dist: rpi-ws281x
Requires-Dist: ws2812
Requires-Dist: sphinx (>=1.5.1); extra == 'docs'
Requires-Dist: flake8; extra == 'qa'
Requires-Dist: rstcheck; extra == 'qa'
Requires-Dist: mock; extra == 'test'
Requires-Dist: pytest (>=3.1); extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'

Note the missing markers.

[thijstriemstra]

Ah. So this is an setuptools issue? cc @rm-hull

[benoit-pierre]

I guess the wheel was generated with setuptools<36.2.0, a version affected by the bug I described above, and so the markers were stripped.

[benoit-pierre]

It is a setuptools issue, but not with 38.2.0.

[thijstriemstra]

Ok, so we (and other projects) will have to re-generate our wheels in order to be compatible with 38.2.0?

[benoit-pierre]

It's not a problem of compatibility with setuptools, try installing luma.led_matrix with pip, you'll see the same failures attempting to build ws2812/rpi_ws281x.

[benoit-pierre]

• short term fix: regenerate the wheel for the latest version of luma.led_matrix with setuptools>=36.2.0
• long term fix: I suggest using extras_require to avoid invalid wheels when using setuptools<36.2.0

[thijstriemstra]

Thanks, will take a look and get back to you (and hopefully close this ticket).

[benoit-pierre]

For the record, I can reproduce the issue locally when running tox in the right project (luma.examples and not luma.led_matrix)...

[thijstriemstra]

@benoit-pierre luma.examples pulls the 'old' wheel in which explains it... ?

[benoit-pierre]

Did you update the wheel on PyPI?

[thijstriemstra]

Sorry btc price distracting me again, I'll take a look soon.

@rm-hull what version of setuptools did you use to generate the wheel for luma.led_matrix? e.g.

pip list

I guess we should also try to install/use the wheel version of the library in a tox build to catch these kind of wheel errors..

[benoit-pierre]

The invalid wheel output above is with the one from PyPi. Regenerating the wheel locally with setuptools==37.0.0 gives me a valid wheel:

> unzip -p dist/luma.led_matrix-1.0.6-py2.py3-none-any.whl '*/METADATA' | grep Requires-Dist
Requires-Dist: luma.core (>=1.1.1)
Requires-Dist: ws2812; platform_machine=="armv7l" and platform_system=="Linux"
Requires-Dist: rpi-ws281x; platform_machine=="armv7l" and platform_system=="Linux"
Requires-Dist: sphinx (>=1.5.1); extra == 'docs'
Requires-Dist: rstcheck; extra == 'qa'
Requires-Dist: flake8; extra == 'qa'
Requires-Dist: pytest (>=3.1); extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: mock; python_version < "3.3" and extra == 'test'

[thijstriemstra]

Awesome, I tried on a Raspberry Pi and your patch produced the same result (as our old solution) when running setup.py and it's compatible with older versions so I'll go ahead and apply that. After we regenerate the wheel (and make sure the local setuptools is always >= 37 it should be fine). Thanks for your help @benoit-pierre!

[thijstriemstra]

@benoit-pierre one last question, does this also apply to:

install_deps = [
    'pillow>=4.0.0',
    'smbus2',
    'monotonic;python_version<"3.3"'
]

Should monotonic be moved to extras_require as well?

[benoit-pierre]

Yes! That's what setuptools>=36.2.0 does internally, move requirements with markers from install_requires to extras_require with the syntax:

':{marker}': [requirements]

[thijstriemstra]

Cool, thanks.

[rm-hull]

what version of setuptools did you use to generate the wheel for luma.led_matrix?

It was 36.0.1

Have now updated to 38.2.1