Cibuildwheel: Future version: pip not pulling macosx_x86_64 on dual-architecture Python?

[Legrandin]

I have a question related to the upcoming change introduced in https://github.com/joerick/cibuildwheel/commit/513ef49cd43b4c4d80fcf11afa48cefde8deb784 that moves all MacOS build to 64 bits. The platform tag is going to change from macosx_10_6_intel to macosx_x86_64.

While I agree that the 32-bit support can be dropped, as far I understand from:

https://github.com/MacPython/wiki/wiki/Spinning-wheels

On dual-architecture Python <3.8 builds (like Python.org Python and system Python), pip will not pick the x86_64 wheel, even if it runs on 64-bits. I have not tested it myself, but I wanted to check if it is also your understanding?

The same website suggests to lie a little, by compiling for 64-bits only, but keeping the intel platform tag (which in theory is reserved to "fat" binaries only). I can't say how valid their suggestion is...

[YannickJadoul]

@Legrandin Hmmm, thanks for bringing this up! I assume you are referring to the following paragraph?

Similarly, you may want to build 64-bit only binaries, but allow your wheel name to claim dual-architecture compatibility. This allows your wheel to work for all system and Python.org Pythons, and you will only be caught out on the tiny, perhaps empty set of people running dual architecture Python in 32-bit mode.

But I can't explicitly find a claim that 64-bit-only wheels won't work on dual-architectures?

I did do a test on a mac I have access to, renaming a ..._intel.whl file to ..._x86_64.whl, and it does seem to be working in my case:

ai31:~ yannick$ ls /Library/Frameworks/Python.framework/Versions/3.6/bin/
2to3            f2py3.6         pip3.6          python3-config      python3.6m-config
2to3-3.6        idle3           pydoc3          python3.6       pyvenv
easy_install-3.6    idle3.6         pydoc3.6        python3.6-32        pyvenv-3.6
f2py            pip         python3         python3.6-config
f2py3           pip3            python3-32      python3.6m
ai31:~ yannick$ /Library/Frameworks/Python.framework/Versions/3.6/bin/pip3 --version
pip 18.1 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip (python 3.6)
ai31:~ yannick$ /Library/Frameworks/Python.framework/Versions/3.6/bin/pip3 install -v -f . praat-parselmouth==0.4.0.dev0
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-ephem-wheel-cache-wts9nyxp
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-svoif636
Created requirements tracker '/private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-svoif636'
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-install-45h5gmqa
Looking in links: .
Collecting praat-parselmouth==0.4.0.dev0
  1 location(s) to search for versions of praat-parselmouth:
  * https://pypi.org/simple/praat-parselmouth/
  Skipping link . (from -f); not a file
  Getting page https://pypi.org/simple/praat-parselmouth/
  Looking up "https://pypi.org/simple/praat-parselmouth/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/praat-parselmouth/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/praat-parselmouth/
    Found link https://files.pythonhosted.org/packages/ea/50/7ffe1a04842aa2cd2a600e054db2e4d53c6741ea4203ca6c3eddff0c59b0/praat-parselmouth-0.1.0.tar.gz#sha256=d7ed44c5ac66c29307f149bb5d7913050d9c70604d652bbca2f8923700036a5e (from https://pypi.org/simple/praat-parselmouth/), version: 0.1.0
    Skipping link https://files.pythonhosted.org/packages/10/0a/594633341df49ade8c9029daa649df112885291c6ee226281e324984cb2a/praat_parselmouth-0.1.0-cp27-cp27m-macosx_10_6_intel.whl#sha256=5406ffd917043e06b897d1a53c89c4e4fa0d2dc3e52961479f5515c601b983bb (from https://pypi.org/simple/praat-parselmouth/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/1b/56/63d8892e1f9834e2a08624dfd421f261246ae120eb7e5b4b2f74a44537c4/praat_parselmouth-0.1.0-cp34-cp34m-macosx_10_6_intel.whl#sha256=00069b033cc95f22815c6e23d5092e3abab348b34116e1f2ee199e40d09b4125 (from https://pypi.org/simple/praat-parselmouth/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/ff/ee/06bf0fd03fde144fefc88338f8a0262eb5472a6178d46e95150f221927d0/praat_parselmouth-0.1.0-cp35-cp35m-macosx_10_6_intel.whl#sha256=9f5c26883acee2a9d3ab8cb130061330fc84c6e70e9a87c739c273bd4da4cd1b (from https://pypi.org/simple/praat-parselmouth/); it is not compatible with this Python

...

  Local files found: /Users/yannick/praat_parselmouth-0.4.0.dev0-cp36-cp36m-macosx_10_14_x86_64.whl
  Using version 0.4.0.dev0 (newest of versions: 0.4.0.dev0)
  Added praat-parselmouth==0.4.0.dev0 from file:///Users/yannick/praat_parselmouth-0.4.0.dev0-cp36-cp36m-macosx_10_14_x86_64.whl to build tracker '/private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-svoif636'
  Removed praat-parselmouth==0.4.0.dev0 from file:///Users/yannick/praat_parselmouth-0.4.0.dev0-cp36-cp36m-macosx_10_14_x86_64.whl from build tracker '/private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-svoif636'
Requirement already satisfied: numpy>=1.7.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from praat-parselmouth==0.4.0.dev0) (1.18.1)
Installing collected packages: praat-parselmouth

Successfully installed praat-parselmouth-0.4.0.dev0
Cleaning up...

But I'd be very keep to know we're right, so if you can think of other situations to test, we should definitely check!

[Legrandin]

But I can't explicitly find a claim that 64-bit-only wheels won't work on dual-architectures?

The claim in the Spinning-Wheel page is this one:

Pip will accept a wheel with platform tags 10_12_x86_64 only on macOS >= 10.12, and for a Python that has single architecture x86_64. In particular, Pip will not install this wheel if you are using a dual architecture Python. Most Python.org Pythons and system Python are dual architecture.

I don't know exactly how to trigger this, but I plan to have a look at pip source code to understand better how their logic works...

[YannickJadoul]

Ah, I had not seen that; sorry!

But is this not trying to explain the platform tags? I.e., that 10_12 is macOS >= 10.12, and that x86_64 is 64-bit?

[Legrandin]

Yes, in the first sentence there is the explanation. However, the rest (Pip will not install this wheel if you are using a dual architecture Python...) hints at x86_64 not being installable by (say) an older system Python.

[YannickJadoul]

Yes, good point. Yet somehow, it works in this one trial. Is there somewhere where/someone who we could ask about this?

Let's tag @mayeut and @Czaki, already, as they both dived into macOS wheel-building.

[YannickJadoul]

@Legrandin

At least on pip's master branch, it seems that both intel and x86_64 are considered to be "supported":

https://github.com/pypa/pip/blob/82b4566a7f2cbca99bec0ca337a299826c311609/src/pip/_internal/pep425tags.py#L107

https://github.com/pypa/pip/blob/82b4566a7f2cbca99bec0ca337a299826c311609/src/pip/_vendor/packaging/tags.py#L352-L407

We'd need to go back in the commit history to figure out if this was ever different and when it changed, I guess.

[Czaki]

Argh...
In this moment I do not have Idea how to properly check this.
I do not divide to pip code and do not understand how it determine which wheel should be installed.

[mayeut]

I did a test on python 3.6 (dual-arch) by installing numpy (unfortunately, 3.5 is published as intel)
pip 8.0.2 does install the x86_64 wheel with no issue. (pip 8.0.2 predate python 3.6)
pip 7.1.2 fails to do so.
Anyway, given the fact that pip version older than 10.0.0 do not have proper SSL support on macOS python 3.5, I think we're safe to say that there won't be any issue with this.

MacBook-Pro-de-Matthieu:cibuildwheel Matt$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -m pip -V
pip 8.0.2 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (python 3.6)
MacBook-Pro-de-Matthieu:cibuildwheel Matt$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -m pip install numpy
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/82/63/eee643cc97f2bd22da87420f28fb6cd4b940c25f6eff6c4d2ca2e24a7022/numpy-1.18.1-cp36-cp36m-macosx_10_9_x86_64.whl (15.2MB)
    3% |█                               | 495kB 71kB/s eta 0:03:25^C
Operation cancelled by user
You are using pip version 8.0.2, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

MacBook-Pro-de-Matthieu:cibuildwheel Matt$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -m pip -V
pip 7.1.2 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (python 3.6)
MacBook-Pro-de-Matthieu:cibuildwheel Matt$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -m pip install numpy
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/40/de/0ea5092b8bfd2e3aa6fdbb2e499a9f9adf810992884d414defc1573dca3f/numpy-1.18.1.zip (5.4MB)
    6% |██                              | 327kB 660kB/s eta 0:00:08^C
Operation cancelled by user
You are using pip version 7.1.2, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[YannickJadoul]

Thank you, once again, so much, @mayeut!

So I should've just tried older versions of pip... But anyway, pip 7.1.2 is 4.5 years old, so do we really still want to support that? pip itself is warning you to update, so I'd argue that we might choose to not support such an old version (if even numpy isn't doing that, should we?).

What do you think, @Legrandin? Do you feel we should still support these old Python versions?

[joerick]

Thanks @mayeut !

Pip versions used to download packages from pypi in the last month (Mac only):

SELECT
  COUNT(*) AS num_downloads,
  REGEXP_EXTRACT(details.installer.version , r"^(\d+\.).*") as pip_version
FROM `the-psf.pypi.downloads*`
WHERE 
    details.installer.name = "pip" 
    AND details.system.name = "Darwin" 
    AND _TABLE_SUFFIX BETWEEN FORMAT_DATE(
          '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
       AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
GROUP BY pip_version
ORDER BY num_downloads DESC

| pip_version | num_downloads | percent
| -- | -- | --
| 20. | 28437581 | 44.85%
| 19. | 24887711 | 39.26%
| 18. | 6366572 | 10.04%
| 10. | 1079742 | 1.70%
| 9. | 2460360 | 3.88%
| 8. | 114387 | 0.18%
| 7. | 12809 | 0.02%
| 6. | 13434 | 0.02%
| 1. | 26256 | 0.04%

If only pip<8 is affected, seems like this issue would affect only 0.08% of Mac PyPI users.

[Legrandin]

Thanks all for the great analysis! If it just boils down to a pip upgrade, I am sure all (the few) affected users will be able to deal with this. Closing the issue...

[YannickJadoul]

Thanks for notifying us, though! It's a tricky subject, so good to have checked it! :-)