Cibuildwheel: Seamless ARM builds using emulation through qemu docker images

[breznak]

I'm not sure this is in scope of the project, but it would be a unique functionality, since you aim to create a truly multiplatform wheels.

Situation:

• no (free) CI supports native (HW) ARM(64) builds
• you can overcome that by spinning a docker image to emulate arm on amd64 HW.
• pypa/manylinux2014 already supports arm* docker images (for building wheels)
• CIBW depends on the os.platform to build the wheels (?)

Here's an example how we build for ARM on x86_64 HW in CI:
https://github.com/htm-community/htm.core/blob/master/.github/workflows/arm.yml
and the Dockerfile:
https://github.com/htm-community/htm.core/blob/master/Dockerfile

The CIBW wheels are built here
https://github.com/htm-community/htm.core/blob/master/.github/workflows/release.yml#L96

for the list of platforms
https://github.com/htm-community/htm.core/blob/master/.github/workflows/release.yml#L17

Workaround/solution:

• I plan to add a duplicit OS, say [ubuntu-latest, ubuntu-18.04] and spin docker if the "hack" OS is ran: if: matrix.os == 'ubuntu-latest' run docker and emulate ARM64.

Would then CIBW pick up the ARM platform for manylinux2014? https://github.com/pypa/manylinux#manylinux2014

Do you see a more elegant way to integrate this?

[henryiii]

Could this be done for other arch's, like arm 32? Since those currently are Travis only, AFAICT.

[breznak]

Could this be done for other arch's, like arm 32?

you can emulate almost any platform, bitness this way. Ofc, speed is an issue.

[YannickJadoul]

• no (free) CI supports native (HW) ARM(64) builds

Travis CI does provide ARM64 (and we support it): https://docs.travis-ci.com/user/multi-cpu-architectures/#testing-on-multiple-cpu-architectures

• you can overcome that by spinning a docker image to emulate arm on amd64 HW.

So you don't need the actual hardware? Do you have an example to play around with?

• pypa/manylinux2014 already supports arm* docker images (for building wheels)

Yes, and we support it if there's a CI offering it, like Travis CI

• CIBW depends on the os.platform to build the wheels (?)

Yes, I believe so, but only to distinguish between Linux, macOS, and Windows. The architecture is detected with platform.machine(). Does this get changed on the docker images? Is there a way to fool/tell platform.machine() it's running on a different architecture?

Here's an example how we build for ARM on x86_64 HW in CI:
https://github.com/htm-community/htm.core/blob/master/.github/workflows/arm.yml
and the Dockerfile:
https://github.com/htm-community/htm.core/blob/master/Dockerfile

Ah, so it's still not running on ARM, but doing cross-compilation? Quite similar to https://github.com/dockcross/dockcross/, then? Or how does it work?

The manylinux wheels need to be built inside a docker image, though. Is it possible to run these docker images inside of one of these docker images for other architectures?

Workaround/solution:

• I plan to add a duplicit OS, say [ubuntu-latest, ubuntu-18.04] and spin docker if the "hack" OS is ran: if: matrix.os == 'ubuntu-latest' run docker and emulate ARM64.

Would then CIBW pick up the ARM platform for manylinux2014? https://github.com/pypa/manylinux#manylinux2014

As laid out above, it'll mostly depend on platform.machine() and the ability to run the manylinux2014 docker images. I'm quite happy/curious to see a proof of concept (maybe with a fork of cibuildwheel), though, if you want to try stuff.

Do you see a more elegant way to integrate this?

Depends a bit on the whole setup. If things work, at first, I could imagine making sure cibuildwheel picks up the correct platform and architecture, etc, and leaving up the setup of the outer docker images to users. Then, as a next step, if it's possible to automate, I could envision cibuildwheel to do the setup of these images. But before making any plans or claims about that, I'd need more details on how things work and some first steps or a proof of concept.

[breznak]

Travis CI does provide ARM64 (and we support it)

that's really cool! :+1: Unfortunately, we've recently migrated to GH Actions, which don't.

So you don't need the actual hardware? Do you have an example to play around with?

yes, you don't need. That's the point of the emulation:
https://github.community/t/testing-against-multiple-architectures/17111/6
in our exaple:
https://github.com/htm-community/htm.core/blob/master/.github/workflows/release.yml#L131

The architecture is detected with platform.machine(). Does this get changed on the docker images?

great, I think this will tell aarch64 on the arm64 docker image.

Ah, so it's still not running on ARM, but doing cross-compilation?

similar, but it is really runing on ARM64 (the normal aarch64 OS image can be used), just SW emulated.

Is it possible to run these docker images inside of one of these docker images for other architectures?

exactly. It should work as x86(real HW) -> aarch64(docker image) -> manylinux2014:aarch64 (the PYPI build env, arch64 version) -> aarch64.whl

I'm quite happy/curious to see a proof of concept (maybe with a fork of cibuildwheel), though, if you want to try stuff.

Cool, I'm happy you see potential of the idea!
About platform.machine() -- works! :heavy_check_mark:

$ docker run  htm-arm64-docker python -c 'import platform; print(platform.machine())'
aarch64

About CIBW fork, I'm not sure I'll have time to dig deep into another (new for me) project now, but I can be of any help with docker x84/arm here.

[breznak]

To run the above example, please try:

docker run --privileged --rm multiarch/qemu-user-static:register
docker build -t htm-arm64-docker -f Dockerfile-pypi .
docker run --rm htm-arm64-docker uname -a
docker run --rm htm-arm64-docker python -c 'import platform; print(platform.machine())'

[YannickJadoul]

Thanks, that's great news!

If I have some time, I'll play around with things a bit, but if it's this easy, there's a chance you could just docker run --rm htm-arm64-docker cibuildwheel or so! As far as I'm concerned, it might then be nicer to not change cibuildwheel too much but just add a few lines to the docs?

[henryiii]

I think there might not be many or any changes in cibuildwheel, since this is already supported on travis; and I think travis also emulates this, doing something similar internally. I would love to be able to build for all arch's on GHA. :)

[YannickJadoul]

Indeed. The one thing I'm thinking of is that we might need to make sure some environment variables are transferred. cibuildwheel checks GITHUB_WORKFLOW to detect if it's running on a CI (unless this is overridden by passing --platform).

[breznak]

you could just docker run --rm htm-arm64-docker cibuildwheel or so! As far as I'm concerned, it might then be nicer to not change cibuildwheel too much but just add a few lines to the docs?

you'd still have to create the Dockerfile and know about docker and run the

docker run --privileged --rm multiarch/qemu-user-static:register
docker build -t htm-arm64-docker -f Dockerfile-pypi .

Since CIBW aims to "make it easy" for all the builds (it already handles python interpreters/versions, which many CI already support), I'd think ability to build for many archs (aarm64, i686, arm32, ...any supported by manylinux2014) this might actually be some useful feature.

Indeed. The one thing I'm thinking of is that we might need to make sure some environment variables are transferred. cibuildwheel checks GITHUB_WORKFLOW to detect if it's running on a CI

might be an issue, we could pass select variables to the docker image. Maybe there's a way to mirror all env vars?

[YannickJadoul]

you'd still have to create the Dockerfile and know about docker and run the

docker run --privileged --rm multiarch/qemu-user-static:register
docker build -t htm-arm64-docker -f Dockerfile-pypi .

Since CIBW aims to "make it easy" for all the builds (it already handles python interpreters/versions, which many CI already support), I'd think ability to build for many archs (aarm64, i686, arm32, ...any supported by manylinux2014) this might actually be some useful feature.

We'll need to see how things go. Step by step.

might be an issue, we could pass select variables to the docker image. Maybe there's a way to mirror all env vars?

Sure. Not saying it's a problem. I'm saying it's the first thing that comes to mind on what might not immediately work out of the box.

[hoffmang9]

@YannickJadoul and others.

We're building ARM64 wheels on Github Actions with two self hosted runners on ec2 a1.xlarge instances running Ubuntu 18.04 lts.

Example workflow: https://github.com/Chia-Network/bls-signatures/blob/master/.github/workflows/build-aarch64.yml

It's a decent temporary workaround until we have Github Hosted ARM.

[YannickJadoul]

Well, this is an open source project. So it's great to hear that things are working beyond what's officially supported in cibuildwheel, but we can't really test it until GitHub actions offers ARM64 support :-/

[hoffmang9]

That's an open source project too - Apache 2.0. Now we can afford to run ARM64 ec2 instances for corporate sponsorship reasons. There are folks offering arm runners for free. Worth googling.

Also, my point wasn't about cibuildwheel - it was more for folks trying to build ARM wheels until we all have github hosted runners for ARM. Sorry to cause confusion.

[YannickJadoul]

That's an open source project too - Apache 2.0. Now we can afford to run ARM64 ec2 instances for corporate sponsorship reasons. There are folks offering arm runners for free. Worth googling.

Oh, right, I see. Sorry for misunderstanding; that's interesting indeed! :-)

Also, my point wasn't about cibuildwheel - it was more for folks trying to build ARM wheels until we all have github hosted runners for ARM. Sorry to cause confusion.

Sorry, got that wrong, then!

[Cadair]

Would it be possible to cross compile wheels using something like these images? https://github.com/dockcross/dockcross I haven't looked into this yet, just wondering if anyone else has?

[henryiii]

In https://github.com/PyTables/PyTables/issues/823, @amotl mentioned https://github.com/marketplace/actions/run-on-architecture - this might be perfect for this? I _might_ have time to test it (dealing with a Dec 1 deadline currently for a package).

[henryiii]

By the way, I've attempted this here, and I ran into:

cibuildwheel: Docker not found. Docker is required to run Linux builds. If you're building on Travis CI, add `services: [docker]` to your .travis.yml.If you're building on Circle CI in Linux, add a `setup_remote_docker` step to your .circleci/config.yml

Since this is already running in a container, should I try to keep going and try to get docker to run inside the image?

[YannickJadoul]

should I try to keep going and try to get docker to run inside the image?

If you know how to do it, sure, but if I recall, running docker in docker isn't too trivial?

[russkel]

Edit: Please see PR #469

Forgive me if I am missing something, but isn't this straightforward given what is in this blog post?

https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/

Seeing as cibuildwheel fires up docker instances, and the images already exist, can we not add a qemu flag to the desired architectures we want to build wheels for, and do the required QEMU/binfmt setup in the initialisation stage as well?

$ russ @ sherlock in ~ [17:57:35]
↳ docker pull docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
a7996909642ee92942dcd6cff44b9b95f08dad64: Pulling from docker/binfmt
5d6ca6c8ba77: Pull complete 
b26a8e2c75fc: Pull complete 
3436361ddd98: Pull complete 
Digest: sha256:758ca0563f371b384cfd67b6590b5be2dc024fef45bc14a050ae104f0caad14e
Status: Downloaded newer image for docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
docker.io/docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

$ russ @ sherlock in ~ [17:57:46]
↳ docker pull quay.io/pypa/manylinux2014_aarch64:latest
latest: Pulling from pypa/manylinux2014_aarch64
9f74aa7d9ab9: Pull complete 
e431742274fa: Pull complete 
a768fe896be5: Pull complete 
Digest: sha256:fcf1fe6dc4f85ba698a78b63d1a91499a7d827b128d950b2d6a1f0a9a4ee9903
Status: Downloaded newer image for quay.io/pypa/manylinux2014_aarch64:latest
quay.io/pypa/manylinux2014_aarch64:latest

$ russ @ sherlock in ~ [18:03:16]
↳ docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

$ russ @ sherlock in ~ [18:03:29]
↳ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64
flags: OCF
offset 0
magic 7f454c460201010000000000000000000200b7
mask ffffffffffffff00fffffffffffffffffeffff

$ russ @ sherlock in ~/Devel/python-moos on  ci-actions ✖︎ [18:14:46]
↳ uname -a
Linux sherlock 5.9.11-2-MANJARO #1 SMP PREEMPT Tue Nov 24 15:40:17 UTC 2020 x86_64 GNU/Linux

$ russ @ sherlock in ~/Devel/python-moos on  ci-actions ✖︎ [18:11:52]
↳ docker run --rm -it quay.io/pypa/manylinux2014_aarch64:latest bash
[root@1050d552dd52 /]# uname -a
Linux 1050d552dd52 5.9.11-2-MANJARO #1 SMP PREEMPT Tue Nov 24 15:40:17 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
[root@1050d552dd52 /]#