Buildkit: How can the worker be configured to support running binaries for multiple archs?

[alicefr]

hi,

I saw this demo and I wonder how I can configure the worker to support running binaries for multiple archs.

Thanks!

[tonistiigi]

Copying some parts of the answer already on slack.

First, you need to configure the kernel to run the binaries for other arch using binfmt_misc kernel feature. There are many tutorials for this so you may look something like "emulating arm binfmt_misc" for your platform. https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html is a good explanation offered in slack. Docker for Mac also comes preconfigured with this support. You don't need to do anything specific for the container or build setup for this.

from @ijc in #buildkit

Yes they do -- see the F flag in https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html. Almost certainly needs a new enough userspace too
Doc for F was added in 2016 (in v4.8-rc1) so I guess the feature itself is a little older than that.

I can confirm the F flag was needed when I set up my system. And probably newer kernels have better results.

I'd recommend testing it out with something like after this setup.

docker run --rm arm64v8/alpine ls
docker run --rm armhf/alpine ls

I took the binaries for my system from https://github.com/multiarch/qemu-user-static/releases but probably any other source(like your package manager) works as well.

echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:OCF' > /proc/sys/fs/binfmt_misc/register
echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-static:OCF' > /proc/sys/fs/binfmt_misc/register

After you have done that and confirmed it works you need to let buildkitd know that additional platforms are supported. Eg. buildkitd ... --oci-worker-platform linux/amd64 --oci-worker-platform linux/arm64 and verify with buildctl debug workers ls

[alicefr]

@tonistiigi many thanks! I think I missed the 'F' option. I'll try it out and if it successful I'll close the issue!

[ijc]

FWIW in the Debian world qemu-user-static version 1:2.11+dfsg-1 (somewhere post Stretch/stable) wasn't new enough, but 1:2.12+dfsg-3 (current Buster/testing) is.

From debian/changelog it looks like 1:2.12~rc3+dfsg-1 was the version which added the F, then 1:2.12+dfsg-1 added a dependency so that the binfmt support tooling understood it. I had to sudo service binfmt-restart to get it to load.

Now the docker run --rm «foreign»/alpine ls but works for me.

[alehaa]

Is there any chance to run it with kernels before 4.8, missing the F option? Docker cloud autobuild service doesn't support it yet (kernel version is 4.4) , thus multi-arch images can't be build there.