Linkerd2: Build: Support Multi Arch

[aliariff]

Feature Request

What problem are you trying to solve?

Right now, if we want to build Linkerd to be able to work with another architecture for example like the ARM by running bin/docker-build inside Raspberry Pi, it will not build correctly(the resulting image is not runnable inside the Raspberry Pi) because of the existing image that is pulled for runtime image and also builder image is not supporting multi-arch[1] therefore it will not run in another architecture.

https://github.com/linkerd/linkerd2/blob/f535c2d3d2f0da76bd62f3cbc1e39994a773ed21/bin/docker-build-base#L17-L23

https://github.com/linkerd/linkerd2/blob/f535c2d3d2f0da76bd62f3cbc1e39994a773ed21/bin/docker-build-go-deps#L17-L24

How should the problem be solved?

It would be great if we can customize the target build architecture.
I am proposing to add a new environment key called BUILD_ARCH so that we can define in what architecture the resulting image should be based on.

The behavior should be like this:
if BUILD_ARCH NOT set: just do as current behavior do.

if BUILD_ARCH is set:

• Add suffix to the tag based on the BUILD_ARCH value ex: BUILD_ARCH=arm -> gcr.io/linkerd-io/go-deps:4c8f4294.arm, gcr.io/linkerd-io/base:2019-02-19.01.arm. With this addition of a suffix, the code will build the image instead of pulling it from the registry because it does not exist.
• Add --platform=(BUILD_ARCH value) when running the docker build[2], with this setting the docker will pull the base images with the defined platform.

Another improvement that I propose is to remove the hardcoded image string in the Dockerfile and move it as an argument, so we can put it in one single place that generates the tag(with or without suffix) and always add --build-arg when running the docker build.

docker build ... \
        --build-arg RUNTIME_IMAGE=repo:base_image_tag \
        BUILDER_IMAGE=repo:go_deps_tag

base_image_tag and go_deps_tag will generate the tag.
The output ex: base_image_tag = base:2019-02-19.01
The output ex: go_deps_tag = go-deps:4c8f4294.arm

Any alternatives you've considered?

The solution that I do in order to build correctly for arm is changing the hardcoded tag(like add suffix) so it forces the code to build it instead of pulling it from registry, another solution is I just remove the code part that doing the docker pull, so straight to docker build without trying to pull. But all of this I need to perform inside my Raspberry Pi in order to docker detects my current architecture and use the correct images when building base and go-deps, another solution I can manually change the code part to use --platform=arm at the docker build. This is works because debian:stretch-20190204-slim and golang:1.11.5 those images are supporting multi-arch.

With the solution I propose, now it is possible to do cross-build/cross-compile. I can just run BUILD_ARCH=arm bin/docker-build in the Mac and push the images to the registry and the raspberry pi can just pull it and run it properly.

How would users interact with this feature?

BUILD_ARCH=arm bin/docker-build
BUILD_ARCH=arm64 bin/docker-build

This will result in the output images is based on the arm.
Supported BUILD_ARCH value

https://github.com/docker/app/blob/d91c8816f56b9b2376f752a06dbd14394a78e4a3/vendor/github.com/containerd/containerd/platforms/database.go#L61

Reference

[1] https://blog.docker.com/2017/11/multi-arch-all-the-things/
[2] https://docs.docker.com/engine/reference/commandline/build/

Related issue

https://github.com/linkerd/linkerd2/issues/1165

[aliariff]

I found that add suffix is not a good practice, better we use a different repo name.

Example:
gcr.io/linkerd-io/go-deps-arm:4c8f4294
gcr.io/linkerd-io/base-arm64:2019-02-19.01

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[joakimr-axis]

Wouldn't using multi-arch manifests for the images that are pulled be a better way than setting them manually with an environment variable, if you are building for the architecture you are running on?

[aliariff]

@joakimr-axis

Yes, that would be the case when the runtime image which is base already support multi-arch manifest, but currently is not.
Another reason to use env var is so that we can reuse the builder image which is go-deps and just use the same image to perform cross-compile.

[joakimr-axis]

Yes, that would be the case when the runtime image which is base already support multi-arch manifest, but currently is not.

Absolutely. I understand this is your step 2 (or step N+1), and that sounds good.

Another reason to use env var is so that we can reuse the builder image which is go-deps and just use the same image to perform cross-compile.

That is good. Cross-compile is nice.

[joakimr-axis]

With images for amd64 and arm32v7 built by jaerik, I created docker manifests providing multi-arch support. I put those manifests on dockerhub to be easily available. I run a cluster of mixed units amd64 & arm32v7, and with an amd64 machine on master where I run the linked command through sed to slightly change the image URLs:

linkerd install --registry docker.io/d97jro |sed 's#\(d97jro/\)#\1linkerd-io-#g' |kubectl apply -f -

Also, for testing the same goes for the emojivoto app:

curl -sL https://run.linkerd.io/emojivoto.yml | sed 's#\(image: \)buoyantio#\1d97jro#g;s#\(svc\|web\):\(.*\)$#\1#g' |kubectl apply -f -

As far as I can see, it seem to works perfectly on my cluster.

From a linkerd user perspective, I reckon one would like the official image URLs to point towards such multiarch manifests in order to just run the same linkerd manifest regardless of underlying architecture.

[grampelberg]

That's super helpful, thank you @joakimr-axis!

[joakimr-axis]

That's super helpful, thank you @joakimr-axis!

I'll ask @jaerik if he used some extra magic/TLC when he built those images that he could/should share here.

[GaruGaru]

I'm trying to rebuild the linkerd images for multiple architectures (linkerd2, linkerd2-proxy, linkerd2-proxy-init).
The golang services are pretty straight forward to build, i manage to compile them properly using both docker buildx and 'raw' docker manifests using github actions.

I also manage to build the linkerd-proxy rust service for multiple architectures using cross but probably using --target option for cargo is enougth.

The tricky part is the fact that linkerd2 downloads the proxy binary during the build (https://github.com/linkerd/linkerd2/blob/master/bin/fetch-proxy) so we need to change the github action to publish different assets (one for each achitecture) and change the fetch-proxy script to fetch the right binary.

I guess that We also need to recompile the controller and web ui images.

Another problem is that linkerd uses custom prometheus and grafana images and not the official ones from dockerhub that actually supports multiple architectures

[joakimr-axis]

...and there we got @jaerik's build setup that for a build host of any architecture will build containers for all the architectures.

[aliariff]

Supported since https://github.com/linkerd/linkerd2/releases/tag/edge-20.8.1

[joakimr-axis]

Supported since https://github.com/linkerd/linkerd2/releases/tag/edge-20.8.1

This is so good!