Cilium: Cilium enable on aarch64

[Jianlin-lv]

We have ported Cilium master version on Arm server and deployed Nginx to verify and test k8s cluster that composed by three Arm nodes, the status of the pods and the connection between the nodes are normal.
I hope to contribute to cilium support for Arm64, and Related work includes the following aspects:

1. Fix compiling and running issue on Arm64;
2. Add arm64 support for building cilium-related images;
3. Add multi-arch support for cilium-related images;
4. Add arm64 support for cilium CI/CD.

[Jianlin-lv]

Does the cilium community have Arm server for CI/CD?

[tgraf]

Does the cilium community have Arm server for CI/CD?

We are using packet.net and several managed k8s providers so I'm sure we can extend coverage to include arm.

[Jianlin-lv]

Does the cilium community have Arm server for CI/CD?

We are using packet.net and several managed k8s providers so I'm sure we can extend coverage to include arm.

If Arm64 bare metal server available for cilium community, it's great convenient to build Arm docker images;
If Arm64 hardware is not available, QEMU is an option to build Arm images on X86 host.
Under these two different conditions, the methods for implementing Dockerfile to support Arm are different;
Woudle you please provide more detailed information about community`s hardware?

[aanm]

@Jianlin-lv https://www.packet.com/cloud/servers/c2-large-arm/

[ulm0]

Hi, is armv8 (arm64) only planned for support? can armv7 (arm32) be supported as well? if there's anything i can do to help in this matter, let me know, pls.

[Jianlin-lv]

Compiling and running issue on Arm64 have been fixed by https://github.com/cilium/cilium/pull/10204

[Jianlin-lv]

hi @tgraf
I dont think this issue have finised yet, Now the community does not support building multi-arch image to support Arm64, the PRs has not been merged. In addition，Travis Arm64 job fails in the unit tests, Im working on unit tests to supprot Arm64 and trying to fixed currently issue.

[aanm]

Re-open by previous comment ^^

[joestringer]

@Jianlin-lv @nickb937 what's the difference between PRs #9969 and #10618 ? /cc @borkmann

[Jianlin-lv]

@Jianlin-lv @nickb937 what's the difference between PRs #9969 and #10618 ? /cc @borkmann

The goals of the two patches above are the same, and the implementation is slightly different.

[aanm]

cc @tinatsou
Next items:

• [ ] Current CI connects to Packet.net, and then runs x86 VMs inside
  Action Item (Tina): Figure out how to run arm64 VMs
  Do we need arm64 hosts provisioned as well, etc.

[aanm]

I've managed to create a dummy image tag with 2 different architectures here:

https://hub.docker.com/layers/cilium/cilium-dev/v1.7.1/images/sha256-f4cee36a89dc550df357bf26d58dfdd304429ac48e4b0574274eec74ccfb58ce?context=explore

This was accomplished by:

1. Enabling experimental feature on docker CLI:

$ cat ~/.docker/config.json 
{
       [...]
        "experimental": "enabled"
}

1. Creating a manifest for a docker image for amd64 and another for arm64 (I used a golang arm64 image just for PoC)

docker manifest create docker.io/cilium/cilium-dev:v1.7.1  docker.io/cilium/cilium:v1.7.1 golang@sha256:925a02df65f7f05968f7acec19adc76ba6ce35f6c8a7edef2edbe2444e291323

1. Push manifest into docker

docker manifest push docker.io/cilium/cilium-dev:v1.7.1

Future work
Right now it does not seem that docker itself supports building arm64 images on docker hub (but I could be wrong) so we would need to find an way to build those images and push them into docker.io and quay.io.

For each release we would need to create a manifest manually. Creating a manifest does not mean that we have to build the images in the same machine where the manifest is created, the images just need to be published before hand.

Since we can't have a docker image and a manifest with the same tag we would need to change the docker tags to cilium/cilium:vX.Y.Z-${ARCH} or cilium/cilium-${ARCH}:vX.Y.Z and the manifest would be the official docker tag docker.io/cilium/cilium:vX.Y.Z.

[dewet22]

@aanm I've recently had to do something similar for another cross-arch docker project. I build it using docker buildx and you can see the pipeline config which is GitLab-specific but should be easily translatable into other CI platforms. At the end of the build process it pushes multiarch manifest automatically:

% docker manifest inspect registry.gitlab.com/dewet/pi-temp:latest
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:493b60922d44eeb254c041bf826d6c0b2b5c2320a2dd9a8605994f7f52569238",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:b93e9e8167f86b2c160af3d634cb21e9847d1468f0b8e661198043d54b998998",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

[Jianlin-lv]

@aanm Regarding the issue of building an arm64 image, it is best to have an Arm server, if not, QEMU is an good option to build Arm images on X86 host.

There is a script in repo that supports creating and pushing the manifest list
https://github.com/cilium/cilium/blob/master/contrib/scripts/push_manifest.sh

[Jianlin-lv]

@aanm I tried to build cilium-builder image on Travis, I think it worked at both AMD64 and Arm64.
https://travis-ci.com/github/Jianlin-lv/cilium/builds/160664536
Using Travis to build multi-arch images is a good option

[aanm]

@Jianlin-lv thank you for testing it out. I've meet with @joestringer to discussed how the distribution mechanism that we will use for Cilium starting in 1.8.

Given that neither docker.io nor quay.io have the ability to perform the build of aarch64 we will stop the automatic builds that we have for docker images on those platforms for 1.8 and foregoing.

As the distribution of an image which has a build for two or more different architures is done with the help of a docker manifest we will create those manifests with github actions suggested in https://github.com/cilium/cilium/issues/9898#issuecomment-613357082.

In the first phase we need to solve https://github.com/cilium/proxy/pull/20#pullrequestreview-396515817

In the second phase, the distribution of those manifests will be done exclusively in docker.io/cilium/cilium-dev, docker.io/cilium/operator-dev, docker.io/cilium/docker-plugin-dev, docker.io/cilium/proxy-dev, quay.io/cilium/cilium-dev, quay.io/cilium/-operator-dev, quay.io/cilium/docker-plugin-dev, quay.io/cilium/proxy-dev and quay.io/cilium/cilium-envoy-builder-dev.

In the third phase, the distribution of those manifests will be backported to 1.7 so that we can test how github actions work when a git tag is done.

In the fourth phase we will remove the automatic builds of latest / master for docker.io/cilium/cilium, docker.io/cilium/operator, docker.io/cilium/docker-plugin, docker.io/cilium/proxy, quay.io/cilium/cilium, quay.io/cilium/operator, quay.io/cilium/docker-plugin, quay.io/cilium/proxy, quay.io/cilium/cilium-envoy-builder and push the manifests into those repositories instead of *-dev.

[aanm]

https://github.com/cilium/proxy/pull/20#discussion_r414582725

[stuartpb]

I'm a little puzzled. What's the status of arm64 support for Cilium? The Docker tags for cilium/cilium only show images available for amd64, even though all the cilium/cilium-dev tags are multi-arch (which sounds like the "second phase" of https://github.com/cilium/cilium/issues/9898#issuecomment-616598428).

Is there a timeline for introducing multi-arch support to the mainline images? How soon until a user would be able to roll out Cilium on a hybrid-architecture cluster?

[joestringer]

@stuartpb we've had advanced users building Cilium themselves since 1.8 or earlier and deploying in their own environments. Over time, we are accepting patches to improve the automation for delivering images. Right now, there are some limitations around Envoy / L7 policies in the cilium/cilium-dev images, but I would suggest trying them out and providing feedback on them. We welcome community participants to help try out the images and report any issues they might face.

We don't have a specific timeline around moving the main cilium/cilium images to multi-arch at the moment, but it's something we are investigating as part of a wider CI improvement effort.

[errordeveloper]

WIP PR to build Envoy image is here: https://github.com/cilium/proxy/pull/27