Pipeline: Add support for other architectures

[vdemeester]

Kubernetes support more architecture than x86_64, Tekton Pipeline should support those too.

What this means is :

• [ ] the project compiles on the architecture (and tests runs too)
• [ ] controller and webhook should be able to run on supported architecture
• [ ] images like entrypoint, creds-init, git-init, … should also be able to run on those architectures
• [ ] container images published from tekton (releases or nightly) should be multi-arch images (aka 1 reference for multiple images chose depending on the architecture)
• [ ] bonus : e2e test on cluster of these architectures

Architecture to support

• [x] arm64 : an initial take on this has been done by @lisa, see here
• [x] ppc64
• [x] s390x
• [ ] arm
• [ ] and maybe of course windows :dancer:

Architecture supported

• [ ] arm64 : an initial take on this has been done by @lisa, see here
• [ ] arm
• [ ] ppc64
• [ ] s390x
• [ ] and maybe of course windows :dancer:

Marking this as an epic as it is a big chunk of work and is gonna need to be split up into several issues.

[lisa]

As part of my investigation that @vdemeester mentioned I had to change the base images in .ko.yaml*, even though busybox has an arm64 manifest. I believe this is because in google/ko's commands/config.go which exhibits underlying behaviour in go-containertools (described https://github.com/GoogleContainerTools/kaniko/pull/646 as it relates to kaniko) to not pick image manifests according to what arch is currently making the request. In short, this issue likely blocks on either go-containerregistry doing something different with its default platform or having ko passing the desired platform along as in the kaniko PR.

[vdemeester]

cc @ImJasonH @mattmoor (for the ko part)

[ImJasonH]

https://github.com/google/ko/issues/6 cc @jonjohnsonjr

[jonjohnsonjr]

I added some more details to that bug with what I had in mind.

[vincent-pli]

@vdemeester @ImJasonH
What about this project? do we have ppc64 supported?

[vdemeester]

@vincent-pli as of today, we only support linux/amd64. We should aim to support more combinaison of GOOS and GOARCH though.

[mattmoor]

The container tools nerd in me would love to see this happen, but I'm concerned about our current testing capability for this matrix.

@jonjohnsonjr can probably guide someone through this (he has an "ugly" PoC), but we probably won't be able to prioritize this before Knative Serving "v1"s.

[jonjohnsonjr]

Published what I was playing with here: https://github.com/google/ko/pull/38

If anyone's interested in picking this up, I'm happy to chat.

[vincent-pli]

@lisa
I saw your article: https://thedoh.dev/kubernetes/tektoncd-pipeline.html
Seems you did not change the defaultBaseImage in .ko.yaml, did the controller work?

[lisa]

@lisa
I saw your article: https://thedoh.dev/kubernetes/tektoncd-pipeline.html
Seems you did not change the defaultBaseImage in .ko.yaml, did the controller work?

@vincent-pli Yes, it worked. The build process (./hack/release.sh) built a controller image which I'm storing as thedoh/arm64-tektoncd-pipeline-cmd-controller.

[skaegi]

I know there is work in-house on doing a Tekton for s390x. Is the idea here that our build/release process would also have images for these other architectures (oops... nevermind I see one image to rule them all above)? What are we looking for here then? Do we want a tick mark when we can confirm Tekton releases works on a particular architecture?

[apasrini]

@skaegi We are right now trying Tekton for s390x. Is it supported or it is yet to be complete? Or is it available in a branch so that we could try?

[vdemeester]

@apasrini It's not "officially" supported, as "we do not ship s390x images" but it should be possible. We (Red Hat & IBM) are working on having OpenShift Pipelines supporting s390x. In parallel we should work upstream to make sure it supports it to (cc @barthy1). To be able to build and release s390x (and other platforms) for tekton, we depend on https://github.com/google/ko/pull/38 — or we should modify our release process to be able to not use ko for other architectures in the meantime.

@afrittoli @ImJasonH @sbwsg @dibyom we also could try to use a patch version of ko (based on https://github.com/google/ko/pull/38) to try this out (and gather feedback for ko so that it gets in at some point).

[barthy1]

@vdemeester I've played a little bit with the latest ko version (without taking into account https://github.com/google/ko/pull/38)
A little bit strange behaviour takes place, but there is some dirty way to use ko for building s390x even now. So what I can see:
1) s390x host and s390x base image in .ko.yaml (defaultBaseImage: docker.io/s390x/busybox:latest) -> s390x image and "Architecture": "s390x" in manifest
2) s390x host and multi-arched base image in .ko.yaml (defaultBaseImage: alpine:latest) -> s390x image and "Architecture": "amd64" in manifest
3) amd64 host and s390x base image in .ko.yaml (defaultBaseImage: docker.io/s390x/busybox:latest) -> amd64 image and "Architecture": "s390x" in manifest
4) amd64 host and multi-arched base image in .ko.yaml (defaultBaseImage: alpine:latest) -> amd64 image and "Architecture": "amd64" in manifest

Result: First option provides a way to have s390x image build with current ko, but it requires to have s390x hardware

[mattmoor]

I'd love to get Jon's PR moving forwards, and we were just talking last week about some thoughts here. Let's discuss there?

[tonysmithio]

Is there any update surrounding ARM64 support for the tekton controller and webhook images, as well as pipeline workloads? ARM64 should be 1st class citizen in regard to architecture support.

[jonjohnsonjr]

https://github.com/google/ko/pull/38 has landed which should unblock this, if anyone wants to jump on it.

Also happy to hear any feedback if that's not sufficient. I still need to write docs and stuff but it's pretty straightforward to play around with:

1. Set a multi-platform image as the base image.
2. Use --platform=all

[docularxu]

I did a native Arm64 build of teckton/pipeline on Arm64 servers. Also, I had docker images built, everything is run and tested on Arm64. Is there anyway I can contribute my work to you?

The description of my work can be found here: Build tekton-v0.15.2 on Arm64.md

For the tests, I run a lighthouse+tekton pipeline.