Kind: Getting Kind works on ARM64

[zhlhahaha]

We are trying to make kind works on ARM64 platform.
I summarize some task may need to do. Looking forward your suggestion
Tasks:

• [ ] 1 Ensure all codes can be built successfully on Arm64.
• [ ] 2 Enable kind cross build base images and node images for ARM platform on x86.
• [ ] 3 Get kind CI works on ARM64 platform
• [ ] 4 e2e-kind passed on Arm

[zhlhahaha]

@BenTheElder

[BenTheElder]

There are existing issues for this that should be looked at.

1) kind can be build for arm64, we publish binaries already, just not the node / base image, for which we only pre-publish a small set of images at the moment.

2) This is not as simple as it sounds. The node image build is not a normal image build, and I'm not sure this is actually something we're interested in at the moment.

3) Our CI is prow, see github.com/kubernetes/test-infra. It is not natively hosted on arm and I do not wish to own any additional infra myself at the moment.

4) kind e2e has already been known to work on arm. we helped figure this out before but nobody maintained it... https://testgrid.k8s.io/conformance-all#kind,%20v1.14%20(dev,%20ARM64)

[BenTheElder]

please see also https://github.com/kubernetes-sigs/kind/issues/166 https://github.com/kubernetes-sigs/kind/issues/188 ...

[BenTheElder]

kind is great in kubernetes CI because we can run it natively in the CI. Given that we need external infra to run ARM CI anyhow, I'm not really sure kind is an improvement over just kubeadm init on whatever external ARM box we have anyhow...?

even for development clusters, see https://github.com/kubernetes/minikube/issues/955

[BenTheElder]

I do want KIND to work on ARM long term, but from what I've gathered the goal here is testing kubernetes on ARM upstream for "official support" ...? In which case I'm not positive this is the most efficient work stream.

[zhlhahaha]

please see also #166 #188 ...

The issue is more complex than I thought, but we can make it step by step.
Thanks for your suggestion and informative comments.
I will read though your comments and related link first.

[lubinszARM]

From my perspective, I think the enablement of Prow on Arm64 is more helpful to our goal here is testing k8s on ARM for "official support".
We have tested the local Prow on x86. And I think the enablement of Prow on our local Arm64 server is our focus for the next job.
What's your opionion, @BenTheElder

[BenTheElder]

Prow itself does not need to be on ARM. Prow schedules workloads onto other clusters, but those clusters are Kubernetes, and the officially maintained ones are running on GKE because we have resources there and don't have much staffing to maintain a cluster.

Prow can run a job that sshes to an ARM box or similar without any changes today.

The issue continues to be that someone _maintains_ the test setup and infra on an ongoing basis, the existing test-infra team does not have the bandwidth to own more infra and is instead working to move infra out of the google team to the community in work-group-k8s-infra.

[BenTheElder]

_Most_ of our test clusters to this day are external clusters merely created from the CI jobs by ssh, cloud commands, etc. with no real relation to the CI itself.

Those resources can be arbitrary, except that someone must own and maintain them. The same team currently provides a pool of GCP projects on which GCE based cluster testing is performed, but the community also has some AWS resources available in CI.

[BenTheElder]

I would recommend the infra WG as the right place to discuss what infra can be in place https://github.com/kubernetes/community/tree/master/wg-k8s-infra

[lubinszARM]

@BenTheElder Thank you very much.
I see.
I will setup a local hybrid cluster to check Prow.

[lubinszARM]

Hi @BenTheElder
At least, we should add arm support to such images, like:
gcr.io/k8s-testimages/kubekins-e2e
gcr.io/k8s-testimages/bootstrap
...

So that, the jobs(integration.yaml, conformance-e2e.yaml...) which were scheduled by Prow can work well on the specific Arm64 nodes.

Is that right?

Thanks.

[BenTheElder]

possibly, I'm not sure how useful implementing integration is tbh versus just e2e, and for e2e as I said we don't really need to run prow on it, prow can just create a cluster on it.

in that regard I don't think testing another architecture would be different versus testing a cloud provider integration like one of the storage drivers, or the cluster API providers.

[BenTheElder]

reading these again, this is a dupe of https://github.com/kubernetes-sigs/kind/issues/166