Cert-manager: cm-acme-http-solver schedules to non amd64 nodes

[Vonor]

Since cert-manager does currently not support multiple archs I had to add an amd64 vm to my arm64 cluster. With Helm I could set up cert-manager to be scheduled to the amd64 node (applying a nodeselector with --set in helm install)

However, I couldn't find a way to tell the cm-acme-http-solver pods to only schedule on the amd64 node. Resulting in non executable pods and quite some fiddling to make it move over to the amd64 node.

I would propose the following ways to fix this.

• keep working on the multi arch support. If cert-manager runs on the k8s supported platforms the issue is basically non-existent anymore.
• add an option (maybe via config map?) to add a nodeselector for the solver pods.
• if a nodeselector is set for the cert-manager controller, apply it to the solver pods too
• while only amd64 is supported, add the nodeselector hardcoded

/kind bug

[munnerz]

Thanks for the issue!

keep working on the multi arch support. If cert-manager runs on the k8s supported platforms the issue is basically non-existent anymore.

1212 has been opened to produce arm and arm64 images for cert-manager itself, although in mixed-arch clusters you'll still need (2)

add an option (maybe via config map?) to add a nodeselector for the solver pods.

So we have #1097 which is tracking allowing a completely custom spec to be configured, this should help here.

if a nodeselector is set for the cert-manager controller, apply it to the solver pods too

This info isn't available via the downward API, and I'm cautious to make the assumption that users want this. I think we should expose first-class config options for this sort of thing (like (2)!) instead 😄

while only amd64 is supported, add the nodeselector hardcoded

We could do this pretty quickly and easily - although with #1212 in-flight, it would then mean you couldn't use these new ARM images 😅

I think for the time being, you'd need to 'taint' your non-amd64 nodes in order to prevent CM scheduling pods there, without changing the pod template itself 😬

[Vonor]

Hi James, thanks for the explanation and digging out the other reports.

When I opened the report all reports and RPs about arm64 seemed to either be stuck for a while already or closed due to the switch to bazel. So was kind of unsure if there still is work going on :)
Great to see the progress. Next Helm needs to follow so i can have a pure arm64 cluster :)

Regarding #1097, would you know if there is any work in progress?
As for mixed arch cluster, would it be possible to read the node label and selecting the image to use based on that? kind of like flannel is doing it when deploying the flannel worker pods

I think for the time being, you'd need to 'taint' your non-amd64 nodes in order to prevent CM scheduling pods there, without changing the pod template itself

Would you be so kind and tell me how to do that without other pods being refused to schedule to the arm64 nodes?

[jwalker343]

@Vonor You may be able to taint a node with "preferNoSchedule" which will tell kubernetes to "try it's best" to avoid scheduling pods on those nodes.

nodeSelector requirements override preferNoSchedule, so pods without nodeSelector will avoid the arm64 nodes, and pods that have a nodeSelector requiring arch=arm64 will still get scheduled on the arm64 nodes.

I am not sure what your existing pods/deployments/apps look like, this is just an example, and is how we're solving this same issue with mixed-os clusters (windows and linux)

[munnerz]

I'm going to close this in favour of #1097 - we'll soon have official ARM images too, which should help your situation 😄

/close

[jetstack-bot]

@munnerz: Closing this issue.

In response to this:

I'm going to close this in favour of #1097 - we'll soon have official ARM images too, which should help your situation 😄

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aaronkjones]

@munnerz
It looks like there are ARM images now. Thank you. :clap:

Do you know where this issue stands now?
I can launch cert-manager-arm, but the solver fails

$ kubectl logs cm-acme-http-solver-4n67h
standard_init_linux.go:207: exec user process caused "exec format error"