Cert-manager: Supporting nodeSelector and tolerations on ACME solver podTemplate

[vslepakov]

Describe the bug:
nodeSelector setting on the helm chart is not respected by the webhook and cainjector components.
cm-acme-http-solver is also deployed to the wrong node pool/nodes
In my case those pods are scheduled on my windows nodes instead of linux nodes (win and lin node pools in AKS cluster)

Expected behaviour:
all components run on the node pool/nodes specifeid in nodeSelector

Steps to reproduce the bug:
On AKS deploy with set value nodeSelector."beta.kubernetes.io/os"=linux

Environment details::

• Kubernetes version (e.g. v1.14.0): v1.14.0
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): Azure AKS
• cert-manager version (e.g. v0.8.0): v0.8.0
• Install method (e.g. helm or static manifests): helm

/kind bug

[thomassampson]

I am also having this issue and I am not finding any workaround for scheduling the pods on my linux node and I am not using Helm 😒

[munnerz]

This is a dupe of #1673 😄

cm-acme-http-solver is also deployed to the wrong node pool/nodes

This is something we're looking into exposing via the newly added podTemplate field on the ACME Issuer's config 😄

[rclsilver]

Hi,
Could you append tolerations too?
Thx!

[alexpkent]

I fixed this by tainting the Windows nodes so the acme-http-solver is forced to the linux node:

kubectl taint nodes OS=Windows:NoSchedule

[cheukwing]

/assign

[TripleEmcoder]

Seems this is still impossible, or am I missing something? With a mixed-os Linux/Windows cluster the solver gets scheduled on Windows and fails to start.

[joostmeijles]

Having the same issue, having a mixed-os Linux/Windows cluster.

Used as values:

nodeSelector:
  "beta.kubernetes.io/os": linux
webhook:
  nodeSelector:
    "beta.kubernetes.io/os": linux
cainjector:
  nodeSelector:
    "beta.kubernetes.io/os": linux

but it does not make the cm-acme-http-resolver to be scheduled on a Linux node.

The proposed kubectl taint nodes OS=Windows:NoSchedule workaround does not work either.

This is quite blocking, is there a workaround?

[kieronlanning]

Can this be re-opened, as-per @joostmeijles's comments... we've exactly the same issue.

[robearlam]

I'm seeing the same issue as @joostmeijles as well, +1 for re-opening.

[robearlam]

For anyone coming to this later the solution is specifying the node selector on the ingress node iteself e.g.

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <VALID EMAIL HERE>
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            spec:
              nodeSelector:
                "kubernetes.io/os": linux

[meyskens]

Adding a default nodeselector seemed to be a sane solution however we currently support Kubernetes versions back up to 1.11 where beta.kubernetes.io/os is present but not kubernetes.io/os. Using the beta flag then again will cause issues in new Kubernetes versions where this will be removed.

At this point I think documenting this on https://cert-manager.io would be a better idea, once we are able to drop support for older versions I would be happy to add a default node selector, or maybe windows container versions ;)