Cert-manager: Installing cert-manager, not able to set nodeSelector for webhook

[marcelfalliere]

Describe the bug:

When installing cert-manager the node selection works for the cert manager but not for the two webhooks pods.

I run this command :

helm211 install stable/cert-manager --namespace kube-system \ --set ingressShim.defaultIssuerName=letsencrypt-prod \ --set ingressShim.defaultIssuerKind=ClusterIssuer \ --set nodeSelector."beta\.kubernetes\.io/os"=linux

It creates 3 pods :

kube-system   pod/pouring-cardinal-cert-manager-5995b6dfb-d9ccr   1/1     Running             0          24m
kube-system   pod/pouring-cardinal-webhook-6bdbb5dd79-fd4wl       0/1     ContainerCreating   0          24m
kube-system   pod/pouring-cardinal-webhook-ca-sync-bq5xs          0/1     ImagePullBackOff    0          24m

Describing each one of these 3 pods :

Pod 1, the cert manager :

````
kubectl114 describe pod/pouring-cardinal-cert-manager-5995b6dfb-d9ccr --namespace kube-system

...
Node: k8s-linuxpool1-32785198-2/10.240.0.4
....
Node-Selectors: beta.kubernetes.io/os=linux
````

Pod 2, the web hook :

````
kubectl114 describe pod/pouring-cardinal-webhook-6bdbb5dd79-fd4wl --namespace kube-system

...
Node: 3278k8s010/10.240.0.97
...
Node-Selectors:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27m default-scheduler Successfully assigned kube-system/pouring-cardinal-webhook-6bdbb5dd79-fd4wl to 3278k8s010
Warning FailedMount 4m35s (x10 over 25m) kubelet, 3278k8s010 Unable to mount volumes for pod "pouring-cardinal-webhook-6bdbb5dd79-fd4wl_kube-system(91085e95-762a-11e9-b263-000d3a496853)": timeout expired waiting for volumes to attach or mount for pod "kube-system"/"pouring-cardinal-webhook-6bdbb5dd79-fd4wl". list of unmounted volumes=[certs]. list of unattached volumes=[certs pouring-cardinal-webhook-token-kj9lj]
Warning FailedMount 31s (x21 over 27m) kubelet, 3278k8s010 MountVolume.SetUp failed for volume "certs" : secret "pouring-cardinal-webhook-webhook-tls" not found
````

Pod 3, the web hook ca (certificate authority ?) :

````
kubectl114 describe pod/pouring-cardinal-webhook-ca-sync-bq5xs --namespace kube-system

...
Node: 3278k8s010/10.240.0.97
...
Node-Selectors:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28m default-scheduler Successfully assigned kube-system/pouring-cardinal-webhook-ca-sync-bq5xs to 3278k8s010
Normal SandboxChanged 28m (x2 over 28m) kubelet, 3278k8s010 Pod sandbox changed, it will be killed and re-created.
Warning Failed 27m kubelet, 3278k8s010 Failed to pull image "quay.io/munnerz/apiextensions-ca-helper:v0.1.0": rpc error: code = Unknown desc = failed to register layer: re-exec error: exit status 1: output: ProcessBaseLayer \?\C:\ProgramData\docker\windowsfilter\1cc9148131131b65a93170304f9e5cb8f3460214e532e2641c762a11def90e43: The system cannot find the path specified.
Warning Failed 27m kubelet, 3278k8s010 Failed to pull image "quay.io/munnerz/apiextensions-ca-helper:v0.1.0": rpc error: code = Unknown desc = failed to register layer: re-exec error: exit status 1: output: ProcessBaseLayer \?\C:\ProgramData\docker\windowsfilter\bc52e3701d0c2f9155a5407387040a5ab913a20a33f243b3d3ec62b24f1532f6: The system cannot find the path specified.
Normal Pulling 26m (x4 over 28m) kubelet, 3278k8s010 Pulling image "quay.io/munnerz/apiextensions-ca-helper:v0.1.0"
Warning Failed 26m (x4 over 28m) kubelet, 3278k8s010 Error: ErrImagePull
Normal BackOff 3m9s (x106 over 28m) kubelet, 3278k8s010 Error: ImagePullBackOff
````

(truncated for readability)

The node 3278k8s010 is the one and only windows node in the cluster. Here the kubectl114 get node output :

NAME                        STATUS   ROLES    AGE   VERSION
3278k8s010                  Ready    agent    4d    v1.14.1
k8s-linuxpool1-32785198-0   Ready    agent    4d    v1.14.1
k8s-linuxpool1-32785198-1   Ready    agent    4d    v1.14.1
k8s-linuxpool1-32785198-2   Ready    agent    4d    v1.14.1
k8s-master-32785198-0       Ready    master   4d    v1.14.1

Expected behaviour:

The node selector be set for the two webhooks pods.

Steps to reproduce the bug:

Create a new k8s cluster (followed this guide to make an hybrid cluster, that is both windows and linux), and simple helm init (tiller was already installed).

Environment details::

kubectl114 version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

helm211 version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

• Cloud-provider/provisioner : AKS

/kind bug

[smax48]

I had the same issue. I managed to fix it by passing nodeSelector in a separate config-values.yaml with the following data:

nodeSelector:
  "beta.kubernetes.io/os": linux
webhook:
  nodeSelector:
    "beta.kubernetes.io/os": linux
cainjector:
  nodeSelector:
    "beta.kubernetes.io/os": linux

So all dependent charts get their own nodeSelector value ;)

[thomassampson]

Desperate for this feature to be added 👍🏾