Cert-manager: secret "cert-manager-webhook-webhook-tls" not found

[cbluth]

Bugs should be filed for issues encountered whilst operating cert-manager.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:

• use helm install jetstack/cert-manager
• webhook fails to start MountVolume.SetUp failed for volume "certs" : secret "cert-manager-webhook-webhook-tls" not found
  cert-manager doesnt work, cant apply clusterissuers, cant issue certs, nothing cert-manager related seems to be functioning properly

Expected behaviour:
functioning properly

Steps to reproduce the bug:
Steps to reproduce the bug should be clear and easily reproducible to help people
gain an understanding of the problem.

Anything else we need to know?:

Environment details::

• Kubernetes version: 1.14.1
• Cloud-provider/provisioner: bare-metal/kubespray
• cert-manager version: helm repo update, 0.8.0?
• Install method: helm
  /kind bug

user@laptop:~$ kubectl --context=dev3 -n kube-system get pod | grep cert-manager
cert-manager-6b7c647548-cgrfj                    1/1     Running             0          10m
cert-manager-cainjector-6bbbd87b97-vjrlc         1/1     Running             0          10m
cert-manager-webhook-5bdc99c85d-r462t            0/1     ContainerCreating   0          10m
user@laptop:~$ kubectl --context=dev3 -n kube-system describe pod cert-manager-webhook-5bdc99c85d-r462t
Name:               cert-manager-webhook-5bdc99c85d-r462t
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               node-6qjgv/10.10.23.251
Start Time:         Tue, 28 May 2019 16:11:33 +0200
Labels:             app=webhook
                    pod-template-hash=5bdc99c85d
                    release=cert-manager
Annotations:        <none>
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/cert-manager-webhook-5bdc99c85d
Containers:
  webhook:
    Container ID:  
    Image:         quay.io/jetstack/cert-manager-webhook:v0.8.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=2
      --secure-port=6443
      --tls-cert-file=/certs/tls.crt
      --tls-private-key-file=/certs/tls.key
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /certs from certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-blf9p (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-webhook-webhook-tls
    Optional:    false
  default-token-blf9p:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-blf9p
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                   From                 Message
  ----     ------       ----                  ----                 -------
  Normal   Scheduled    10m                   default-scheduler    Successfully assigned kube-system/cert-manager-webhook-5bdc99c85d-r462t to node-6qjgv
  Warning  FailedMount  108s (x4 over 8m39s)  kubelet, node-6qjgv  Unable to mount volumes for pod "cert-manager-webhook-5bdc99c85d-r462t_kube-system(7f21058b-8152-11e9-9a03-525400aaece4)": timeout expired waiting for volumes to attach or mount for pod "kube-system"/"cert-manager-webhook-5bdc99c85d-r462t". list of unmounted volumes=[certs]. list of unattached volumes=[certs default-token-blf9p]
  Warning  FailedMount  26s (x13 over 10m)    kubelet, node-6qjgv  MountVolume.SetUp failed for volume "certs" : secret "cert-manager-webhook-webhook-tls" not found
user@laptop:~$ kubectl --context=dev3 -n kube-system get secret --all-namespaces | grep cert-manager-webhook-webhook-tls
user@laptop:~$ 

[jaitaiwan]

Out of curiosity, does the cert-manager namespace have the correct label?

[cbluth]

```$ kubectl --context=dev3 get ns kube-system -oyaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"certmanager.k8s.io/disable-validation":"true","name":"kube-system"},"name":"kube-system"}}
creationTimestamp: "2019-05-22T12:54:58Z"
labels:
certmanager.k8s.io/disable-validation: "true"
name: kube-system
name: kube-system
resourceVersion: "1663"
selfLink: /api/v1/namespaces/kube-system
uid: ce1a4e71-7c90-11e9-9a03-525400aaece4
spec:
finalizers:

• kubernetes
  status:
  phase: Active
  ```
  @jaitaiwan

[cbluth]

is it a requirement to put it in the cert-manager namespace?

[cbluth]

kubectl --context=<some other cluster> -n kube-system get secret cert-manager-webhook-webhook-tls -oyaml | kubectl --context=dev3 -n kube-system apply -f -

copying the secret from another cluster seemed to fix it.

[remmeier]

I'm facing now the same issue. Myself I also installed everything in kube-system. Is there a problem with that? who is responsible for creating that tls secret? Namespace is also annotated as asked above.

[grrywlsn]

This is not a closed bug, also having this issue. Copying it manually from another cluster defeats the point of installing with Helm.

[lainra]

I got this bug but for maybe a different reason:

Make sure you don't have another certmanager running in your cluster as it will own the lease for leader controller election.
If the new cert-manager pod cannot get the lease, it will not run the secret bootstrap controller hence it will not be able to initialize the webhook pod!

[r1cebank]

I am having the same issue with v0.11.0 on a fresh cluster. Please reopen the issue

[DeamonMV]

I had the same trouble with secret "cert-manager-webhook-webhook-tls" not found

My cert-manager deployed from manifest, so update i did by this command:

kubectl apply      \
         --validate=false  \
        -f https://github.com/jetstack/cert-manager/releases/download/v0.8.1/cert-manager.yaml

For me helped just apply new manifest twice. After this pod cert-manager-webhook started as expected.

[antoniogomezalvarado]

The solution by @DeamonMV shows that version 0.8.0 has some bug since version 0.8.1 modifies the namespace to cert-manager ( instead of default)

[JannikSt]

I ran into the same issue, but @lainra hint helped. I removed and installed a new version of cert-manager using regular manifests, while not aware that a version of cert-manager was running that has been installed using helm.
Check:
helm ls --all cert-manager
if you have the same issue.
After removing all versions of cert-manager including the helm version using:
helm del --purge cert-manager
and installing v 0.12.0 using helm everything seems to work fine.

[papanito]

I am running into the same but there is only one cert-manager

helm ls --namespace cert-manager 
NAME            NAMESPACE       REVISION    UPDATED                                 STATUS      CHART                   APP VERSION
cert-manager    cert-manager    2           2019-12-16 18:40:14.296856384 +0100 CET deployed    cert-manager-v0.12.0    v0.12.0    

and

kubectl get pods --all-namespaces 
NAMESPACE       NAME                                       READY   STATUS              RESTARTS   AGE
cert-manager    cert-manager-784bc9c58b-xq25x              1/1     Running             0          20m
cert-manager    cert-manager-cainjector-85fbdf788-d8s5l    0/1     CrashLoopBackOff    9          28m
cert-manager    cert-manager-webhook-76f9b64b45-brpp5      0/1     ContainerCreating   0          28m
default         multitool                                  1/1     Running             0          88m
ingress-nginx   default-http-backend-67cf578fc4-lr5jw      1/1     Running             0          32h
ingress-nginx   nginx-ingress-controller-7gczj             1/1     Running             0          32h
ingress-nginx   nginx-ingress-controller-x5j2x             1/1     Running             0          32h
kube-system     calico-kube-controllers-5fd6f588f8-jhtl5   1/1     Running             1          107m
kube-system     calico-node-82s74                          1/1     Running             0          92m
kube-system     calico-node-qv7fg                          1/1     Running             0          92m
kube-system     coredns-5c59fd465f-nlwcw                   1/1     Running             0          32h
kube-system     coredns-5c59fd465f-z8jvg                   1/1     Running             0          32h
kube-system     coredns-autoscaler-d765c8497-hrkzk         1/1     Running             0          32h
kube-system     metrics-server-64f6dffb84-5mwrk            1/1     Running             0          32h
kube-system     rke-coredns-addon-deploy-job-mldcf         0/1     Completed           0          32h
kube-system     rke-ingress-controller-deploy-job-wxvt7    0/1     Completed           0          32h
kube-system     rke-metrics-addon-deploy-job-szd4v         0/1     Completed           0          32h
kube-system     rke-network-plugin-deploy-job-d9cbg        0/1     Completed           0          32h

and there is definitively no such secret cert-manager-webhook-webhook-tls

kubectl  get secret  -n cert-manager
NAME                                  TYPE                                  DATA   AGE
cert-manager-cainjector-token-m65nj   kubernetes.io/service-account-token   3      18m
cert-manager-token-rzmdx              kubernetes.io/service-account-token   3      18m
cert-manager-webhook-token-59qnz      kubernetes.io/service-account-token   3      18m

[AdimUser]

we are also facing the same issue @papanito facing.

[AdimUser]

I could able to fix this issue downgrading the cert-version to version 10, why decided to downgrade is, when you check the cert-manager releases, from release version 11, cert-manager has changes the apiVersions : https://github.com/jetstack/cert-manager/releases/tag/v0.11.0

Version 11, version 12 is not compatible with my kuberenetes cluster, friends please take this point into your mind also

cc @papanito

[JannikSt]

@papanito Whats your output of: kubectl describe pod cert-manager-cainjector-85fbdf788-d8s5l and kubectl logs cert-manager-cainjector-85fbdf788-d8s5l?
@AdimUser Glad it works for you, but why is version 11 / 12 not compatible? We have version 12 running smoothly, but had to adapt our api references.

[papanito]

 kubectl describe pod  cert-manager-cainjector-6659d6844d-mpxc7 -n cert-manager
Name:         cert-manager-cainjector-6659d6844d-mpxc7
Namespace:    cert-manager
Priority:     0
Node:         x.x.x.x/192.168.100.2
Start Time:   Tue, 17 Dec 2019 17:55:34 +0100
Labels:       app=cainjector
              app.kubernetes.io/instance=cert-manager
              app.kubernetes.io/managed-by=Tiller
              app.kubernetes.io/name=cainjector
              helm.sh/chart=cert-manager-v0.12.0
              pod-template-hash=6659d6844d
Annotations:  cni.projectcalico.org/podIP: 10.42.111.203/32
Status:       Running
IP:           10.42.111.203
IPs:
  IP:           10.42.111.203
Controlled By:  ReplicaSet/cert-manager-cainjector-6659d6844d
Containers:
  cert-manager:
    Container ID:  docker://674aeca3b8baed3c230c349e9bfea0f50b3cc287adddb6733e282e306712ed49
    Image:         quay.io/jetstack/cert-manager-cainjector:v0.12.0
    Image ID:      docker-pullable://quay.io/jetstack/cert-manager-cainjector@sha256:9ff6923f6c567573103816796df283d03256bc7a9edb7450542e106b349cf34a
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=2
      --leader-election-namespace=kube-system
    State:          Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 17 Dec 2019 17:56:11 +0100
      Finished:     Tue, 17 Dec 2019 17:56:41 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 17 Dec 2019 17:55:38 +0100
      Finished:     Tue, 17 Dec 2019 17:56:08 +0100
    Ready:          False
    Restart Count:  1
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-cainjector-token-lhz85 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  cert-manager-cainjector-token-lhz85:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-cainjector-token-lhz85
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                Message
  ----     ------     ----              ----                -------
  Normal   Scheduled  <unknown>         default-scheduler   Successfully assigned cert-manager/cert-manager-cainjector-6659d6844d-mpxc7 to x.x.x.x
  Normal   Pulled     9s (x2 over 42s)  kubelet, x.x.x.x  Container image "quay.io/jetstack/cert-manager-cainjector:v0.12.0" already present on machine
  Normal   Created    8s (x2 over 41s)  kubelet, x.x.x.x  Created container cert-manager
  Normal   Started    8s (x2 over 41s)  kubelet, x.x.x.x  Started container cert-manager
  Warning  BackOff    <invalid>         kubelet, x.x.x.x  Back-off restarting failed container

and

kubectl describe pod cert-manager-webhook-547567b88f-b7fzk    -n cert-manager
Name:           cert-manager-webhook-547567b88f-b7fzk
Namespace:      cert-manager
Priority:       0
Node:           x.x.x.x/192.168.100.1
Start Time:     Tue, 17 Dec 2019 17:55:36 +0100
Labels:         app=webhook
                app.kubernetes.io/instance=cert-manager
                app.kubernetes.io/managed-by=Tiller
                app.kubernetes.io/name=webhook
                helm.sh/chart=cert-manager-v0.12.0
                pod-template-hash=547567b88f
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/cert-manager-webhook-547567b88f
Containers:
  cert-manager:
    Container ID:  
    Image:         quay.io/jetstack/cert-manager-webhook:v0.12.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=2
      --secure-port=10250
      --tls-cert-file=/certs/tls.crt
      --tls-private-key-file=/certs/tls.key
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:6080/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:6080/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /certs from certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-webhook-token-lf56p (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-webhook-tls
    Optional:    false
  cert-manager-webhook-token-lf56p:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-webhook-token-lf56p
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                       From                   Message
  ----     ------       ----                      ----                   -------
  Normal   Scheduled    <unknown>                 default-scheduler      Successfully assigned cert-manager/cert-manager-webhook-547567b88f-b7fzk to y.y.y.y
  Warning  FailedMount  <invalid>                 kubelet, y.y.y.y  Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[cert-manager-webhook-token-lf56p certs]: timed out waiting for the condition
  Warning  FailedMount  <invalid> (x9 over 118s)  kubelet, y.y.y.y  MountVolume.SetUp failed for volume "certs" : secret "cert-manager-webhook-tls" not found

[papanito]

Can the issue please be re-opened?

[papanito]

Any clue how to fix this on short term? (Workaround)

[AdimUser]

@papanito Whats your output of: kubectl describe pod cert-manager-cainjector-85fbdf788-d8s5l and kubectl logs cert-manager-cainjector-85fbdf788-d8s5l?
@AdimUser Glad it works for you, but why is version 11 / 12 not compatible? We have version 12 running smoothly, but had to adapt our api references.

I suspect, my kubernetes cluster version is tool old to support new certs apis, This is only a doubt

[papanito]

I tried now version 0.10.0, 0.11.0 and 0.12.0 with the same issue. May this be related to latest k8s 1.16?

[papanito]

I've created a separate issue #2484 cause I guess is worth it.

[hanzvanaardt]

Hi there, I received this exact error and upon pod description received this information: Warning FailedMount (x8 over ) kubelet, gke-test-cluster-default-pool-fea900f5-sk7r MountVolume.SetUp failed for volume "certs" : secret "cert-manager-webhook-tls" not found

After two minutes the pod was still displaying the error ,however, after ten minutes the pod was working perfectly fine. Really strange.

[Seolhun]

I don't know why cert-manager didn't work suddenly.
I got help on the site(https://cert-manager.io/docs/tutorials/acme/ingress/.) and solved it.

[ainiml]

I get the error Secret \"cert-manager-webhook-tls\" not found:

E0526 04:13:09.767432 1 sources.go:193] cert-manager/inject-controller "msg"="unable to fetch associated secret" "error"="Secret \"cert-manager-webhook-tls\" not found" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"={"Namespace":"cert-manager","Name":"cert-manager-webhook-tls"}

But it looks like the secret exists?

$ ./kubectl-get-secrets.sh cert-manager      cert-manager-webhook-tls
Name:         cert-manager-webhook-tls
Namespace:    cert-manager
Labels:       <none>
Annotations:  cert-manager.io/allow-direct-injection: true

Type:  kubernetes.io/tls

Data
====
ca.crt:   1192 bytes
tls.crt:  1289 bytes
tls.key:  1679 bytes

[cbluth]

https://github.com/jetstack/cert-manager/issues/1730#issuecomment-541370895

[meyskens]

Can you please test this with 0.15 or 0.16, we did a lot of improvements. We no longer support 0.11.

Going to close this again if you see this in the latest version feel free to make a new issue for this.

/close

[jetstack-bot]

@meyskens: Closing this issue.

In response to this:

Can you please test this with 0.15 or 0.16, we did a lot of improvements. We no longer support 0.11.

Going to close this again if you see this in the latest version feel free to make a new issue for this.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Kampe]

testing with 1.0.1, this is still an issue

[meyskens]

@Kampe we no longer mount any secrets in, can you make a new issue?

[Kampe]

I spoke too soon, this is only an issue if you have multiple version issues going on, wipe clean and all is well.