Cert-manager: Stuck at "Created new CertificateRequest resource..."

[juliohm1978]

Describe the bug:
Certificate object is created from existing ingress, but cert-manager is unable to issue the certificate.

Steps to reproduce the bug:
Installed cert-manager OTB with

helm upgarde --install cert-manager jetstack/cert-manager --version v0.12.0

Environment details::

• Kubernetes version (e.g. v1.16.3):
• Cloud-provider/provisioner: baremetal / kubeadm
• cert-manager version (e.g. v0.12.0):
• Install method: helm

/kind bug

After cert-manager installation, all Ingresses are scanned and Certificate objects are created. However, a new Ingress that needs a new Certificate is stuck waiting for the CertificateRequest.

# kubectl describe cert valerianogarolli.com-tls
...
Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  GeneratedKey  4m51s  cert-manager  Generated a new private key
  Normal  Requested     4m51s  cert-manager  Created new CertificateRequest resource "valerianogarolli.com-tls-559059946"

# kubectl describe CertificateRequest valerianogarolli.com-tls-559059946
Error from server (NotFound): certificaterequests.certmanager.k8s.io "valerianogarolli.com-tls-559059946" not found

This is the issuer used.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [REDACTED]
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Related log messages

I1207 16:49:03.006678       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.006746       1 sync.go:57] cert-manager/controller/certificates "level"=0 "msg"="certificate resource not found for key"  "key"="default/valerianogarolli.com-tls"
I1207 16:49:03.006768       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.006776       1 controller.go:129] cert-manager/controller/ingress-shim "level"=0 "msg"="syncing item" "key"="default/valeria" 
I1207 16:49:03.017374       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/valeria" 
I1207 16:49:03.020579       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.021055       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" 
I1207 16:49:03.021253       1 sync.go:479] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "state"="Pending"
E1207 16:49:03.021435       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
I1207 16:49:03.021464       1 conditions.go:155] Setting lastTransitionTime for Certificate "valerianogarolli.com-tls" condition "Ready" to 2019-12-07 16:49:03.021459244 +0000 UTC m=+705.884105375
I1207 16:49:03.021947       1 controller.go:129] cert-manager/controller/ingress-shim "level"=0 "msg"="syncing item" "key"="default/valeria" 
I1207 16:49:03.022095       1 sync.go:163] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="valeria" "resource_namespace"="default" 
I1207 16:49:03.022154       1 sync.go:176] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate resource is already up to date for ingress" "related_resource_kind"="Certificate" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="valeria" "resource_namespace"="default" 
I1207 16:49:03.022211       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/valeria" 
I1207 16:49:03.042844       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.042877       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.043130       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" 
I1207 16:49:03.043247       1 sync.go:479] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "state"="Pending"
E1207 16:49:03.043375       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
I1207 16:49:03.043451       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.043498       1 controller.go:129] cert-manager/controller/ingress-shim "level"=0 "msg"="syncing item" "key"="default/valeria" 
I1207 16:49:03.043583       1 sync.go:163] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="valeria" "resource_namespace"="default" 
I1207 16:49:03.043684       1 sync.go:176] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate resource is already up to date for ingress" "related_resource_kind"="Certificate" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="valeria" "resource_namespace"="default" 
I1207 16:49:03.043756       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/valeria" 
I1207 16:49:03.132035       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.497458       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.497511       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.497521       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
E1207 16:49:03.497548       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
I1207 16:49:03.497574       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.497586       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.497465       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.497620       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
I1207 16:49:03.497620       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.497639       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.497472       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.497653       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
I1207 16:49:03.497665       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.497669       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
I1207 16:49:03.497687       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.585138       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
I1207 16:49:03.585258       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.585289       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.585727       1 sync.go:361] cert-manager/controller/certificates "level"=0 "msg"="no existing CertificateRequest resource exists, creating new request..." "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" 
I1207 16:49:03.661986       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.662047       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "valerianogarolli.com-tls-559059946" condition "Ready" to 2019-12-07 16:49:03.662042817 +0000 UTC m=+706.524688928
I1207 16:49:03.662253       1 sync.go:373] cert-manager/controller/certificates "level"=0 "msg"="created certificate request" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "request_name"="valerianogarolli.com-tls-559059946"
I1207 16:49:03.662405       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.662805       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "valerianogarolli.com-tls-559059946" condition "Ready" to 2019-12-07 16:49:03.662798482 +0000 UTC m=+706.525444594
I1207 16:49:03.662433       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.663168       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "valerianogarolli.com-tls-559059946" condition "Ready" to 2019-12-07 16:49:03.663162052 +0000 UTC m=+706.525808164
I1207 16:49:03.662504       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.665018       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "valerianogarolli.com-tls-559059946" condition "Ready" to 2019-12-07 16:49:03.664996439 +0000 UTC m=+706.527642636
I1207 16:49:03.662577       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.665281       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "valerianogarolli.com-tls-559059946" condition "Ready" to 2019-12-07 16:49:03.66527298 +0000 UTC m=+706.527919109
E1207 16:49:03.669310       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
I1207 16:49:03.682926       1 controller.go:129] cert-manager/controller/ingress-shim "level"=0 "msg"="syncing item" "key"="default/valeria" 
I1207 16:49:03.683015       1 sync.go:163] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="valeria" "resource_namespace"="default" 
I1207 16:49:03.683044       1 sync.go:176] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate resource is already up to date for ingress" "related_resource_kind"="Certificate" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="valeria" "resource_namespace"="default" 
I1207 16:49:03.683066       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/valeria" 
I1207 16:49:03.684713       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.684760       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.685045       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" 
I1207 16:49:03.685158       1 sync.go:479] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "state"=""
E1207 16:49:03.685318       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
I1207 16:49:03.685447       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.705154       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.705230       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.705236       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls" 
I1207 16:49:03.705494       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.705638       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" 
I1207 16:49:03.705892       1 sync.go:479] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="valerianogarolli.com-tls-559059946" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "state"="Pending"
E1207 16:49:03.706087       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
I1207 16:49:03.706221       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls" 
E1207 16:49:03.709040       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.709288       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.709551       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.710672       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-vault "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.710723       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.710872       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.710901       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.710672       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-venafi "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.710913       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.710945       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.711082       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:03.711083       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.709589       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.710202       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.710997       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.711236       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.711323       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.711661       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.711819       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 
I1207 16:49:08.712344       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="finished processing work item" "key"="default/valerianogarolli.com-tls-559059946" 

For clarity, the following lines are filtered to show only errors

E1207 16:49:03.021435       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
E1207 16:49:03.043375       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
E1207 16:49:03.497521       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
E1207 16:49:03.497548       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
E1207 16:49:03.497620       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
E1207 16:49:03.497653       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
E1207 16:49:03.497669       1 controller.go:189] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"valerianogarolli.com-tls-559059946\" not found"  
E1207 16:49:03.585138       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
E1207 16:49:03.669310       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
E1207 16:49:03.685318       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
E1207 16:49:03.706087       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="valerianogarolli.com-tls" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="valerianogarolli.com-tls" "resource_namespace"="default" "secret_key"="tls.crt" 
E1207 16:49:03.709040       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.710672       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-vault "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.710872       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 
E1207 16:49:03.710672       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-venafi "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"valerianogarolli.com-tls-559059946\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/valerianogarolli.com-tls-559059946" 

I can't be sure if any of those error messages are relevant.

I tried deleting the Certificate and its current invalid TLS secret. That triggers cert-manager to act and create a new Certificate. However, the EXACT same CertificateRequest is used on every attempt valerianogarolli.com-tls-559059946.

# kubectl describe CertificateRequest valerianogarolli.com-tls-559059946
Error from server (NotFound): certificaterequests.certmanager.k8s.io "valerianogarolli.com-tls-559059946" not found

Is that supposed to happen? There are no signs of an Order or Challenge being created here.

[juliohm1978]

I managed to find the cause of my problem to a classic RTFM. So, that's on me 🙃

Previous installations of cert-manager left some of the CRDs behind in the cluster.

Following upgrade instruction, I managed to double check my cluster.

## find all CRDs left behind from previous installations
kubectl get crd | grep certmanager.k8s.io

## delete them
kubectl delete crd CRD_NAME

After that, uninstall and reinstall cert-manager from scratch fixed the issue. The CertificateRequest was successfully created and the challenge worked!

[SCLogo]

can we reopen this? I have totally the same issue
CertificateRequest is not in a final state, waiting until CertificateRequest is complete

[juliohm1978]

My experience with cert-manager had been flaky up until recently. It's only now beginning to show signs that we are getting actually stable releases.

If you end up stuck at some point, I'd advise to remove the entire installation, all CRDs and all resources it created and reinstall from scratch. It's a burden, but that usually fixed most of the problems I had in the past.

[andrenarchy]

Reinstalling doesn't solve my issue. I get the same error that @SCLogo gets with the most recent helm chart (0.15.1). Any updates since March?

[ibottamike]

I had a similar issue, I was using ingress annotation to generate the cert and using ClusterIssuer. The generated certificate had issuerRef.kind: Issuer.
Two solutions:

• add the certificate with issuerRef.kind: ClusterIssuer and have ingress use the same secret to reference the existing cert
• change cert-manager's default issuer for the ingress shim. In helm ingressShim.defaultIssuerKind: ClusterIssuer

[eohtake]

I had a similar issue, I was using ingress annotation to generate the cert and using ClusterIssuer. The generated certificate had issuerRef.kind: Issuer.
Two solutions:

• add the certificate with issuerRef.kind: ClusterIssuer and have ingress use the same secret to reference the existing cert
• change cert-manager's default issuer for the ingress shim. In helm ingressShim.defaultIssuerKind: ClusterIssuer

This has totally solved the problem for me. Many thanks!

Since I'm not using Helm, my certificate yaml now has the issuerRef.kind: ClusterIssuer

For reference, this is what I'm using.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: my-app.example.com
  namespace: my-app
spec:
  issuerRef:
    kind: ClusterIssuer
    name: vault-issuer
  dnsNames:
    - my-app.example.com
    - www.my-app.example.com
  commonName: my-app.example.com
  keySize: 4096
  secretName: my-app-example-com-tls

[mialkin]

Check out my Stack Overflow answer which is using Issuer instead of ClusterIssuer. Maybe it will work for you too.