Cert-manager: Unable to renew certificate for some cert hosts with propagation check failed

[gidesh]

Describe the bug:

Certmanager expectedly renews the certificate of all our tenant's host that we have configured on our K8s cluster. Unfortunately, it is not able to renew some of the hosts for some unknown reason. There is not enough in the logs to suggest why except for the propagation check failed error.

We fix the issue by deleting the secret and the order and let cert-manager create a fresh new certificate secret.

Logs:

cert-manager-6c84f74b-6tr6c cert-manager E0127 15:39:07.236320       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 
cert-manager-6c84f74b-6tr6c cert-manager E0127 15:39:17.294674       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 
cert-manager-6c84f74b-6tr6c cert-manager E0127 15:39:27.346391       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 
cert-manager-6c84f74b-6tr6c cert-manager E0127 15:39:37.362929       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 
cert-manager-6c84f74b-6tr6c cert-manager E0127 15:39:47.388588       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 
cert-manager-6c84f74b-6tr6c cert-manager E0127 15:39:57.449904       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 
cert-manager-6c84f74b-6tr6c cert-manager E0127 15:40:07.506074       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 
cert-manager-6c84f74b-6tr6c cert-manager E0127 15:40:17.540044       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"order-fulfilment-mgmt.prod.jl-digital.net\" not yet propagated" "dnsName"="order-fulfilment-mgmt.prod.jl-digital.net" "resource_kind"="Challenge" "resource_name"="wildcard-tls-78dhb-3091505254-2298386629" "resource_namespace"="order-fulfilment-mgmt" "resource_version"="v1" "type"="DNS-01" 

Certificate

❯ k get cert wildcard-tls -o yaml                                                                                                                                                                                                          ─╯
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"wildcard-tls","namespace":"order-fulfilment-mgmt"},"spec":{"commonName":"*.order-fulfilment-mgmt.prod.jl-digital.net","dnsNames":["*.order-fulfilment-mgmt.prod.jl-digital.net"],"issuerRef":{"kind":"ClusterIssuer","name":"letsencrypt-jl"},"secretName":"wildcard-tls"}}
  creationTimestamp: "2020-09-24T14:38:33Z"
  generation: 1
  name: wildcard-tls
  namespace: order-fulfilment-mgmt
  resourceVersion: "681884543"
  selfLink: /apis/cert-manager.io/v1/namespaces/order-fulfilment-mgmt/certificates/wildcard-tls
  uid: 9d4fdc8c-5b1f-4405-8ea4-0aa0d073aeac
spec:
  commonName: '*.order-fulfilment-mgmt.prod.jl-digital.net'
  dnsNames:
  - '*.order-fulfilment-mgmt.prod.jl-digital.net'
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-jl
  secretName: wildcard-tls
status:
  conditions:
  - lastTransitionTime: "2020-09-24T14:42:03Z"
    message: Certificate is up to date and has not expired
    reason: Ready
    status: "True"
    type: Ready
  - lastTransitionTime: "2021-01-22T12:44:03Z"
    message: Renewing certificate as renewal was scheduled at 2021-01-22 12:44:03
      +0000 UTC
    reason: Renewing
    status: "True"
    type: Issuing
  nextPrivateKeySecretName: wildcard-tls-dpcnk
  notAfter: "2021-02-21T12:44:03Z"
  notBefore: "2020-11-23T12:44:03Z"
  renewalTime: "2021-01-22T12:44:03Z"

Expected behaviour:

Certmanager should renew all the certificates when required.

Steps to reproduce the bug:

None

Anything else we need to know?:

Please let me know if there is anything else you need?

Environment details::

• Kubernetes version: 1.17
• Cloud-provider/provisioner: GCP-GKE, Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.14-gke.400"}
• cert-manager version: quay.io/jetstack/cert-manager-controller:v1.1.0
• Install method: e.g. static manifests

/kind bug

[udhos]

I am hitting this exact same issue on cert-manager v1.0.3.
I can work-around the automatic renewal by deleting the secret.

[udhos]

@gidesh I am facing a very similar issue and wondering what setting might be triggering such an effect in my environment.

Issue is, once the automatic renewal fails, cert-manager logs continuously report "propagation check failed".

I can work-around the renewal failure by manually deleting the secret. However I am looking for the root cause preventing automatic renewal. I am puzzled on why cert-manager can solve the DNS01 challenge for a new certificate but not for renewal.

One tricky piece of information is, my certificate has two DNS hostnames, and the DNS challenge works for one hostname but gets stuck for the other one. But then again, only for automatic renewal. At certificate creation, everything works.

My case has these specifics:

1) Kubernetes: Amazon EKS 1.17
2) cert-manager v1.0.3 installed from Helm
3) Challenge: DNS01 with Route53
4) Certificate for two DNS hostnames
5) Using a less priviledged dns zone with "cnameStrategy: Follow"

Can you please confirm whether you have a similar setup?

[gidesh]

Hi @udhos , The cert order is still stuck in a pending state for the cert that I had raised the issue for.

❯ k get order     
NAME                            STATE     AGE
wildcard-tls-78dhb-3091505254   pending   7d2h

My setup is similar but in GCP

1. Kubernetes: GKE v1.17.14-gke.1600
2. cert-manager: v1.1.0 using static manifests
3. Challenge: DNS01 with GCP Cloud DNS
4. Just one host name :
   spec: commonName: '*.order-fulfilment-mgmt.prod.jl-digital.net' dnsNames: <ul> <li>'*.order-fulfilment-mgmt.prod.jl-digital.net'<br />


5. Not using such a strategy of less privileged zone.

[udhos]

@gidesh Thanks for sharing.

I have environments still stuck in pending as well.
I know I can work-around the failed renewal by deleting the secret, but I am searching for the root cause preventing automatic renewal.
It currently looks like this:

$ k get cr
NAME                    READY   AGE
secret-xxxx-com-gpc26   False   20d
secret-xxxx-com-ld7lq   True    80d

$ k get order
NAME                               STATE     AGE
secret-xxxx-com-gpc26-1299057733   pending   20d
secret-xxxx-com-ld7lq-1299057733   valid     80d

$ k get challenge
NAME                                          STATE     DOMAIN                 AGE
secret-xxxx-com-gpc26-1299057733-2619509026   pending   api-hom.xxxx.com       20d
secret-xxxx-com-gpc26-1299057733-340693238    valid     sandbox-hom.xxxx.com   20d

[udhos]

Just tested this recipe for manually renewing a certificate previously stuck in failed automatic renewal.

# step 1: delete certificate stuck in automatic renewal
kubectl delete cert <certificate_name>

# step 2: trigger manual renewal
kubectl cert-manager renew <certificate_name>

Didn't find yet why automatic renewal failed the DNS challenge.