Cert-manager: Certificate sometimes fails to issue properly

Created on 20 Jan 2021 · 8Comments · Source: jetstack/cert-manager

Describe the bug:

We are using cert-manager + letsencrypt staging in our test pipeline. The pipeline only runs a few times a week, so it's well under the limits of letsencrypt staging. Our certificate sometimes fails to issue properly. We know that our config is OK since it usually works fine. However in the last week we had two failures caused by cert-manager failing to issue the certificate. We did not encounter this problem before this week.

Our certificate has this spec:

  spec:
    dnsNames:
    - '*.test-khhyml7mh2jkg732.loci.ubi.com'
    issuerRef:
      kind: ClusterIssuer
      name: letsencrypt-test
    secretName: wildcard-cert-tls

Our cluster issuer has this spec:

  spec:
    acme:
      email: <redacted>@ubisoft.com
      preferredChain: ""
      privateKeySecretRef:
        name: letsencrypt-test-issuer-account-key
      server: https://acme-staging-v02.api.letsencrypt.org/directory
      solvers:
      - dns01:
          cloudDNS:
            project: bob-dbaas-dev
            serviceAccountSecretRef:
              key: credentials.json
              name: cert-manager-google-secret

The cert manager logs are below:

I0120 16:14:44.476975       1 conditions.go:173] Setting lastTransitionTime for Certificate "wildcard-cert" condition "Issuing" to 2021-01-20 16:14:44.476965945 +0000 UTC m=+13.944761228
I0120 16:14:44.477078       1 conditions.go:173] Setting lastTransitionTime for Certificate "wildcard-cert" condition "Ready" to 2021-01-20 16:14:44.477064448 +0000 UTC m=+13.944859754
E0120 16:14:44.575829       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"wildcard-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="mongo-operator/wildcard-cert" 
I0120 16:14:44.575909       1 conditions.go:173] Setting lastTransitionTime for Certificate "wildcard-cert" condition "Issuing" to 2021-01-20 16:14:44.575903168 +0000 UTC m=+14.043698423
I0120 16:14:44.580373       1 setup.go:90] cert-manager/controller/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1" 
I0120 16:14:44.813391       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "wildcard-cert-mjj2b" condition "Ready" to 2021-01-20 16:14:44.813382149 +0000 UTC m=+14.281177396
I0120 16:14:44.922381       1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1" 
I0120 16:14:45.391446       1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1" 
I0120 16:14:45.391473       1 conditions.go:92] Setting lastTransitionTime for Issuer "letsencrypt-test" condition "Ready" to 2021-01-20 16:14:45.391467192 +0000 UTC m=+14.859262437
I0120 16:14:45.526098       1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1" 
E0120 16:14:45.620396       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="mongo-operator/wildcard-cert-mjj2b-2407777396" 
I0120 16:14:46.048213       1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1" 
I0120 16:14:49.922449       1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1" 
I0120 16:14:50.388598       1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1" 
E0120 16:14:53.757052       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:14:53.791518       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:15:03.762095       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:15:13.767389       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:15:23.789744       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:15:33.794373       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:15:43.799491       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:15:53.835277       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
E0120 16:17:34.960129       1 sync.go:354] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for test-khhyml7mh2jkg732.loci.ubi.com: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.test-khhyml7mh2jkg732.loci.ubi.com - check that a DNS record exists for this domain" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01" 
I0120 16:17:35.135532       1 conditions.go:162] Found status change for Certificate "wildcard-cert" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-20 16:17:35.135521875 +0000 UTC m=+184.603317157
E0120 16:17:35.174806       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"wildcard-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="mongo-operator/wildcard-cert" 
I0120 16:17:35.176107       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="mongo-operator/wildcard-cert" "retry_after"="2021-01-20T17:17:35Z"
I0120 16:17:35.233041       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="mongo-operator/wildcard-cert" "retry_after"="2021-01-20T17:17:35Z"
E0120 16:17:35.247545       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"wildcard-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="mongo-operator/wildcard-cert"

Also, status on certificate

- lastTransitionTime: "2021-01-20T17:17:35Z"
      message: 'The certificate request has failed to complete and will be retried:
        Failed to wait for order resource "wildcard-cert-mjj2b-2407777396" to become
        ready: order is in "invalid" state: '
      reason: Failed
      status: "False"
      type: Issuing

Expected behaviour:

Certificate should issue properly

Steps to reproduce the bug:

Setup a cluster issuer with letsencrypt staging, ask for a certificate.

Environment details:

Kubernetes version: 1.16
Cloud-provider/provisioner: GKE
cert-manager version: 1.1.0
Install method: helm

/kind bug

areacme kinbug triagneeds-information

Source

sbstp

👀1

Most helpful comment

I had some similar issues on multiple clusters with version 1.0.4, seems to appear randomly 😢 .
Symptoms: some challenges are stuck in an invalid state that will never recover (i.e: 70% will be good and valid, the rest are stuck in invalid, got 12 hostnames including wildcards to validate on this particular certificate).

It takes some tinkering to unlock the situation (delete invalid challenges or delete the whole certificate request + wait for the hour retry condition to kick in).

Here are the logs (the one I have between a pod restart and when I actually connected to the cluster after some custom alerts raised my attention).

I0119 17:34:45.983004       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name
"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0119 17:34:45.987641       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="ACME client for issuer not initialised/available" "key"="fleet-system/gateway-routing-dd963271-t6ss2-339290
5940" 
I0119 17:34:45.988254       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T17:51:59Z"
I0119 17:34:46.987142       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system
" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0119 17:34:50.953838       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name
"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0119 17:34:50.988402       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="ACME client for issuer not initialised/available" "key"="fleet-system/gateway-routing-dd963271-t6ss2-339290
5940" 
I0119 17:34:51.520383       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system
" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0119 17:51:59.001471       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 17:51:59.001455514 +0000 UTC m=+1033.276325779
I0119 17:51:59.190463       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 17:51:59.190452521 +0000 UTC m=+1033.465322769
I0119 17:51:59.221353       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T18:51:59Z"
E0119 17:51:59.251015       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": t
he object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 17:51:59.322411       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T18:51:59Z"
I0119 18:51:59.001656       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 18:51:59.001640479 +0000 UTC m=+4633.276510746
I0119 18:51:59.148195       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 18:51:59.14818407 +0000 UTC m=+4633.423054320
I0119 18:51:59.192546       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T19:51:59Z"
E0119 18:51:59.218734       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": t
he object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 18:51:59.272712       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T19:51:59Z"
I0119 19:51:59.001682       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 19:51:59.00166827 +0000 UTC m=+8233.276538589
I0119 19:51:59.138926       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 19:51:59.138915438 +0000 UTC m=+8233.413785668
E0119 19:51:59.182501       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": 
the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 19:51:59.198833       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T20:51:59Z"
E0119 19:51:59.225983       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 19:51:59.278316       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T20:51:59Z"
I0119 20:51:59.001479       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 20:51:59.00146562 +0000 UTC m=+11833.276335885
I0119 20:51:59.136032       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 20:51:59.136022062 +0000 UTC m=+11833.410892301
E0119 20:51:59.188299       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 20:51:59.191348       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T21:51:59Z"
I0119 20:51:59.258718       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T21:51:59Z"
I0119 21:51:59.002095       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 21:51:59.002082432 +0000 UTC m=+15433.276952675
I0119 21:51:59.214110       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 21:51:59.214100805 +0000 UTC m=+15433.488971044
E0119 21:51:59.284494       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
E0119 21:51:59.287904       1 controller.go:158] cert-manager/controller/CertificateIssuing "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 21:51:59.288498       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 21:51:59.288486701 +0000 UTC m=+15433.563356974
I0119 21:51:59.347098       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T22:51:59Z"
I0119 21:51:59.409006       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T22:51:59Z"
E0119 21:51:59.438028       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 22:51:59.001937       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 22:51:59.00192406 +0000 UTC m=+19033.276794310
I0119 22:51:59.118746       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 22:51:59.118736797 +0000 UTC m=+19033.393607035
I0119 22:51:59.324823       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T23:51:59Z"
E0119 22:51:59.390958       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271"

FYI: Last time I had the problem I deleted the whole certificatrequest (which cleaned-up order and challenges), when it retried, it only issued some of the challenges (probably those that were invalid before???) before issuing the certificate successfully.

Seeing a lot of https://github.com/jetstack/cert-manager/issues/3516 problematic logs so just mentioning it here.

gmauleon on 25 Jan 2021

👍2

All 8 comments

It takes some tinkering to unlock the situation (delete invalid challenges or delete the whole certificate request + wait for the hour retry condition to kick in).

Here are the logs (the one I have between a pod restart and when I actually connected to the cluster after some custom alerts raised my attention).

I0119 17:34:45.983004       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name
"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0119 17:34:45.987641       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="ACME client for issuer not initialised/available" "key"="fleet-system/gateway-routing-dd963271-t6ss2-339290
5940" 
I0119 17:34:45.988254       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T17:51:59Z"
I0119 17:34:46.987142       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system
" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0119 17:34:50.953838       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name
"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0119 17:34:50.988402       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="ACME client for issuer not initialised/available" "key"="fleet-system/gateway-routing-dd963271-t6ss2-339290
5940" 
I0119 17:34:51.520383       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system
" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0119 17:51:59.001471       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 17:51:59.001455514 +0000 UTC m=+1033.276325779
I0119 17:51:59.190463       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 17:51:59.190452521 +0000 UTC m=+1033.465322769
I0119 17:51:59.221353       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T18:51:59Z"
E0119 17:51:59.251015       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": t
he object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 17:51:59.322411       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T18:51:59Z"
I0119 18:51:59.001656       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 18:51:59.001640479 +0000 UTC m=+4633.276510746
I0119 18:51:59.148195       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 18:51:59.14818407 +0000 UTC m=+4633.423054320
I0119 18:51:59.192546       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T19:51:59Z"
E0119 18:51:59.218734       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": t
he object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 18:51:59.272712       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T19:51:59Z"
I0119 19:51:59.001682       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 19:51:59.00166827 +0000 UTC m=+8233.276538589
I0119 19:51:59.138926       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 19:51:59.138915438 +0000 UTC m=+8233.413785668
E0119 19:51:59.182501       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": 
the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 19:51:59.198833       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021
-01-19T20:51:59Z"
E0119 19:51:59.225983       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 19:51:59.278316       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T20:51:59Z"
I0119 20:51:59.001479       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 20:51:59.00146562 +0000 UTC m=+11833.276335885
I0119 20:51:59.136032       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 20:51:59.136022062 +0000 UTC m=+11833.410892301
E0119 20:51:59.188299       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 20:51:59.191348       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T21:51:59Z"
I0119 20:51:59.258718       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T21:51:59Z"
I0119 21:51:59.002095       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 21:51:59.002082432 +0000 UTC m=+15433.276952675
I0119 21:51:59.214110       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 21:51:59.214100805 +0000 UTC m=+15433.488971044
E0119 21:51:59.284494       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
E0119 21:51:59.287904       1 controller.go:158] cert-manager/controller/CertificateIssuing "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 21:51:59.288498       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 21:51:59.288486701 +0000 UTC m=+15433.563356974
I0119 21:51:59.347098       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T22:51:59Z"
I0119 21:51:59.409006       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T22:51:59Z"
E0119 21:51:59.438028       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0119 22:51:59.001937       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-01-19 22:51:59.00192406 +0000 UTC m=+19033.276794310
I0119 22:51:59.118746       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-19 22:51:59.118736797 +0000 UTC m=+19033.393607035
I0119 22:51:59.324823       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-01-19T23:51:59Z"
E0119 22:51:59.390958       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271"

Seeing a lot of https://github.com/jetstack/cert-manager/issues/3516 problematic logs so just mentioning it here.

gmauleon on 25 Jan 2021

👍2

Would it be possible to get a kubectl describe on the order and challenge resources? That would help us to look what is the latest state on those resources.

/triage needs-information
/area acme

meyskens on 28 Jan 2021

@meyskens: The label(s) area/ cannot be applied, because the repository doesn't have them

In response to this:

Would it be possible to get a kubectl describe on the order and challenge resources? That would help us to look what is the latest state on those resources.

/triage needs-information
/area acme

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jetstack-bot on 28 Jan 2021

Will do when I witness the problem next time since I deleted problematic orders and challenges to make it pass clusters that were failing.

gmauleon on 28 Jan 2021

I haven't run into the problem this week over maybe 7 or 8 test runs. Not sure if it's luck of if something was wrong with the LE server last week.

sbstp on 28 Jan 2021

Hi!

I have one environment with challenge forever stuck on pending (and logging DNS not propagated).
Though it is not using letsencrypt staging, it is on production.

$ k get cr
NAME                    READY   AGE
secret-xxxx-com-dj8fq   True    81d
secret-xxxx-com-jdzv6   False   21d

$ k get challenge
NAME                                       STATE     DOMAIN             AGE
secret-xxxx-com-jdzv6-5481715-1791462026   valid     sandbox.xxxx.com   21d
secret-xxxx-com-jdzv6-5481715-472512675    pending   api.xxxx.com       21d

I am able to work-around that automatic renewal failure by deleting the secret.
Will apply such a work-around on Feb 3rd.

Is there any information I can collect to help in understanding this issue?

udhos on 1 Feb 2021

Ok I just got a somehow similar problem. Start with the same symptoms, some challenges are invalid, this one is with AWS Route53 (was GCP before). Sorry for the wall of text 😅

So I witness some challenges were invalid and dumped some describes below while the challenge were there:
One invalid challenge

Name:         gateway-routing-dd963271-kf8vs-2797388668-1082162377
Namespace:    fleet-system
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2021-02-03T15:04:45Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  gateway-routing-dd963271-kf8vs-2797388668
    UID:                   a812f3eb-fcf7-48b1-a327-614613828aa8
  Resource Version:        8238
  Self Link:               /apis/acme.cert-manager.io/v1/namespaces/fleet-system/challenges/gateway-routing-dd963271-kf8vs-2797388668-1082162377
  UID:                     3992532f-82cc-45de-a1f4-c2694ee45d0e
Spec:
  Authorization URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618146
  Dns Name:           XXX
  Issuer Ref:
    Kind:  Issuer
    Name:  gateway-routing-dd963271
  Key:     XXX
  Solver:
    dns01:
      route53:
        Access Key ID:  XXX
        Region:         us-east-1
        Secret Access Key Secret Ref:
          Key:   content
          Name:  fleet-gateway-gateway-dns01-route53-00
  Token:         XXX
  Type:          DNS-01
  URL:           https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618146/ZAl_zg
  Wildcard:      false
Status:
  Presented:   false
  Processing:  false
  Reason:      Error accepting authorization: acme: authorization error for XXX: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: SERVFAIL looking up TXT for _acme-challenge.XXX - the domain's nameservers may be malfunctioning
  State:       invalid
Events:
  Type     Reason     Age   From          Message
  ----     ------     ----  ----          -------
  Normal   Started    53m   cert-manager  Challenge scheduled for processing
  Normal   Presented  52m   cert-manager  Presented challenge using DNS-01 challenge mechanism
  Warning  Failed     51m   cert-manager  Accepting challenge authorization failed: acme: authorization error for XXX: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: SERVFAIL looking up TXT for _acme-challenge.XXX - the domain's nameservers may be malfunctioning

Order

Name:         gateway-routing-dd963271-kf8vs-2797388668
Namespace:    fleet-system
Labels:       app.kubernetes.io/component=routing
              app.kubernetes.io/name=fleet-gateway-gateway
              fleet.ubisoft.com/gateway=dd963271
Annotations:  cert-manager.io/certificate-name: gateway-routing-dd963271
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: gateway-routing-dd963271-7r7l9
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2021-02-03T15:04:38Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  gateway-routing-dd963271-kf8vs
    UID:                   770d780e-878b-4a84-afb0-7bad5b1f3d8f
  Resource Version:        7863
  Self Link:               /apis/acme.cert-manager.io/v1/namespaces/fleet-system/orders/gateway-routing-dd963271-kf8vs-2797388668
  UID:                     a812f3eb-fcf7-48b1-a327-614613828aa8
Spec:
  Dns Names:
    *.live.admin.XXX
    *.live.public.XXX
    *.live.s2s.XXX
    *.preflight.admin.XXX
    *.preflight.public.XXX
    *.preflight.s2s.XXX
    live.admin.XXX
    live.public.XXX
    live.s2s.XXX
    preflight.admin.XXX
    preflight.public.XXX
    preflight.s2s.XXX
  Issuer Ref:
    Kind:   Issuer
    Name:   gateway-routing-dd963271
  Request:  XXX
Status:
  Authorizations:
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618133/xRzmCw
    Identifier:     live.admin.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618133
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618134/JHD6Qg
    Identifier:     live.public.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618134
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618135/88TVFA
    Identifier:     live.s2s.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618135
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618137/O2BYuw
    Identifier:     preflight.admin.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618137
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618138/jJIQgQ
    Identifier:     preflight.public.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618138
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618139/z6ngFw
    Identifier:     preflight.s2s.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618139
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618141/lFK5RA
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618141/Z4szGg
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618141/avE5kQ
    Identifier:     live.admin.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618141
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618142/sflvsA
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618142/I2Gkzg
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618142/fI9kqw
    Identifier:     live.public.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618142
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618143/3tKHsQ
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618143/2P3zcw
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618143/uLbafQ
    Identifier:     live.s2s.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618143
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618144/dwFDfA
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618144/P46ZSA
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618144/ZCFpkg
    Identifier:     preflight.admin.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618144
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618146/yOK4-Q
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618146/ZAl_zg
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618146/d9thCQ
    Identifier:     preflight.public.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618146
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618147/woHM5g
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618147/68TEOw
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618147/WBMN2A
    Identifier:     preflight.s2s.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618147
    Wildcard:       false
  Failure Time:     2021-02-03T15:08:37Z
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/17901163/233452968
  State:            invalid
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/17901163/233452968
Events:
  Type    Reason   Age                From          Message
  ----    ------   ----               ----          -------
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-2135797261" for domain "live.admin.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-790807248" for domain "live.public.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-2356799498" for domain "live.s2s.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-2963433089" for domain "preflight.admin.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-2846237678" for domain "preflight.public.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-2151601995" for domain "preflight.s2s.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-3219159356" for domain "live.admin.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-2589084657" for domain "live.public.XXX"
  Normal  Created  60m                cert-manager  Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-2051963654" for domain "live.s2s.XXX"
  Normal  Created  60m (x3 over 60m)  cert-manager  (combined from similar events): Created Challenge resource "gateway-routing-dd963271-kf8vs-2797388668-1537543059" for domain "preflight.s2s.XXX"

So looks like some DNS problem here. Shortly after for me (but probably one hour after the initial try) all challenges were deleted automatically and the order switched to a valid state with a certificate:

Name:         gateway-routing-dd963271-lc4s8-2797388668
Namespace:    fleet-system
Labels:       app.kubernetes.io/component=routing
              app.kubernetes.io/name=fleet-gateway-gateway
              fleet.ubisoft.com/gateway=dd963271
Annotations:  cert-manager.io/certificate-name: gateway-routing-dd963271
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: gateway-routing-dd963271-z6xzs
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2021-02-03T16:08:37Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  gateway-routing-dd963271-lc4s8
    UID:                   82391dca-5fab-479d-8113-44da4dca6e72
  Resource Version:        28234
  Self Link:               /apis/acme.cert-manager.io/v1/namespaces/fleet-system/orders/gateway-routing-dd963271-lc4s8-2797388668
  UID:                     9d8e787d-63fc-4a83-9b51-7fa1b3b42dca
Spec:
  Dns Names:
    *.live.admin.XXX
    *.live.public.XXX
    *.live.s2s.XXX
    *.preflight.admin.XXX
    *.preflight.public.XXX
    *.preflight.s2s.XXX
    live.admin.XXX
    live.public.XXX
    live.s2s.XXX
    preflight.admin.XXX
    preflight.public.XXX
    preflight.s2s.XXX
  Issuer Ref:
    Kind:   Issuer
    Name:   gateway-routing-dd963271
  Request:  XXX
Status:
  Authorizations:
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618133/xRzmCw
    Identifier:     live.admin.XXX
    Initial State:  valid
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618133
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618134/JHD6Qg
    Identifier:     live.public.XXX
    Initial State:  valid
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618134
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618135/88TVFA
    Identifier:     live.s2s.XXX
    Initial State:  valid
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618135
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618137/O2BYuw
    Identifier:     preflight.admin.XXX
    Initial State:  valid
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618137
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618138/jJIQgQ
    Identifier:     preflight.public.XXX
    Initial State:  valid
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618138
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618141/Z4szGg
    Identifier:     live.admin.XXX
    Initial State:  valid
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618141
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202618147/68TEOw
    Identifier:     preflight.s2s.XXX
    Initial State:  valid
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202618147
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649888/MDifQw
    Identifier:     preflight.s2s.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202649888
    Wildcard:       true
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649889/KCQFxg
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649889/PL7Uuw
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649889/wVCYcQ
    Identifier:     live.public.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202649889
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649890/sU3Oeg
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649890/YKUxPQ
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649890/INGACw
    Identifier:     live.s2s.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202649890
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649892/NtWxdg
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649892/eq57pQ
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649892/ZvPMUg
    Identifier:     preflight.admin.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202649892
    Wildcard:       false
    Challenges:
      Token:        XXX
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649893/Dh3SGw
      Token:        XXX
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649893/2oPTgQ
      Token:        XXX
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/202649893/xhNbFQ
    Identifier:     preflight.public.XXX
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/202649893
    Wildcard:       false
  Certificate:      XXX
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/17901163/233485533
  State:            valid
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/17901163/233485533
Events:
  Type    Reason    Age   From          Message
  ----    ------    ----  ----          -------
  Normal  Created   24m   cert-manager  Created Challenge resource "gateway-routing-dd963271-lc4s8-2797388668-1089111828" for domain "preflight.s2s.XXX"
  Normal  Created   24m   cert-manager  Created Challenge resource "gateway-routing-dd963271-lc4s8-2797388668-749294436" for domain "live.public.XXX"
  Normal  Created   24m   cert-manager  Created Challenge resource "gateway-routing-dd963271-lc4s8-2797388668-1049270256" for domain "live.s2s.XXX"
  Normal  Created   24m   cert-manager  Created Challenge resource "gateway-routing-dd963271-lc4s8-2797388668-771636978" for domain "preflight.admin.XXX"
  Normal  Created   24m   cert-manager  Created Challenge resource "gateway-routing-dd963271-lc4s8-2797388668-2479260129" for domain "preflight.public.XXX"
  Normal  Complete  22m   cert-manager  Order completed successfully

But the certificate itself was not valid....

Name:         gateway-routing-dd963271
Namespace:    fleet-system
Labels:       app.kubernetes.io/component=routing
              app.kubernetes.io/name=fleet-gateway-gateway
              fleet.ubisoft.com/gateway=dd963271
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2021-02-03T15:04:37Z
  Generation:          1
  Owner References:
    API Version:           networking.fleet.ubisoft.com/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Gateway
    Name:                  fleet-gateway-gateway
    UID:                   9febd91a-a291-4ccc-be96-8dc5a7653094
  Resource Version:        27460
  Self Link:               /apis/cert-manager.io/v1/namespaces/fleet-system/certificates/gateway-routing-dd963271
  UID:                     02cad1b4-865b-4181-b356-abac7ce2281e
Spec:
  Dns Names:
    *.live.admin.XXX
    *.live.public.XXX
    *.live.s2s.XXX
    *.preflight.admin.XXX
    *.preflight.public.XXX
    *.preflight.s2s.XXX
    live.admin.XXX
    live.public.XXX
    live.s2s.XXX
    preflight.admin.XXX
    preflight.public.XXX
    preflight.s2s.XXX
  Issuer Ref:
    Kind:       Issuer
    Name:       gateway-routing-dd963271
  Secret Name:  gateway-routing-cm-cert-dd963271
Status:
  Conditions:
    Last Transition Time:  2021-02-03T16:08:37Z
    Message:               The certificate request has failed to complete and will be retried: Failed to wait for order resource "gateway-routing-dd963271-kf8vs-2797388668" to become ready: order is in "invalid" state: 
    Reason:                Failed
    Status:                False
    Type:                  Issuing
    Last Transition Time:  2021-02-03T15:04:37Z
    Message:               Issuing certificate as Secret does not exist
    Reason:                DoesNotExist
    Status:                False
    Type:                  Ready
  Last Failure Time:       2021-02-03T16:08:37Z
Events:
  Type     Reason     Age                From          Message
  ----     ------     ----               ----          -------
  Normal   Issuing    25m (x2 over 89m)  cert-manager  Issuing certificate as Secret does not exist
  Warning  Failed     25m (x2 over 85m)  cert-manager  The certificate request has failed to complete and will be retried: Failed to wait for order resource "gateway-routing-dd963271-kf8vs-2797388668" to become ready: order is in "invalid" state:
  Normal   Generated  25m                cert-manager  Stored new private key in temporary Secret resource "gateway-routing-dd963271-z6xzs"
  Normal   Requested  25m                cert-manager  Created new CertificateRequest resource "gateway-routing-dd963271-lc4s8"

Here are the controller logs

I0203 14:59:25.135832       1 start.go:74] cert-manager "msg"="starting controller"  "git-commit"="4d870e49b43960fad974487a262395e65da1373e" "version"="v1.0.4"
W0203 14:59:25.135893       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0203 14:59:25.136886       1 controller.go:161] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["172.20.0.10:53"] 
I0203 14:59:25.137726       1 controller.go:124] cert-manager/controller "msg"="starting leader election"  
I0203 14:59:25.137916       1 metrics.go:162] cert-manager/controller/build-context/metrics "msg"="listening for connections on" "address"={"IP":"::","Port":9402,"Zone":""} 
I0203 14:59:25.138396       1 leaderelection.go:243] attempting to acquire leader lease  kube-system/cert-manager-controller...
I0203 14:59:25.159329       1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-controller
I0203 14:59:25.160480       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" 
I0203 14:59:25.160536       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="CertificateIssuing" 
I0203 14:59:25.160566       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="CertificateTrigger" 
I0203 14:59:25.160601       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I0203 14:59:25.160642       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="CertificateRequestManager" 
I0203 14:59:25.160677       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="CertificateKeyManager" 
I0203 14:59:25.160783       1 reflector.go:207] Starting reflector *v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.260621       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I0203 14:59:25.260848       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="issuers" 
I0203 14:59:25.260851       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="CertificateReadiness" 
I0203 14:59:25.260934       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I0203 14:59:25.261094       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" 
I0203 14:59:25.261106       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="CertificateMetrics" 
I0203 14:59:25.261137       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" 
I0203 14:59:25.261188       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I0203 14:59:25.261245       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I0203 14:59:25.261923       1 reflector.go:207] Starting reflector *v1.Pod (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.262317       1 controller.go:98] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I0203 14:59:25.262471       1 reflector.go:207] Starting reflector *v1.Certificate (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.262765       1 reflector.go:207] Starting reflector *v1.CertificateRequest (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.263095       1 reflector.go:207] Starting reflector *v1.Issuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.263331       1 reflector.go:207] Starting reflector *v1.ClusterIssuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.263496       1 reflector.go:207] Starting reflector *v1.Challenge (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.263854       1 reflector.go:207] Starting reflector *v1.Order (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.263979       1 reflector.go:207] Starting reflector *v1.Service (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.264108       1 reflector.go:207] Starting reflector *v1beta1.Ingress (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 14:59:25.264166       1 reflector.go:207] Starting reflector *v1.Secret (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0203 15:01:46.371427       1 conditions.go:173] Setting lastTransitionTime for Certificate "aws-load-balancer-serving-cert" condition "Issuing" to 2021-02-03 15:01:46.37141557 +0000 UTC m=+141.254131842
I0203 15:01:46.372127       1 conditions.go:173] Setting lastTransitionTime for Certificate "aws-load-balancer-serving-cert" condition "Ready" to 2021-02-03 15:01:46.372121522 +0000 UTC m=+141.254837780
I0203 15:01:46.580037       1 conditions.go:92] Setting lastTransitionTime for Issuer "aws-load-balancer-selfsigned-issuer" condition "Ready" to 2021-02-03 15:01:46.580030443 +0000 UTC m=+141.462746687
E0203 15:01:47.061287       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"aws-load-balancer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="kube-system/aws-load-balancer-serving-cert" 
I0203 15:01:47.061337       1 conditions.go:173] Setting lastTransitionTime for Certificate "aws-load-balancer-serving-cert" condition "Issuing" to 2021-02-03 15:01:47.061332295 +0000 UTC m=+141.944048544
E0203 15:01:47.284691       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"aws-load-balancer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="kube-system/aws-load-balancer-serving-cert" 
I0203 15:01:47.301677       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "aws-load-balancer-serving-cert-v8rdx" condition "Ready" to 2021-02-03 15:01:47.301670815 +0000 UTC m=+142.184387051
I0203 15:01:47.329670       1 conditions.go:162] Found status change for Certificate "aws-load-balancer-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-03 15:01:47.329664523 +0000 UTC m=+142.212380750
E0203 15:01:47.451529       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"aws-load-balancer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="kube-system/aws-load-balancer-serving-cert" 
I0203 15:01:47.451845       1 conditions.go:162] Found status change for Certificate "aws-load-balancer-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-03 15:01:47.451839983 +0000 UTC m=+142.334556227
E0203 15:01:47.851096       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"aws-load-balancer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="kube-system/aws-load-balancer-serving-cert" 
I0203 15:01:47.851414       1 conditions.go:162] Found status change for Certificate "aws-load-balancer-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-03 15:01:47.851408034 +0000 UTC m=+142.734124257
I0203 15:04:37.342399       1 conditions.go:173] Setting lastTransitionTime for Certificate "gateway-routing-dd963271" condition "Issuing" to 2021-02-03 15:04:37.342388133 +0000 UTC m=+312.225104424
I0203 15:04:37.342454       1 conditions.go:173] Setting lastTransitionTime for Certificate "gateway-routing-dd963271" condition "Ready" to 2021-02-03 15:04:37.34244918 +0000 UTC m=+312.225165435
I0203 15:04:37.423379       1 setup.go:90] cert-manager/controller/issuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 15:04:37.443133       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0203 15:04:37.443265       1 conditions.go:173] Setting lastTransitionTime for Certificate "gateway-routing-dd963271" condition "Ready" to 2021-02-03 15:04:37.443259683 +0000 UTC m=+312.325975928
I0203 15:04:37.770140       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 15:04:38.384208       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 15:04:38.384239       1 conditions.go:92] Setting lastTransitionTime for Issuer "gateway-routing-dd963271" condition "Ready" to 2021-02-03 15:04:38.384233594 +0000 UTC m=+313.266949833
I0203 15:04:38.394627       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 15:04:38.645413       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
E0203 15:04:38.689239       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="ACME client for issuer not initialised/available" "key"="fleet-system/gateway-routing-dd963271-kf8vs-2797388668" 
I0203 15:04:38.690121       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "gateway-routing-dd963271-kf8vs" condition "Ready" to 2021-02-03 15:04:38.690113042 +0000 UTC m=+313.572829285
I0203 15:04:38.974149       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 15:04:42.769952       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 15:04:43.235680       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 15:08:37.001014       1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for preflight.s2s.XXX: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: SERVFAIL looking up TXT for _acme-challenge.preflight.s2s.XXX - the domain's nameservers may be malfunctioning" "dnsName"="preflight.s2s.XXX" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2151601995" "resource_namespace"="fleet-system" "resource_version"="v1" "type"="DNS-01" 
I0203 15:08:37.124725       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-02-03 15:08:37.124718716 +0000 UTC m=+552.007434954
I0203 15:08:37.154921       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T16:08:37Z"
E0203 15:08:37.229839       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
I0203 15:08:37.336457       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T16:08:37Z"
E0203 15:09:16.257780       1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for preflight.admin.XXX: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: SERVFAIL looking up TXT for _acme-challenge.preflight.admin.XXX - the domain's nameservers may be malfunctioning" "dnsName"="preflight.admin.XXX" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1095463117" "resource_namespace"="fleet-system" "resource_version"="v1" "type"="DNS-01" 
E0203 15:09:16.314074       1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for preflight.public.XXX: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: SERVFAIL looking up TXT for _acme-challenge.preflight.public.XXX - the domain's nameservers may be malfunctioning" "dnsName"="preflight.public.XXX" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1082162377" "resource_namespace"="fleet-system" "resource_version"="v1" "type"="DNS-01" 
E0203 15:09:16.816317       1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for live.s2s.XXX: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: SERVFAIL looking up TXT for _acme-challenge.live.s2s.XXX - the domain's nameservers may be malfunctioning" "dnsName"="live.s2s.XXX" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2051963654" "resource_namespace"="fleet-system" "resource_version"="v1" "type"="DNS-01" 
E0203 15:09:54.358612       1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for live.public.XXX: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: SERVFAIL looking up CAA for ubi.com - the domain's nameservers may be malfunctioning" "dnsName"="live.public.XXX" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2589084657" "resource_namespace"="fleet-system" "resource_version"="v1" "type"="DNS-01" 
I0203 16:08:37.000261       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-02-03 16:08:37.00024419 +0000 UTC m=+4151.882960426
I0203 16:08:37.297034       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-02-03 16:08:37.297025704 +0000 UTC m=+4152.179741972
I0203 16:08:37.314635       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
E0203 16:08:37.339945       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-kf8vs\" not found"  
E0203 16:08:37.339986       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-kf8vs\" not found"  
E0203 16:08:37.339992       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-kf8vs\" not found"  
E0203 16:08:37.340042       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-kf8vs\" not found"  
I0203 16:08:37.340063       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
E0203 16:08:37.340044       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-kf8vs\" not found"  
E0203 16:08:37.379857       1 controller.go:142] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found"  
E0203 16:08:37.379879       1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-kf8vs\" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="gateway-routing-dd963271-kf8vs" "related_resource_namespace"="fleet-system" "resource_kind"="Order" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 16:08:37.386141       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
I0203 16:08:37.394539       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
E0203 16:08:37.394641       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
E0203 16:08:37.623240       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
E0203 16:08:37.626268       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2846237678" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.626321       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1095463117" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.627315       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1082162377" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.627376       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2356799498" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.627955       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2051963654" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.628014       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2589084657" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.629666       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-790807248" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.629745       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2963433089" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.630137       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2151601995" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.631463       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1537543059" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.631513       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2135797261" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.631896       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-3219159356" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 16:08:37.700958       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "gateway-routing-dd963271-lc4s8" condition "Ready" to 2021-02-03 16:08:37.70095042 +0000 UTC m=+4152.583666661
I0203 16:08:37.737441       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
E0203 16:08:37.759108       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2356799498" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.765283       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2846237678" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.774993       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1095463117" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.779714       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2051963654" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:37.793679       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1082162377" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 16:08:38.012407       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
E0203 16:08:38.850602       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2589084657" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:39.046626       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-790807248" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:39.245451       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2963433089" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:39.245847       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-2356799498\" not found"  
E0203 16:08:39.245872       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-2846237678\" not found"  
E0203 16:08:39.245888       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-1095463117\" not found"  
E0203 16:08:39.245913       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-2051963654\" not found"  
E0203 16:08:39.245931       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-1082162377\" not found"  
E0203 16:08:39.245947       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-2589084657\" not found"  
E0203 16:08:39.245964       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-790807248\" not found"  
E0203 16:08:39.245982       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-2963433089\" not found"  
E0203 16:08:39.449753       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2151601995" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:39.450165       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-2151601995\" not found"  
E0203 16:08:39.647892       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-1537543059\" not found"  
E0203 16:08:39.647892       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-1537543059" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:40.246928       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-2135797261\" not found"  
E0203 16:08:40.246931       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-2135797261" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 16:08:40.473322       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
E0203 16:08:40.645437       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668\" not found" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-kf8vs-2797388668" "related_resource_namespace"="fleet-system" "resource_kind"="Challenge" "resource_name"="gateway-routing-dd963271-kf8vs-2797388668-3219159356" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 16:08:40.646170       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-kf8vs-2797388668-3219159356\" not found"  
I0203 16:10:46.884327       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
I0203 16:10:47.230476       1 acme.go:184] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-lc4s8-2797388668" "related_resource_namespace"="fleet-system" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="gateway-routing-dd963271-lc4s8" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 16:10:47.230680       1 conditions.go:222] Found status change for CertificateRequest "gateway-routing-dd963271-lc4s8" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-03 16:10:47.230674576 +0000 UTC m=+4282.113390812
I0203 16:10:47.256331       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
E0203 16:10:54.153349       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on challenges.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-771636978\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271-lc4s8-2797388668-771636978" 
E0203 16:10:54.165485       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on challenges.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-1049270256\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271-lc4s8-2797388668-1049270256" 
E0203 16:10:54.221355       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on challenges.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-2479260129\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271-lc4s8-2797388668-2479260129" 
E0203 16:10:54.468652       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-771636978\" not found"  
E0203 16:10:54.580774       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-1049270256\" not found"  
E0203 16:10:55.372064       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-2479260129\" not found"  
E0203 16:10:59.153541       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-771636978\" not found"  
E0203 16:10:59.165646       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-1049270256\" not found"  
E0203 16:10:59.221539       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-2479260129\" not found"  
E0203 16:11:07.620964       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on challenges.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-1089111828\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271-lc4s8-2797388668-1089111828" 
E0203 16:11:07.996834       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-1089111828\" not found"  
E0203 16:11:12.621152       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-1089111828\" not found"  
E0203 16:11:21.179397       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on challenges.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-749294436\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271-lc4s8-2797388668-749294436" 
E0203 16:11:21.594101       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-749294436\" not found"  
E0203 16:11:26.179591       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668-749294436\" not found"  
I0203 16:15:18.879518       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="fleet-system/gateway-routing-dd963271" "retry_after"="2021-02-03T17:08:37Z"
I0203 17:03:09.145399       1 setup.go:178] cert-manager/controller/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 17:03:09.669000       1 setup.go:270] cert-manager/controller/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt" "related_resource_namespace"="fleet-system" "resource_kind"="Issuer" "resource_name"="gateway-routing-dd963271" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 17:08:37.000237       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2021-02-03 17:08:37.000224731 +0000 UTC m=+7751.882940993
I0203 17:08:37.315835       1 issuing_controller.go:261] cert-manager/controller/CertificateIssuing "msg"="next private key does not match CSR public key, waiting for requestmanager controller" "key"="fleet-system/gateway-routing-dd963271" "resource_kind"="Secret" "resource_name"="gateway-routing-dd963271-lxlwg" "resource_namespace"="fleet-system" "resource_version"="v1" 
E0203 17:08:37.327840       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
E0203 17:08:37.333579       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-lc4s8\" not found"  
E0203 17:08:37.333794       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-lc4s8\" not found"  
E0203 17:08:37.333832       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-lc4s8\" not found"  
E0203 17:08:37.334011       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-lc4s8\" not found"  
E0203 17:08:37.334050       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-lc4s8\" not found"  
I0203 17:08:37.372744       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "gateway-routing-dd963271-6m5rq" condition "Ready" to 2021-02-03 17:08:37.372735979 +0000 UTC m=+7752.255452228
I0203 17:08:37.389201       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "gateway-routing-dd963271-6m5rq" condition "Ready" to 2021-02-03 17:08:37.389193826 +0000 UTC m=+7752.271910069
E0203 17:08:37.402630       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"gateway-routing-dd963271-6m5rq\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271-6m5rq" 
E0203 17:08:37.758818       1 controller.go:142] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"gateway-routing-dd963271-lc4s8-2797388668\" not found"  
E0203 17:08:37.758867       1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"gateway-routing-dd963271-lc4s8\" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="gateway-routing-dd963271-lc4s8" "related_resource_namespace"="fleet-system" "resource_kind"="Order" "resource_name"="gateway-routing-dd963271-lc4s8-2797388668" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 17:08:39.224066       1 acme.go:184] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="gateway-routing-dd963271-6m5rq-2797388668" "related_resource_namespace"="fleet-system" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="gateway-routing-dd963271-6m5rq" "resource_namespace"="fleet-system" "resource_version"="v1" 
I0203 17:08:39.224264       1 conditions.go:222] Found status change for CertificateRequest "gateway-routing-dd963271-6m5rq" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-03 17:08:39.224257558 +0000 UTC m=+7754.106973808
I0203 17:08:39.285130       1 conditions.go:162] Found status change for Certificate "gateway-routing-dd963271" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-03 17:08:39.285123397 +0000 UTC m=+7754.167839634
E0203 17:08:39.330783       1 controller.go:158] cert-manager/controller/CertificateIssuing "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271" 
E0203 17:08:40.010710       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gateway-routing-dd963271\": the object has been modified; please apply your changes to the latest version and try again" "key"="fleet-system/gateway-routing-dd963271"

In that case, after another hour, I finally got a valid certificate, which is quite some time 😄
I wonder if this is a "normal behavior" a bit like the problem mentionned in the back-off logic here https://github.com/jetstack/cert-manager/issues/3250

Anyway, this seems to be a different case than the initial reported problem since this one fixed itself after some time VS being stuck invalid forever. I still wanted to add some meat to the discussion though.

gmauleon on 3 Feb 2021

Perhaps there is an underlying issue in the code for solving DNS challenges, like this https://github.com/jetstack/cert-manager/issues/3621

udhos on 3 Feb 2021

Was this page helpful?

0 / 5 - 0 ratings