Cert-manager: Cannot create certificates with Cloudflare DNS valildation

[irizzant]

Bugs should be filed for issues encountered whilst operating cert-manager.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:
I can't create a Certificate with Cloudflare DNS challenge.
I installed cert-manager using regular manifests and also verified the installation as reported here.

Then I created 2 ClusterIssuer and requested a Certificate:

cat << EOF | kubectl apply -f-
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: mysecret
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  labels:
    name: letsencrypt-prod
  name: letsencrypt-prod
spec:
  acme:
    email: myemail
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        cloudflare:
          apiKeySecretRef:
            key: api-token
            name: cloudflare-api-token-secret
          email: myemail
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  labels:
    name: letsencrypt-staging
  name: letsencrypt-staging
spec:
  acme:
    email: myemail
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        cloudflare:
          apiKeySecretRef:
            key: api-token
            name: cloudflare-api-token-secret
          email: myemail
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  labels:
    name: wildcard-certificate
  name: wildcard-certificate
  namespace: cert-manager
spec:
  dnsNames:
  - '*.sdb-k8s.gq'
  - sdb-k8s.gq
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: wildcard-certificate
EOF

mysecret is the Global API Key.
In logs, I can see the following Cloudflare API error:

I0524 12:27:19.626896       1 setup.go:162] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" 
I0524 12:27:19.626919       1 controller.go:147] cert-manager/controller/clusterissuers "msg"="finished processing work item" "key"="letsencrypt-staging" 
I0524 12:27:19.659158       1 controller.go:147] cert-manager/controller/challenges "msg"="finished processing work item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:19.659191       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:19.659210       1 controller.go:141] cert-manager/controller/orders "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912" 
I0524 12:27:19.659351       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:27:19.659412       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:27:19.659369       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="sdb-k8s.gq" "domain"="sdb-k8s.gq" "resource_kind"="Challenge" "resource_name"="wildcard-certificate-2352044872-2365235912-1982448156" "resource_namespace"="cert-manager" "type"="dns-01" 
I0524 12:27:19.659453       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="wildcard-certificate-2352044872-2365235912" "resource_namespace"="cert-manager" 
I0524 12:27:19.659473       1 controller.go:147] cert-manager/controller/orders "msg"="finished processing work item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912" 
I0524 12:27:20.640192       1 controller.go:141] cert-manager/controller/orders "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912" 
I0524 12:27:20.640353       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:27:20.640394       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:27:20.640448       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="wildcard-certificate-2352044872-2365235912" "resource_namespace"="cert-manager" 
I0524 12:27:20.640475       1 controller.go:147] cert-manager/controller/orders "msg"="finished processing work item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912" 
E0524 12:27:20.640878       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API error" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:20.640904       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:20.641017       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="sdb-k8s.gq" "domain"="sdb-k8s.gq" "resource_kind"="Challenge" "resource_name"="wildcard-certificate-2352044872-2365235912-1982448156" "resource_namespace"="cert-manager" "type"="dns-01" 
E0524 12:27:21.669664       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API error" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:25.641111       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:25.641339       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="sdb-k8s.gq" "domain"="sdb-k8s.gq" "resource_kind"="Challenge" "resource_name"="wildcard-certificate-2352044872-2365235912-1982448156" "resource_namespace"="cert-manager" "type"="dns-01" 
E0524 12:27:27.096254       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API error" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:47.096434       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-1982448156" 
I0524 12:27:47.096569       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="sdb-k8s.gq" "domain"="sdb-k8s.gq" "resource_kind"="Challenge" "resource_name"="wildcard-certificate-2352044872-2365235912-1982448156" "resource_namespace"="cert-manager" "type"="dns-01" 

I then changed the ClusterIssuer configuration to use apiTokenSecretRef:

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: mysecret
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  labels:
    name: letsencrypt-prod
  name: letsencrypt-prod
spec:
  acme:
    email: myemail
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            key: api-token
            name: cloudflare-api-token-secret
          email: myemail
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  labels:
    name: letsencrypt-staging
  name: letsencrypt-staging
spec:
  acme:
    email: myemail
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            key: api-token
            name: cloudflare-api-token-secret
          email: myemail
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  labels:
    name: wildcard-certificate
  name: wildcard-certificate
  namespace: cert-manager
spec:
  dnsNames:
  - '*.sdb-k8s.gq'
  - sdb-k8s.gq
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: wildcard-certificate

Still I get the same error in the logs:

I0524 12:49:44.832044       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:49:44.832079       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:49:44.832136       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="wildcard-certificate-2352044872-2365235912" "resource_namespace"="cert-manager" 
I0524 12:49:44.832200       1 controller.go:147] cert-manager/controller/orders "msg"="finished processing work item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912" 
I0524 12:49:44.832304       1 controller.go:147] cert-manager/controller/challenges "msg"="finished processing work item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-3394390930" 
I0524 12:49:44.832329       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-3394390930" 
I0524 12:49:44.832422       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="sdb-k8s.gq" "domain"="sdb-k8s.gq" "resource_kind"="Challenge" "resource_name"="wildcard-certificate-2352044872-2365235912-3394390930" "resource_namespace"="cert-manager" "type"="dns-01" 
E0524 12:49:45.946254       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API error" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-3394390930" 
I0524 12:49:45.946529       1 controller.go:141] cert-manager/controller/orders "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912" 
I0524 12:49:45.946668       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:49:45.946703       1 logger.go:149] Calling DNS01ChallengeRecord
I0524 12:49:45.946810       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="wildcard-certificate-2352044872-2365235912" "resource_namespace"="cert-manager" 
I0524 12:49:45.946831       1 controller.go:147] cert-manager/controller/orders "msg"="finished processing work item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912" 
I0524 12:49:45.946897       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-3394390930" 
I0524 12:49:45.947024       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="sdb-k8s.gq" "domain"="sdb-k8s.gq" "resource_kind"="Challenge" "resource_name"="wildcard-certificate-2352044872-2365235912-3394390930" "resource_namespace"="cert-manager" "type"="dns-01" 
I0524 12:49:46.931717       1 controller.go:141] cert-manager/controller/certificates "msg"="syncing item" "key"="cert-manager/wildcard-certificate" 
I0524 12:49:46.931950       1 sync.go:386] cert-manager/controller/certificates "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="wildcard-certificate-2352044872" "related_resource_namespace"="cert-manager" "resource_kind"="Certificate" "resource_name"="wildcard-certificate" "resource_namespace"="cert-manager" 
I0524 12:49:46.932031       1 sync.go:511] cert-manager/controller/certificates "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="wildcard-certificate-2352044872" "related_resource_namespace"="cert-manager" "resource_kind"="Certificate" "resource_name"="wildcard-certificate" "resource_namespace"="cert-manager" "state"="Pending"
I0524 12:49:46.932164       1 controller.go:147] cert-manager/controller/certificates "msg"="finished processing work item" "key"="cert-manager/wildcard-certificate" 
E0524 12:49:46.972348       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API error" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-3394390930" 
I0524 12:49:50.946466       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-3394390930" 
I0524 12:49:50.946635       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="sdb-k8s.gq" "domain"="sdb-k8s.gq" "resource_kind"="Challenge" "resource_name"="wildcard-certificate-2352044872-2365235912-3394390930" "resource_namespace"="cert-manager" "type"="dns-01" 
E0524 12:49:51.991121       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API error" "key"="cert-manager/wildcard-certificate-2352044872-2365235912-3394390930"

Expected behaviour:
Certificate should be issued.

Steps to reproduce the bug:
See above

Anything else we need to know?:

Environment details::

• Kubernetes version (e.g. v1.10.2):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): minikube
• cert-manager version (e.g. v0.4.0): 0.15.0
• Install method (e.g. helm or static manifests): manifests

/kind bug

[meyskens]

.gq is blacklisted by Cloudflare for API based updates of DNS: https://community.cloudflare.com/t/unable-to-update-ddns-using-api-for-some-tlds/167228

I recommend filing an issue with them about this or using another DNS provider.

/close

[jetstack-bot]

@meyskens: Closing this issue.

In response to this:

.gq is blacklisted by Cloudflare for API based updates of DNS: https://community.cloudflare.com/t/unable-to-update-ddns-using-api-for-some-tlds/167228

I recommend filing an issue with them about this or using another DNS provider.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.