Cert-manager: Cloudflare DNS resolver fails: Error: 6003: Invalid request headers

Created on 18 Jun 2020  ·  23Comments  ·  Source: jetstack/cert-manager

Describe the bug:
Cannot get DNS resolver to work with cloudflare account

cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header"

Expected behaviour:
The challenge to be accepted

Steps to reproduce the bug:
I strictly followed the documentation with an api-token:
https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/#api-tokens
Tried also with the global api-key.

Anything else we need to know?:
The HTTP does not work neither.

"msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "type"="http-01"

Environment details::

  • Kubernetes version (e.g. v1.10.2): v1.18.3
  • Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): bare-metal opennebula
  • cert-manager version (e.g. v0.4.0): 0.15.1
  • Install method (e.g. helm or static manifests): helm

/kind bug

kinbug triagneeds-information
Source
picture ltetrel
👍6

Most helpful comment

I did fixed with global token using 

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key
  namespace: cert-manager
type: Opaque
stringData:
  api-token: xxxxxxxxxx
----
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiKeySecretRef:
            name: cloudflare-api-key
            key: api-token
picture luishdez on 2 Oct 2020
🎉1 👍1

All 23 comments

Getting this too with API keys as well as the API tokens being broken

keslerm picture keslerm on 20 Jun 2020

Also experiencing this, noticing this error:

Status:
  Presented:   false
  Processing:  true
  Reason:      Cloudflare API Error
                Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
  State:       pending
rklubenspies picture rklubenspies on 20 Jun 2020

Did anyone resolve this issue yet? I'm having to manually renew expiring certs and its not fun.

jmgilman picture jmgilman on 24 Jun 2020

Yep this is broken. With global and origin key.

I0625 06:21:26.945303       1 logger.go:149] Calling DNS01ChallengeRecord
I0625 06:21:26.945424       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="cert-isent-co-113337196-2722511178" "resource_namespace"="default" 
I0625 06:21:26.945478       1 controller.go:147] cert-manager/controller/orders "msg"="finished processing work item" "key"="default/cert-isent-co-113337196-2722511178" 
I0625 06:21:28.432671       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
I0625 06:21:28.433044       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="instasent.com" "domain"="instasent.com" "resource_kind"="Challenge" "resource_name"="cert-instasent-com-2799934583-3189091394-3676235352" "resource_namespace"="default" "type"="dns-01" 
I0625 06:21:28.587134       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-isent-co-113337196-2722511178-1495611924" 
I0625 06:21:28.587426       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="isent.co" "domain"="isent.co" "resource_kind"="Challenge" "resource_name"="cert-isent-co-113337196-2722511178-1495611924" "resource_namespace"="default" "type"="dns-01" 
E0625 06:21:28.637534       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
E0625 06:21:28.799572       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-isent-co-113337196-2722511178-1495611924" 
I0625 06:21:48.637843       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
I0625 06:21:48.638227       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="instasent.com" "domain"="instasent.com" "resource_kind"="Challenge" "resource_name"="cert-instasent-com-2799934583-3189091394-3676235352" "resource_namespace"="default" "type"="dns-01" 
I0625 06:21:48.799777       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-isent-co-113337196-2722511178-1495611924" 
I0625 06:21:48.800242       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="isent.co" "domain"="isent.co" "resource_kind"="Challenge" "resource_name"="cert-isent-co-113337196-2722511178-1495611924" "resource_namespace"="default" "type"="dns-01" 
E0625 06:21:48.834186       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
E0625 06:21:48.995882       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-isent-co-113337196-2722511178-1495611924"
luishdez picture luishdez on 25 Jun 2020

For what it's worth, I found and followed this article and somehow this started working again for me. Not sure what was done differently in this article than what I was doing, but it started working for me 🤷‍♂️

rklubenspies picture rklubenspies on 1 Jul 2020

@rklubenspies
So not sure but maybe it is related to the fact that you set:
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

which is not on the official documentation..

I had previous issues with the webhook validation but resolved it (https://github.com/jetstack/cert-manager/issues/2918#issuecomment-646260562), maybe it is relate to that :raised_eyebrow:

ltetrel picture ltetrel on 1 Jul 2020

@ltetrel I think this might have something to do with it—I didn't notice this until you pointed it out, but this is definitely something I didn't set the first time around.

rklubenspies picture rklubenspies on 1 Jul 2020

Still having the same issue even after strictly following https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/ ..

ltetrel picture ltetrel on 2 Jul 2020

@ltetrel Another thing I had to do was restart all the cert-manager resources after making some configuration changes before it would properly process the DNS challenges:

% kubectl get pods -n cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-9b8969d86-242p4               1/1     Running   0          14h
cert-manager-cainjector-8545fdf87c-zqkmd   1/1     Running   0          14h
cert-manager-webhook-8c5db9fb6-b77s9       1/1     Running   0          14h

% kubectl delete pod -n cert-manager cert-manager-9b8969d86-242p4
% kubectl delete pod -n cert-manager cert-manager-cainjector-8545fdf87c-zqkmd
% kubectl delete pod -n cert-manager cert-manager-webhook-8c5db9fb6-b77s9
rklubenspies picture rklubenspies on 2 Jul 2020

Is everything solved now?

/triage needs-information

meyskens picture meyskens on 4 Aug 2020

Hi,

Sadly no. I rely on a certificate I manually asked from certbot and injected into the k8s cluster for now.
I am also trying with an http challenge without more success. This can maybe causes by our network configuration, I will let you know..
Does the DNS challenge need access to the domain (something on our server), or it is just relying on the provider server (cloudflare in our case) ?

ltetrel picture ltetrel on 3 Sep 2020

It needs valid DNS resolvers to look up the SOA record to find the zone to use in Cloudflare

meyskens picture meyskens on 8 Sep 2020

Running into the exact same issue. Disabling the validation did not fix this

gitnik picture gitnik on 1 Oct 2020

OK so using a new API Token instead of the global one works, as we had set apiTokenSecretRef

gitnik picture gitnik on 1 Oct 2020

I did fixed with global token using 

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key
  namespace: cert-manager
type: Opaque
stringData:
  api-token: xxxxxxxxxx
----
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiKeySecretRef:
            name: cloudflare-api-key
            key: api-token
luishdez picture luishdez on 2 Oct 2020
🎉1 👍1

I am running v1.0.2 and was able to get a valid certificate with a HTTP01 challenge.
Did not try the DNS01 challenge but it could be worth that someone check it with the newest cert-manager version.

ltetrel picture ltetrel on 7 Oct 2020

I'm using cert-manager v0.12.0. This is working when I use the Global API key from the admin account.
I'd prefer to swap over to apiTokenSecretRef and use a more restricted API Token, however I keep getting * spec.acme.solvers.dns01.cloudflare.apiKeySecretRef: Required value
I'm using apiVersion: cert-manager.io/v1alpha2

timothyclarke picture timothyclarke on 29 Oct 2020

@timothyclarke this got solved in newer releases of cert-manager

meyskens picture meyskens on 30 Oct 2020

@ltetrel I am currently running cert-manager v1.0.4 (Kubernetes v1.19.4) and still get the same behavior.

Creating ClusterIssuer:

---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret 
  namespace: cert-manager
type: Opaque
stringData:
  api: globalAPIToken
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: sample
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cloudflare-account-key
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api

After requesting a new certificate, the certificaterequest is stuck in pending state:

Status:
  Conditions:
    Last Transition Time:  2020-11-17T16:36:57Z
    Message:               Waiting on certificate issuance from order cattle-system/sample-sz4bc-3622413353: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  11m   cert-manager  Created Order resource cattle-system/sample-sz4bc-3622413353

And the challange has the error:

Status:
  Presented:   false
  Processing:  true
  Reason:      Cloudflare API Error for GET "/zones?name=myzone.ch" 
                Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header
  State:       pending
Events:
  Type     Reason        Age                  From          Message
  ----     ------        ----                 ----          -------
  Normal   Started       13m                  cert-manager  Challenge scheduled for processing
  Warning  PresentError  2m38s (x8 over 13m)  cert-manager  Error presenting challenge: Cloudflare API Error for GET "/zones?name=myzone.ch" 
            Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header

Controller Logs during this Workflow (Certificate Creation):

E1117 17:36:14.468887       1 requestmanager_controller.go:127] cert-manager/controller/CertificateRequestManager "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.469173       1 trigger_controller.go:142] cert-manager/controller/CertificateTrigger "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.469260       1 keymanager_controller.go:137] cert-manager/controller/CertificateKeyManager "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.469888       1 readiness_controller.go:130] cert-manager/controller/CertificateReadiness "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.470115       1 issuing_controller.go:152] cert-manager/controller/CertificateIssuing "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
I1117 17:36:14.831039       1 conditions.go:173] Setting lastTransitionTime for Certificate "buttahtoast" condition "Issuing" to 2020-11-17 17:36:14.831023665 +0000 UTC m=+4255.315979130
I1117 17:36:14.832117       1 conditions.go:173] Setting lastTransitionTime for Certificate "buttahtoast" condition "Ready" to 2020-11-17 17:36:14.832109032 +0000 UTC m=+4255.317064539
E1117 17:36:15.051840       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"buttahtoast\": the object has been modified; please apply your changes to the latest version and try again" "key"="cattle-system/buttahtoast" 
I1117 17:36:15.051942       1 conditions.go:173] Setting lastTransitionTime for Certificate "buttahtoast" condition "Issuing" to 2020-11-17 17:36:15.051933985 +0000 UTC m=+4255.536889506
I1117 17:36:15.861518       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "buttahtoast-pk846" condition "Ready" to 2020-11-17 17:36:15.861506304 +0000 UTC m=+4256.346461717
E1117 17:36:15.913175       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913235       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913335       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913619       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913864       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:16.155564       1 controller.go:142] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353\" not found"  
E1117 17:36:16.155588       1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="buttahtoast-gnbtb" "related_resource_namespace"="cattle-system" "resource_kind"="Order" "resource_name"="buttahtoast-gnbtb-3622413353" "resource_namespace"="cattle-system" "resource_version"="v1" 
E1117 17:36:16.313696       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353\" not found" "related_resource_kind"="Order" "related_resource_name"="buttahtoast-gnbtb-3622413353" "related_resource_namespace"="cattle-system" "resource_kind"="Challenge" "resource_name"="buttahtoast-gnbtb-3622413353-2047870714" "resource_namespace"="cattle-system" "resource_version"="v1" 
E1117 17:36:17.373377       1 sync.go:287] cert-manager/controller/challenges/finalizer "msg"="error cleaning up challenge" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "dnsName"="rancher.myzone.ch" "resource_kind"="Challenge" "resource_name"="buttahtoast-gnbtb-3622413353-2047870714" "resource_namespace"="cattle-system" "resource_version"="v1" "type"="DNS-01" 
E1117 17:36:17.802316       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353\" not found" "related_resource_kind"="Order" "related_resource_name"="buttahtoast-gnbtb-3622413353" "related_resource_namespace"="cattle-system" "resource_kind"="Challenge" "resource_name"="buttahtoast-gnbtb-3622413353-2047870714" "resource_namespace"="cattle-system" "resource_version"="v1" 
E1117 17:36:17.824646       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353-2047870714\" not found"  
E1117 17:36:20.149917       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714" 
E1117 17:36:20.993013       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714" 
E1117 17:36:26.007599       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714" 
E1117 17:36:46.820924       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714"

Not quiet sure what's causing this, I will keep on trying and let you know if I find any solution to it.

oliverbaehler picture oliverbaehler on 17 Nov 2020

siggghhh, I tried different reconfigurations etc. but as it turns out the 5th new generated Cloudflare API Token somehow worked. I don't think it really made a difference that I started double quoting the String in the secret. Maybe someone else encountering this issue can confirm, that a new, correct API Token resolves this issue. Cert-Manager documented which access roles are required: https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/#api-tokens. For my part, idk what the problem with my first attempts was...

oliverbaehler picture oliverbaehler on 17 Nov 2020

I'm facing a similar problem. 

Status:
  Presented:   false
  Processing:  true
  Reason:      Cloudflare API Error for GET "/zones?name=xxx.dev"
                Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
  State:       pending
Events:
  Type     Reason        Age                 From          Message
  ----     ------        ----                ----          -------
  Normal   Started       103s                cert-manager  Challenge scheduled for processing
  Warning  PresentError  37s (x5 over 102s)  cert-manager  Error presenting challenge: Cloudflare API Error for GET "/zones?name=xxx.dev"
            Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header

I tried to use api-token and api-key but without success.

czystyl picture czystyl on 4 Dec 2020

@luishdez which version of certmanager are you running?

I'm on the latest version of certmanager, your yaml simply results in Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header

xlanor picture xlanor on 31 Dec 2020

I was just able to deploy it using an api token generated from cloudflare.

Follow the steps here :
https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/

but, instead of using apiKeySecretRef, using a token here, use apiTokenSecretRef.

You do not need to base64 encode the secret. I read somewhere else that they only got it to work by doing that, so I was on the wrong track.

xlanor picture xlanor on 31 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

matthew-muscat picture matthew-muscat  ·  4Comments
timblakely picture timblakely  ·  4Comments
jbeda picture jbeda  ·  4Comments
jonathan-kosgei picture jonathan-kosgei  ·  4Comments
f-f picture f-f  ·  4Comments