Cert-manager: acme: authorization for identifier is invalid

[discostur]

Describe the bug:
Trying to get a a letsencrypt certificate via cert-manager but always getting the following error:

acme: authorization for identifier cafe.example.com is invalid

Here is the full log output from cert-manager:

I0917 09:58:26.865414       1 prepare.go:488] Accepting challenge for domain "cafe.example.com"
I0917 09:58:26.865469       1 logger.go:63] Calling AcceptChallenge
I0917 09:58:27.586322       1 prepare.go:500] Waiting for authorization for domain "cafe.example.com"
I0917 09:58:27.586368       1 logger.go:78] Calling WaitAuthorization
I0917 09:58:34.435711       1 controller.go:140] clusterissuers controller: syncing item 'letsencrypt-production'
E0917 09:58:34.435777       1 controller.go:170] issuer "letsencrypt-production" in work queue no longer exists
I0917 09:58:34.435810       1 controller.go:154] clusterissuers controller: Finished processing work item "letsencrypt-production"
I0917 09:58:38.360355       1 controller.go:140] clusterissuers controller: syncing item 'letsencrypt-production'
I0917 09:58:38.360686       1 logger.go:88] Calling GetAccount
I0917 09:58:39.137147       1 setup.go:93] letsencrypt-production: verified existing registration with ACME server
I0917 09:58:39.137196       1 helpers.go:147] Setting lastTransitionTime for ClusterIssuer "letsencrypt-production" condition "Ready" to 2018-09-17 09:58:39.137188475 +0000 UTC m=+1613.041914901
I0917 09:58:39.143144       1 controller.go:154] clusterissuers controller: Finished processing work item "letsencrypt-production"
I0917 09:58:39.152763       1 controller.go:140] clusterissuers controller: syncing item 'letsencrypt-production'
I0917 09:58:39.153016       1 logger.go:88] Calling GetAccount
I0917 09:58:39.925087       1 setup.go:93] letsencrypt-production: verified existing registration with ACME server
I0917 09:58:39.925187       1 controller.go:154] clusterissuers controller: Finished processing work item "letsencrypt-production"
I0917 09:58:44.941148       1 helpers.go:201] Found status change for Certificate "cafe-secret" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-09-17 09:58:44.941132175 +0000 UTC m=+1618.845858621
I0917 09:58:44.941208       1 sync.go:276] Error preparing issuer for certificate default/cafe-secret: acme: authorization for identifier cafe.example.com is invalid
E0917 09:58:44.941242       1 sync.go:197] [default/cafe-secret] Error getting certificate 'cafe-secret': secret "cafe-secret" not found
I0917 09:58:44.948056       1 controller.go:168] ingress-shim controller: syncing item 'default/cafe-ingress'
I0917 09:58:44.948139       1 sync.go:140] Certificate "cafe-secret" for ingress "cafe-ingress" already exists
I0917 09:58:44.948198       1 sync.go:143] Certificate "cafe-secret" for ingress "cafe-ingress" is up to date
I0917 09:58:44.948227       1 controller.go:182] ingress-shim controller: Finished processing work item "default/cafe-ingress"
E0917 09:58:44.948498       1 controller.go:180] certificates controller: Re-queuing item "default/cafe-secret" due to error processing: acme: authorization for identifier cafe.example.com is invalid
I0917 09:59:07.306583       1 controller.go:168] ingress-shim controller: syncing item 'default/cm-acme-http-solver-khgj4'
I0917 09:59:07.306684       1 sync.go:65] Not syncing ingress default/cm-acme-http-solver-khgj4 as it does not contain necessary annotations
I0917 09:59:07.306714       1 controller.go:182] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-khgj4"
I0917 09:59:44.948581       1 controller.go:171] certificates controller: syncing item 'default/cafe-secret'
I0917 09:59:44.948769       1 sync.go:274] Preparing certificate default/cafe-secret with issuer
I0917 09:59:44.948964       1 logger.go:43] Calling GetOrder
I0917 09:59:45.168699       1 prepare.go:263] Cleaning up previous order for certificate default/cafe-secret
I0917 09:59:45.168776       1 prepare.go:279] Cleaning up old/expired challenges for Certificate default/cafe-secret
I0917 09:59:45.168790       1 prepare.go:303] Cleaning up challenge for domain "cafe.example.com" as part of Certificate default/cafe-secret
I0917 09:59:45.279308       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=2662927066,certmanager.k8s.io/acme-http-token=1173753626
I0917 09:59:45.292421       1 logger.go:38] Calling CreateOrder
I0917 09:59:45.297443       1 controller.go:168] ingress-shim controller: syncing item 'default/cm-acme-http-solver-khgj4'
E0917 09:59:45.297565       1 controller.go:198] ingress 'default/cm-acme-http-solver-khgj4' in work queue no longer exists
I0917 09:59:45.297599       1 controller.go:182] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-khgj4"
I0917 09:59:46.112522       1 acme.go:126] Created order for domains: [{dns cafe.example.com}]
I0917 09:59:46.112707       1 logger.go:73] Calling GetAuthorization
I0917 09:59:46.279173       1 logger.go:93] Calling HTTP01ChallengeResponse
I0917 09:59:46.279244       1 prepare.go:279] Cleaning up old/expired challenges for Certificate default/cafe-secret
I0917 09:59:46.279258       1 logger.go:68] Calling GetChallenge
I0917 09:59:46.479577       1 http.go:134] wrong status code '404'
I0917 09:59:46.479785       1 pod.go:65] No existing HTTP01 challenge solver pod found for Certificate "default/cafe-secret". One will be created.
I0917 09:59:46.491208       1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/cafe-secret". One will be created.
I0917 09:59:46.556794       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=2662927066,certmanager.k8s.io/acme-http-token=1255345787
I0917 09:59:46.556878       1 ingress.go:102] No existing HTTP01 challenge solver ingress found for Certificate "default/cafe-secret". One will be created.
I0917 09:59:46.586262       1 helpers.go:201] Found status change for Certificate "cafe-secret" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-09-17 09:59:46.586243512 +0000 UTC m=+1680.490970014
I0917 09:59:46.586336       1 sync.go:276] Error preparing issuer for certificate default/cafe-secret: http-01 self check failed for domain "cafe.example.com"
E0917 09:59:46.586380       1 sync.go:197] [default/cafe-secret] Error getting certificate 'cafe-secret': secret "cafe-secret" not found
I0917 09:59:46.590557       1 controller.go:168] ingress-shim controller: syncing item 'default/cm-acme-http-solver-xf27r'
I0917 09:59:46.590615       1 sync.go:65] Not syncing ingress default/cm-acme-http-solver-xf27r as it does not contain necessary annotations
I0917 09:59:46.590648       1 controller.go:182] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-xf27r"
E0917 09:59:46.594168       1 controller.go:180] certificates controller: Re-queuing item "default/cafe-secret" due to error processing: http-01 self check failed for domain "cafe.example.com"
I0917 09:59:46.594357       1 controller.go:168] ingress-shim controller: syncing item 'default/cafe-ingress'
I0917 09:59:46.594469       1 sync.go:140] Certificate "cafe-secret" for ingress "cafe-ingress" already exists
I0917 09:59:46.594531       1 sync.go:143] Certificate "cafe-secret" for ingress "cafe-ingress" is up to date
I0917 09:59:46.594593       1 controller.go:182] ingress-shim controller: Finished processing work item "default/cafe-ingress"
I0917 10:00:07.303689       1 controller.go:168] ingress-shim controller: syncing item 'default/cm-acme-http-solver-xf27r'
I0917 10:00:07.303749       1 sync.go:65] Not syncing ingress default/cm-acme-http-solver-xf27r as it does not contain necessary annotations
I0917 10:00:07.303771       1 controller.go:182] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-xf27r"
I0917 10:00:46.594445       1 controller.go:171] certificates controller: syncing item 'default/cafe-secret'
I0917 10:00:46.594570       1 sync.go:274] Preparing certificate default/cafe-secret with issuer

I tried to curl the ingress path from the acme challenge manually and it looks good for me:

curl cafe.example.com/.well-known/acme-challenge/t2c_vohgYKgeLGkvd8G2Qae4JtjQbB-uKN2P9X-zi0o

t2c_vohgYKgeLGkvd8G2Qae4JtjQbB-uKN2P9X-zi0o.INzU8Z71WEdzjhNLo97HkqShzZoj32SMR177MoI1dNY

I'm running a clusterissuer with the following configuration:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  clusterName: ""
  name: letsencrypt-production
  namespace: ""
spec:
  acme:
    email: [email protected]
    http01: {}
    privateKeySecretRef:
      key: ""
      name: letsencrypt-production
    server: https://acme-v02.api.letsencrypt.org/directory
status:
  acme:
    uri: https://acme-v02.api.letsencrypt.org/acme/acct/41673558
  conditions:
  - lastTransitionTime: 2018-09-17T09:58:39Z
    message: The ACME account was registered with the ACME server
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

Does anybody have an idea whats wrong here?

Environment details::

• Kubernetes version (e.g. v1.10.2): 1.10.2
• cert-manager version (e.g. v0.4.0): 0.5.0

/kind bug

[EamonKeane]

Running into this as well (k8s 1.12.0, cert-manager 0.5.0). Did you resolve it @discostur ? This same setup worked for a few days ago nothing really has changed afaik.

[discostur]

No, did not found any solution yet...

Am 08.10.2018 um 18:44 schrieb Eamon Keane <[email protected][e-mail]>:

Running into this as well (k8s 1.12.0, cert-manager 0.5.0). Did you resolve it @discosturhttps://github.com/discostur ? This same setup worked for a few days ago nothing really has changed afaik.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/jetstack/cert-manager/issues/913#issuecomment-427789826, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACfofDGomg4f1_pRY51yTF6grmEFrKnWks5uiyyygaJpZM4WrnF2.

[EamonKeane]

thanks. In my case I found the culprit to be incorrectly specifying the IP address to nginx-ingress (using metallb as a load balancer for bare metal). Works again now.

[discostur]

any news on this? still seeing the error on cert-manager 0.5.2

[nmammeri]

I see similar error with cert-manager 0.5.2 ... Any new news on this

[pehlert]

@EamonKeane I ran into this using a similar setup (metallb / nginx ingress). What did you do to fix this exactly? Especially where did you specify the IP address to the ingress?

[pehlert]

For me, this was due to an IPv6 record on the same domain that was pointing to a different host. Let's Encrypt seems to prefer IPv6 when resolving addresses. I figured this by visiting the ACME Challenge URL that can be retrieved by a "kubectl describe certificate". It has an "addressUsed" field in its response.

[mcluseau]

for me, it was due to let's encrypt using TCP to query the servers. The DNS IPs of the DNS servers were managed with metallb, which does not allow mixing UDP and TCP in the same service for now. So I had to make 2 more LB services for TCP, and to set the externalTrafficPolicy to Cluster (warning: you then lose the real client IP).

[makomatic]

thanks. In my case I found the culprit to be incorrectly specifying the IP address to nginx-ingress (using metallb as a load balancer for bare metal). Works again now.

Could you give me a quick . hint, please? We are running the same setup (metallb with onprem nodes)

[PabloCastellano]

I have also seen this error using cert-manager 0.5.0 but I have not been able to dig down to the root issue. I found it while I was investigating why a domain was not getting a certificate issued and turns out it had hit the rate limit. So this could be the reason.

I can confirm that after upgrading to cert-manager 0.6.0 this error does not appear anymore. It's also worth noting that according to 0.6.0 release notes this version is much more careful about rate limits and handles them better.