Cert-manager: Acme challenge keeps failing due to propagation check failed

[khwerhahn]

The cert-manager pod logs the following on GKE kubernetes cluster:

cert-manager/controller/challenges "msg"="propagation check failed" "error"="presented key (\n\n\u003c!DOCTYPE html\u003e\n\u003chtml lang=\"en\"\u003e\n\n \u003chead\u003e\n \n \u003cmeta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" /\u003e\n \u003ctitle\u003eKonstantin Werhahn | \u003c/title\u003e\n \u003cmeta name=\"description\ m .....CUT..... ss=\"large\"\u003ePage Not Found\u003c/h1\u003e\n \u003cp\u003eThe page you requested couldn't be found - this could be due to a spelling error in the URL or a removed page.\u003c/p\u003e\n \u003ca class=\"btn\" href=\"home\"\u003eGo Back Home\u003c/a\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n .... CUT...../script\u003e\n\n\u003c/html\u003e\n\n) did not match expected (VKC1CqDsd27eEL5Pge8eF14Trm6WOhtRVq5_2xJrc94.AKkjxkjtuUfCyRBifGSsXmUSkyIVj7_vVlXwjXmMU8c)" "dnsName"="khw.io" "resource_kind"="Challenge" "resource_name"="khw-prod-595774067-0" "resource_namespace"="production" "type"="http-01"

On the container of the app I can see the .acme request being logged. Do I need to create an endpoint?

Where do I need to look? What could be the problem?

Environment details::

• Kubernetes version (e.g. v1.10.2): 1.11.10-gke.5
• GKE

/kind bug

[garciapd]

Also happening to me since a couple of days.

Using as a ClusterIssuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: #removed
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable HTTP01 validations
    http01: {}

And

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: platform-tls
  namespace: default
spec:
  secretName: platform-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: {{ .Values.ingress.hostname }}
  dnsNames:
    - {{ .Values.ingress.hostname }}
  acme:
    config:
      - http01:
          ingress: myingress
        domains:
          - {{ .Values.ingress.hostname }}

[Spritekin]

Having this same issue right now.
E0912 23:41:07.960417 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="presented key (\u003c!DOCTYPE html\u003e\r\n\u003chtml\u003e\u003chead\u003e\r\n\u003cscript language=\"javascript\" type=\"text/javascript\" src=\"/tj.js\"\u003e\u003c/script\u003e\r\n\u003cscript language=\"javascript\" type=\"text/javascript\" src=\"/common.js\"\u003e\u003c/script\u003e\r\n \u003cmeta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"\u003e\r\n \u003cmeta name=\"viewport\" content=\"width=device-width,initial-scale=1\"\u003e\r\n \u003ctitle\u003epp VERY LONG TEXT THAT I WONT PASTE HERE VERY LONG TEXT THAT I WONT PASTE HERE VERY LONG TEXT THAT I WONT PASTE HERE VERY LONG TEXT THAT I WONT PASTE HERE VERY LONG TEXT THAT I WONT PASTE HERE href=\"http://lkj7tyj.cn/\" title=\"藍月棋牌\" target=\"_blank\"\u003e藍月棋牌\u003c/a\u003e\u003c/td\u003e\r\n\t\t\u003ctd\u003e\u003ca href=\"http://bs9bne.cn/\" title=\"发发棋牌\" target=\"_blank\"\u003e发发棋牌\u003c/a\u003e\u003c/td\u003e\r\n\t\t\u003ctd\u003e\u003ca href=\"http://soiet.cn/\" title=\"红鼎棋牌\" target=\"_blank\"\u003e红鼎棋牌\u003c/a\u003e\u003c/td\u003e\r\n\t\u003c/tr\u003e\r\n\u003c/table\u003e\r\n\r\n\u003c/body\u003e\r\n\u003c/html\u003e) did not match expected (r8qukrfJT8LaZBZYQffi7H3Q6bHXKnMwoBm9Hor5Ktk.dKpjYj83if1nmLEhqS5zLGANE-MbW_3H6jIdvu0pDc0)" "dnsName"="test.mydomain.com" "resource_kind"="Challenge" "resource_name"="test-mydomain-crt-3891502022-0" "resource_namespace"="default" "type"="http-01"

Not really sure how to solve this. Maybe all that page is the response from a server that instead of returning a proper key to compare it returns all that.

[austinmcconnell]

The same issue is happening to me. In our case, the http01 validation request appears to be routed to the app/service pod, not the HTTP01 Solver pod, and is returning the entire rendered html page instead of the expected challenge response. Any ideas for how to fix this?

[jeffijoe]

I have the same issue as @austinmcconnell which seems to be the same as the OP.

[munnerz]

Can you please try again with the newest version of cert-manager?

I don't think this is a bug as such, rather a misconfiguration somewhere. If you're using ingress-gce, which it looks like you might be, please don't forget to set the acme.cert-manager.io/http01-override-ingress-name annotation on your Certificate resource to the name of the Ingress used to serve traffic for your domain: https://github.com/jetstack/cert-manager/blob/8d12d351e8a098e7d1323ffb99600571f6368095/pkg/apis/acme/v1alpha2/types.go#L20-L25

[austinmcconnell]

In case it helps others, my issue was indeed a misconfiguration; however, it was a misconfiguration in the DNSSEC authentication chain, not in the cert-manager deployment or config itself.

[jeffijoe]

@austinmcconnell how did that make it serve some pod's HTML? 😮

[austinmcconnell]

@jeffijoe That answer is a bit beyond my knowledge level lol. I struggled with getting past this error for several days, so it's possible there was another issue that I fixed in the interim. I don't see any commits in my infrastructure project showing cert-manager config changes in that time though 🤷‍♂. Sorry to not be more help.

[spacejam101]

Seeingt this also to my challenges. Any solutions?

[Sudharma]

Any solutions?? I have same problem and I am using azure with istio gateway.

[woxxy]

I am also seeing this, both on Azure and GCP. I am using ingress-nginx.

The error (third line) shows the HTML fetched from the domain of the certificate itself rather than a hash.

I1126 17:41:28.276214       1 sync.go:50] cert-manager/controller/ingress-shim "level"=0 "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="cm-acme-http-solver-xxxxx" "resource_namespace"="default" 

I1126 17:41:28.276232       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/cm-acme-http-solver-xxxxx" 

E1126 17:41:28.293900       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="presented key (**HTML HERE**) did not match expected (**HASH HERE**)" "dnsName"="abc.domain.com" "resource_kind"="Challenge" "resource_name"="tls-secret-XXXXXXXX-XXXXXXXXX-XXXXXX" "resource_namespace"="default" "type"="http-01" 

[seifane]

I had the same error in my case it seems it was a dns propagation issue. I was migrating a service to a cluster. I changed the subdomain and everything started working again.

[54853315]

@seifane Hello, can show your yaml files to help us review the way to fix this problem ? thank you .

[54853315]

I have some problem here , logs show me ,request get my HTML code :

I1220 07:38:36.370085       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="example.nzcymy00ndy.testing.sfkedu.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-h4f6c" "related_resource_namespace"="example-testing" "resource_kind"="Challenge" "resource_name"="example-testing-acme-2797826842-435170453-3357797531" "resource_namespace"="example-testing" "type"="http-01"
E1220 07:38:36.383399       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="presented key (\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n\t\u003chead\u003e\n\t\t\u003cmeta charset=\"utf-8\"\u003e\n\t\t\u003ctitle\u003eLoader #10\u003c/title\u003e\n\t\t\u003clink rel=\"stylesheet\" type=\"text/css\" href=\"css/style-10.css\" /\u003e\n\t\t\u003c!--[if IE]\u003e\n\t\t\u003cscript src=\"js/html5.js\"\u003e\u003c/script\u003e\n\t\t\u003c![endif]--\u003e\n\t\t\u003cstyle\u003e\n \n \na{\nbackground: #13A3A5;\npadding:5px;\nmargin:10px;\ndisplay:block;\nfont-weight:100;\ncursor:pointer;\nfont-size:1.5em;\nfloat:left;\ntext-decoration:none;\nfont-size:18px;\ncolor:white\n}\n\u003c/style\u003e\n\t\u003c/head\u003e\n\u003cbody\u003e\n\n\t\u003cdiv class=\"loader\"\u003e\n\t\t\u003cdiv\u003eW\u003c/div\u003e\n\t\t\u003cdiv\u003eA\u003c/div\u003e\n\t\t\u003cdiv\u003eH\u003c/div\u003e\n\t\t\u003cdiv\u003eA\u003c/div\u003e\n\t\t\u003cdiv\u003eH\u003c/div\u003e\n\t\t\u003cdiv\u003eA\u003c/div\u003e\n\t\t\u003cdiv\u003e\u003c/div\u003e\n\t\t\u003cdiv\u003e\u003c/div\u003e\n\t\t\u003cdiv\u003e\u003c/div\u003e\n\t\t\u003cdiv\u003e\u003c/div\u003e\n\t\u003c/div\u003e\n\n\n\u003c/body\u003e\n\u003c/html\u003e\n) did not match expected (eATLBe9P_yV2AnQ3ZnUwxe54CGS78XpkJS2B54ee9LU.PcWusSqohOZqPLEeei5nwGxsyvaTtaCG6ot0SFVPCig)" "dnsName"="example.nzcymy00ndy.testing.sfkedu.com" "resource_kind"="Challenge" "resource_name"="example-testing-acme-2797826842-435170453-3357797531" "resource_namespace"="example-testing" "type"="http-01"
I1220 07:38:36.383453       1 controller.go:135] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="example-testing/example-testing-acme-2797826842-435170453-3357797531"

[anatoly-kryzhanovsky]

i have encountered same issue now
this is my certificate definition:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: service-certificate
  namespace: host
spec:
  secretName: service-tls-certificate
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - service.myprettydomain.com

and my ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: service-ingress
  namespace: host
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - service.myprettydomain.com
    secretName: service-tls-certificate
  rules:
  - host: service.myprettydomain.com
    http:
      paths:
      - backend:
          serviceName: service
          servicePort: 80
        path: /
---
apiVersion: v1
kind: Service
metadata:
  name: service
  namespace: host
spec:
  type: ExternalName
  externalName: service.local

service.myprettydomain.com changed, but real domain has correct dns entries (i can ping it from lan and wan), k8s cluster can ping service.local which service in lan area

after i'm apply this configuration i see that challenge stuck in state for validation key because it receive raw html from service.local but not validation key

is this information enough or you need something else?

[anatoly-kryzhanovsky]

one more additional.
after i applied configuration i can see additional ingress:
kubectl get ingress -n host

NAME                        HOSTS                         ADDRESS       PORTS     AGE
......................
cm-acme-http-solver-77wkt   service.myprettydomain.com     192.168.0.6   80        22h
service-ingress              service.myprettydomain.com     192.168.0.6   80, 443   4d19h

and this ingress look for following rule:

spec:
  rules:
  - host: service.myprettydomain.com
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-qw7l2
          servicePort: 8089
        path: /.well-known/acme-challenge/_<edited>_

so everything looks fine, but still didn't work

[anatoly-kryzhanovsky]

i try to update to last stable version (previous i use v0.11.0) (to v0.12.0), but still no progress - i have only one ingress which can issue certificates without any problem, but other failed on validation key because handler does not handle /.well-known path and forward it to service

[anatoly-kryzhanovsky]

ok... i solve my issue, i'm stupid, sorry for noise..
my mistake was linked to dns records - in my local dns all my hosts, which i used in external names resolve to real internal ip of computers, not the cluster ip..