Cert-manager: Certificate status is stuck in FALSE state

[sgraza]

Hello, I just setup cert-manager with letsencrypt clusterissuer.
When I tried to create kubernetes ingress, cert was created but in FALSE state and the challenges stuck in PENDING state.

Kubernetes Version: v1.14.5
Cert-manager: v0.9.1

Error from challenges shows: Waiting for http-01 challenge propagation: wrong status code ‘404’, expected ‘200’

letsencrpty-prod:

Name: letsencrypt-prod
Namespace:
Labels:
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2019-09-03T09:28:23Z
Generation: 2
Resource Version: 247873
Self Link: /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-prod
UID: 2cd96251-ce2d-11e9-82c5-848f69e1c04f
Spec:
Acme:
Email: *****
http01:
Private Key Secret Ref:
Name: letsencrypt-prod
Server: https://acme-v02.api.letsencrypt.org/directory
Status:
Acme:
Last Registered Email: ****
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/**
Conditions:
Last Transition Time: 2019-09-03T09:28:24Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events:

ingress:

Name: external-wordpress
Namespace: wordpress
Address: 11.0.0.36,11.0.0.45
Default backend: default-http-backend:80 ()
TLS:
wordpress-tls terminates blog.paloitcloud.com.sg
Rules:
Host Path Backends
blog.paloitcloud.com.sg
/.well-known/acme-challenge/u7vTuRvEYPch15hfEWTiGd9XUbRFJ-LXuKB7o_y9rEU cm-acme-http-solver-v98mb:8089 (10.42.2.9:8089)
/ external-service:80 (192.168.0.6:443)
Annotations:
nginx.ingress.kubernetes.io/from-to-www-redirect: true
nginx.ingress.kubernetes.io/permanent-redirect: https://b.paloitcloud.com.sg
nginx.ingress.kubernetes.io/ssl-redirect: true
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
kubernetes.io/tls-acme: true
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/secure-backends: true
certmanager.k8s.io/acme-http01-edit-in-place: true
field.cattle.io/publicEndpoints: [{“addresses”:[“11.0.0.36”],“port”:443,“protocol”:“HTTPS”,“serviceName”:“wordpress:cm-acme-http-solver-v98mb”,“ingressName”:“wordpress:external-wordpress”,“hostname”:“blog.paloitcloud.com.sg”,“path”:"/.well-known/acme-challenge/u7vTuRvEYPch15hfEWTiGd9XUbRFJ-LXuKB7o_y9rEU",“allNodes”:true},{“addresses”:[“11.0.0.36”],“port”:443,“protocol”:“HTTPS”,“serviceName”:“wordpress:external-service”,“ingressName”:“wordpress:external-wordpress”,“hostname”:“blog.paloitcloud.com.sg”,“path”:"/",“allNodes”:true}]
Events:
Type Reason Age From Message
Normal CREATE 4m12s nginx-ingress-controller Ingress wordpress/external-wordpress
Normal CREATE 4m12s nginx-ingress-controller Ingress wordpress/external-wordpress
Normal CreateCertificate 4m12s cert-manager Successfully created Certificate “wordpress-tls”
Normal UPDATE 3m51s (x3 over 4m10s) nginx-ingress-controller Ingress wordpress/external-wordpress
Normal UPDATE 3m51s (x3 over 4m10s) nginx-ingress-controller Ingress wordpress/external-wordpress

certificate:

Name: wordpress-tls
Namespace: wordpress
Labels:
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-09-03T09:32:46Z
Generation: 2
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: external-wordpress
UID: c9dad3ae-ce2d-11e9-82c5-848f69e1c04f
Resource Version: 248594
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/wordpress/certificates/wordpress-tls
UID: c9dbd29f-ce2d-11e9-82c5-848f69e1c04f
Spec:
Acme:
Config:
Domains:
blog.paloitcloud.com.sg
http01:
Ingress: external-wordpress
Dns Names:
blog.paloitcloud.com.sg
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: wordpress-tls
Status:
Conditions:
Last Transition Time: 2019-09-03T09:32:46Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message
Normal OrderCreated 6m4s cert-manager Created Order resource “wordpress-tls-1424225308”

challenge:

Name: wordpress-tls-1424225308-0
Namespace: wordpress
Labels: acme.cert-manager.io/order-name=wordpress-tls-1424225308
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Challenge
Metadata:
Creation Timestamp: 2019-09-03T09:32:48Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 4
Owner References:
API Version: certmanager.k8s.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Order
Name: wordpress-tls-1424225308
UID: c9de73aa-ce2d-11e9-82c5-848f69e1c04f
Resource Version: 248619
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/wordpress/challenges/wordpress-tls-1424225308-0
UID: caac4663-ce2d-11e9-82c5-848f69e1c04f
Spec:
Authz URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/159262939
Config:
http01:
Ingress: external-wordpress
Dns Name: blog.paloitcloud.com.sg
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod
Key: u7vTuRvEYPch15hfEWTiGd9XUbRFJ-LXuKB7o_y9rEU.ZdVrcRAApk77sHsqfEUWSNz9a9Zu6bg5QScLJtjpW0o
Token: u7vTuRvEYPch15hfEWTiGd9XUbRFJ-LXuKB7o_y9rEU
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/159262939/UqERSA
Wildcard: false
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation: wrong status code ‘404’, expected ‘200’
State: pending
Events:
Type Reason Age From Message
Normal Started 7m30s cert-manager Challenge scheduled for processing
Normal Presented 7m30s cert-manager Presented challenge using http-01 challenge mechanism

[jonfairbanks]

I'm also running into this issue on v0.9.0 and now v0.10.0. I've posted in the kubernetes #cert-manager Slack channel as well.

Hoping someone can advise on how to troubleshoot this further as the troubleshooting portion of the Challenges doc is still a To-do.

Thanks!

[angelosanramon]

I am also getting a similar issue. I am using dns01 challenge solver though. I hope there is a good suggestion on how to resolve this.

[daniel-schulz]

I have also a similar issue.

I try to get the cert-manager in my environment running (with letsencrypt).
The challenge got stuck in state 'pending' with the following reason: Waiting for http-01 challenge propagation: wrong status code '404', expected '200'.

I also checked similar issues on GitHub but they was not helpful.

I'm thinking that this might be because of my environment with istio and GKE:
GKE: 1.14.6-gke-2
Istio: 1.1.13-gke-0
cert manager: 0.10.1

This is the 'ClusterIssuer' I've configured:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    email: info@**************
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-key
    solvers:
    - http01:
        ingress: {}

I also tried to set 'spec.acme.solvers.http01.ingress.class' to 'nginx' like described in the install guide but also didn't work.

My 'Certificate' settings:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: api-mydomain-de
  namespace: istio-system
spec:
  secretName: api-mydomain-de-tls
  renewBefore: 360h # 15d
  commonName: api.mydomain.de
  dnsNames:
  - api.mydomain.de
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer

And my 'Gateway' config:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-gateway-gw
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true # sends 301 redirect for http requests
    hosts:
    - api.mydomain.de
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - api.mydomain.de

Has anyone an idea what the issue is and how to solve it?
Just tell me if you need further information.

Thanks in advance.
Daniel

[anatoly-kryzhanovsky]

i'm stuck with same issue, any workaround?

[adibrastegarnia]

@munnerz
Any updates on this. Here is the log of k8s:

kubectl describe certificaterequests.cert-manager.io quickstart-example-tls-1141217006
Name:         quickstart-example-tls-1141217006
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: quickstart-example-tls
              cert-manager.io/private-key-secret-name: quickstart-example-tls
API Version:  cert-manager.io/v1alpha2
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2019-11-14T16:25:10Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  quickstart-example-tls
    UID:                   9c7b86a6-eaa2-4de0-aded-9507395b5de9
  Resource Version:        12692
  Self Link:               /apis/cert-manager.io/v1alpha2/namespaces/default/certificaterequests/quickstart-example-tls-1141217006
  UID:                     b41a2afb-d5d8-459d-a23b-5ae41eb77118
Spec:
  Csr:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2l6Q0NBWE1DQVFBd0Z6RVZNQk1HQTFVRUNoTU1ZMlZ5ZEMxdFlXNWhaMlZ5TUlJQklqQU5CZ2txaGtpRwo5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBdDhRb09pbkNBUElVQklHQWQ2dWJDMHk2TWZoQ090ZnJFUjZ5Cjlva0VOdEI2dnpkTUJUbHVncDh0dUF1eGdJSE1NeG40cHBnVVdGdlNndklqdXNwYnRsL1k3alc3aTAvOU5idTIKL1lIeVEwYWZvVVB1VGNIdTIrZ0FCSFB6by9kTFVlaFYwVEpyYjM3N2lhQnI0MzFyUTRGR2hkNGlaV2t2SWFhSAp1cjI1Y21YdzI2ZlFNRTBIcFdGb1ExSmlkS3BuY2ZMcmRvZWpmV0pkRis2VkhyRlVRUlcxMkZSY1dBcU5Rc3FrCk5uVU1KaDhQWno1VGkyN2h2WFdkMC84R2xDdVAySFlmWDlwczhpRWpWTExpNjNCdEdBU09vZDV5QWRrZjh0bS8KeHNoZHFtNUY3b0k2UFp3QThNcGQ1K2N2Q0YrR0Y4Yjk1NEthcGltczRtYzlQSjBuTFFJREFRQUJvQzh3TFFZSgpLb1pJaHZjTkFRa09NU0F3SGpBY0JnTlZIUkVFRlRBVGdoRmhaR2xpZEdWemRDNXVaM0p2YXk1cGJ6QU5CZ2txCmhraUc5dzBCQVFzRkFBT0NBUUVBc1ZTM3hMbVNjaG52NnFBUnN1dDNTbzhHQWx3bHFqeUQ3ZHIyNHIrMDBPcGIKcDBkSDBNeVBnQm9pblR5cWJCTUtXdU92YXczdjBGMnc3MGNFd2dsS0h1aE5ZTWQzNlJ3N3RQdFlwbHhOaU1URwpQcWZ5THl3b2JmbE4xcEs2QXlJMnFsU2hCVXh6c3VzNHNhRlpkM254MVpGNTVpNFk1TTU1czVBaVVDYjdWYjU3CkVvVForbjdiNGtKSGExTEZDN091aWhtR2syVGx3cVZkays0L21xdm9HS2lwemxnMHk0aENKR1lHRXlYTWdhWWkKMkcvTUZSQU5aRllPTTRGd1prOUF6WnpTZjlhZElqUCtCbUwzNWdrbWFEWFVTRjlqT1V4K0t4N04wL0JjeWl3ZApFZHVGT2dXMzZHVWkrK3JMWEZKUzBwY01vOW0vcGs4cis0LzlkWUZDaHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-staging
Status:
  Conditions:
    Last Transition Time:  2019-11-14T16:25:10Z
    Message:               Waiting on certificate issuance from order default/quickstart-example-tls-1141217006-3939871124: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  OrderCreated  5m36s  cert-manager  Created Order resource default/quickstart-example-tls-1141217006-3939871124

[yann-soubeyrand]

@adibrastegarnia kubectl describe challenges should give you more information.

[adibrastegarnia]

@yann-soubeyrand : Thanks. Perhaps, it is related to external IP thing? what do you think?

kubectl describe challenges.acme.cert-manager.io

  Solver:
    http01:
      Ingress:
        Class:  nginx
  Token:        pTf24V8yzV14w1pOPeQlBid8MhEsjp7CcBfa5gMyESw
  Type:         http-01
  URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21092633/ml0Ejw
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '401', expected '200'
  State:       pending
Events:        <none>

[yann-soubeyrand]

Waiting for http-01 challenge propagation: wrong status code '401', expected '200'

You should verify your ingress controller configuration. Which ingress controller are you using?

[adibrastegarnia]

@yann-soubeyrand nginx. I am testing on localhost but I use ngrok to expose the service but I am not sure if we still need to have an external IP address? How can we test ACME certificate locally if we don't have a public IP address?

[yann-soubeyrand]

Yes, you need a public IP address. ACME challenge of type HTTP-01 works by checking that a token is returned at a special path under the domain which a certificate is requested for. If you don't have a public IP address, you can use DNS-01 challenge (you'll need to have control over the DNS zone containing your domain and the DNS provider must be supported by cert-manager).

[adibrastegarnia]

@yann-soubeyrand
Thanks. Do you know what DNS providers supported by cert-manager?
I found them here: https://docs.cert-manager.io/en/latest/tasks/issuers/setup-acme/dns01/

[JoshDaone]

refer to:

1636

[archonic]

It sure would be handy to see the errors visible with kubectl describe challenges on the certificate and/or issuer events.

[zored]

I have the same problem only on *.de domain #2517
Are there any ways to debug the reason of it?

[munnerz]

I'm going to close this issue as it seems to have become a catch-all for issues relating to failing to validate HTTP01/DNS01 challenges. If you've got general discussion points/questions to bring up, please go to the Slack channel (#cert-manager on slack.k8s.io).

It sure would be handy to see the errors visible with kubectl describe challenges on the certificate and/or issuer events.

v0.12 includes https://github.com/jetstack/cert-manager/pull/2261 which improves the information we display when a Challenge is failing. That said, if there's any more specific issues/information that is not being surfaced, please open a new specific issue to describe the information that you found via e.g. logs that was not made clear/available via the Challenge resource, as we aim to not require users to check controller logs in order to discern what's going on with their Certificate.

[InsOpDe]

For those who get here wanting possible solutions:
my clusterissue-definition was erroneous:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: [email protected]
    privateKeySecretRef:
      name: cluster-issuer-account-key
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx-ingress

it should be

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: [email protected]
    privateKeySecretRef:
      name: cluster-issuer-account-key
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx

instead.
Notice the solvers: part

if this part isnt correctly set, it leads to the 404 error