Cert-manager: ClusterIssuer LetsEncrypt Certificate not ready

Created on 30 Sep 2020  ·  14Comments  ·  Source: jetstack/cert-manager

Followed steps mentioned at https://cert-manager.io/docs/installation/kubernetes/
$ kubectl create namespace cert-manager

$ helm repo add jetstack https://charts.jetstack.io

$ helm repo update

$ helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.0.2 \
  --set installCRDs=true

$  kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7cdc47446d-q6cq8              1/1     Running   0          97m
cert-manager-cainjector-6754f97f69-7kcx8   1/1     Running   0          97m
cert-manager-webhook-7b56df6ddb-hzgzl      1/1     Running   0          97m

ClusterIssuer

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-test-key
    solvers:
    - dns01:
        route53:
          region: us-west-2
          hostedZoneID: xxxxxxxx
          accessKeyID: xxxxxx
          secretAccessKeySecretRef:
            name: aws-secret
            key: secret_key
      selector:
          dnsZones:
            - "example.com"

Certificate

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: test-cert
  namespace: cert-manager
spec:
  commonName: '*.test.example.com'
  secretName: test-cert
  dnsNames:
    - '*.test.example.com'
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer

$ kubectl -n cert-manager get certificate
NAME        READY   SECRET      AGE
test-cert   False   test-cert   25s

certificate test-cert

$ kubectl -n cert-manager describe certificate test-cert
NAME        READY   SECRET      AGE
test-cert   False   test-cert   25s
roydon.pereira@lc-mbp-09 cert-manager % kubectl -n cert-manager describe certificate test-cert
Name:         test-cert
Namespace:    cert-manager
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"test-cert","namespace":"cert-manager"},...
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-09-30T18:20:14Z
  Generation:          1
  Resource Version:    114851206
  Self Link:           /apis/cert-manager.io/v1/namespaces/cert-manager/certificates/test-cert
  UID:                 c552c42a-6202-40f8-8e9d-f47387f3cf1c
Spec:
  Common Name:  *.test.example.com
  Dns Names:
    *.test.example.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt
  Secret Name:  test-cert
Status:
  Conditions:
    Last Transition Time:        2020-09-30T18:20:14Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
    Last Transition Time:        2020-09-30T18:20:14Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
  Next Private Key Secret Name:  test-cert-j2bdf
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    116s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  116s  cert-manager  Stored new private key in temporary Secret resource "test-cert-j2bdf"
  Normal  Requested  116s  cert-manager  Created new CertificateRequest resource "test-cert-2glmx"


$ kubectl -n cert-manager get CertificateRequest
NAME              READY   AGE
test-cert-2glmx   False   6m6s

CertificateRequest

$ kubectl -n cert-manager describe CertificateRequest test-cert-2glmx
Name:         test-cert-2glmx
Namespace:    cert-manager
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: test-cert
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: test-cert-j2bdf
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"test-cert","namespace":"cert-manager"},...
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2020-09-30T18:20:14Z
  Generate Name:       test-cert-
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  test-cert
    UID:                   c552c42a-6202-40f8-8e9d-f47387f3cf1c
  Resource Version:        114851218
  Self Link:               /apis/cert-manager.io/v1/namespaces/cert-manager/certificaterequests/test-cert-2glmx
  UID:                     d275cb9f-a1d0-417c-a0de-6a1a76193c31
Spec:
  Issuer Ref:
    Kind:   ClusterIssuer
    Name:   letsencrypt
  Request:  LS0t...002b1JBCkZGREU3Mk.....nOFlqQW9......FvWkZhb....NjlzM3RtZm....gvW...0tLUV....0tLS0K
Status:
  Conditions:
    Last Transition Time:  2020-09-30T18:20:14Z
    Message:               Waiting on certificate issuance from order cert-manager/test-cert-2glmx-2027085711: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  OrderCreated  6m42s  cert-manager  Created Order resource cert-manager/test-cert-2glmx-2027085711

Certificate is in pending state. What I am missing here ?

Environment details::

  • Kubernetes version (e.g. v1.10.2): v1.16.13-eks-2ba888
  • Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AWS
  • cert-manager version (e.g. v0.4.0): v1.0.2
  • Install method (e.g. helm or static manifests): helm

/kind bug

triagsupport
Source
picture Roydon
👍2

All 14 comments

Have you looked at https://cert-manager.io/docs/faq/acme/

/remove-kind bug
/triage support

meyskens picture meyskens on 2 Oct 2020

according to https://cert-manager.io/docs/configuration/acme/dns01/#setting-nameservers-for-dns01-self-check
I re-install cert-manager 

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.0.2 \
  --set 'installCRDs=true,extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}'

DNS-01 challenges failed with following error

Status:
  Presented:   false
  Processing:  false
  Reason:      Error accepting authorization: acme: authorization error for test.example.com: 400 urn:ietf:params:acme:error:dns: DNS problem: SERVFAIL looking up CAA for test.example.com - the domain's nameservers may be malfunctioning
  State:       invalid
Events:
  Type     Reason     Age   From          Message
  ----     ------     ----  ----          -------
  Normal   Started    23m   cert-manager  Challenge scheduled for processing
  Normal   Presented  22m   cert-manager  Presented challenge using DNS-01 challenge mechanism
  Warning  Failed     21m   cert-manager  Accepting challenge authorization failed: acme: authorization error for test.example.com: 400 urn:ietf:params:acme:error:dns: DNS problem: SERVFAIL looking up CAA for test.example.com - the domain's nameservers may be malfunctioning

I see _acme-challenge.test.example.com TXT gets created in route53.

Roydon picture Roydon on 2 Oct 2020

Same steps, same issue, except I'm using http01.

Rancher 2.4.8 over K3s
Kubernetes:

  • Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

  • Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9+k3s1", GitCommit:"630bebf94b9dce6b8cd3d402644ed023b3af8f90", GitTreeState:"clean", BuildDate:"2020-09-17T19:05:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Helm: v3.3.4

On Certificate Requests:

Status:
  Conditions:
    Last Transition Time:  2020-10-07T01:54:24Z
    Message:               Waiting on certificate issuance from order wordpress/tls-demo-ingress2-svdvs-2320172803: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  16m   cert-manager  Created Order resource wordpress/tls-demo-ingress2-svdvs-2320172803
  Normal  OrderPending  16m   cert-manager  Waiting on certificate issuance from order wordpress/tls-demo-ingress2-svdvs-2320172803: ""

Logs from cert-manager:

Normal   OrderCreated  47s   cert-manager  Created Order resource wordpress/tls-demo-ingress-5jv9w-2320172803
  Normal   OrderPending  47s   cert-manager  Waiting on certificate issuance from order wordpress/tls-demo-ingress-5jv9w-2320172803: ""
  Warning  OrderFailed   12s   cert-manager  Failed to wait for order resource "tls-demo-ingress-5jv9w-2320172803" to become ready: order is in "invalid" state:

And from acme solver:

I1007 01:01:47.893291       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="demo.*****.br" "path"="/.well-known/acme-challenge/PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "expected_host"="demo.*****.br"
I1007 01:01:47.893353       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="demo.*****.br" "path"="/.well-known/acme-challenge/PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "expected_token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY"
I1007 01:01:47.893408       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="demo.****.br" "path"="/.well-known/acme-challenge/PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY"
E1007 01:02:01.852568       1 app.go:43] cert-manager/acmesolver "msg"="error shutting down acmesolver server" "error"="context canceled"
Error: http: Server closed
Usage:
  acmesolver [flags]
Flags:
      --domain string     the domain name to verify
  -h, --help              help for acmesolver
      --key string        the challenge key to respond with
      --listen-port int   the port number to listen on for connections (default 8089)
      --token string      the challenge token to verify against
http: Server closed
bemanuel picture bemanuel on 7 Oct 2020

Same steps, same issue, except I'm using http01.

Rancher 2.4.8 over K3s
Kubernetes:

* Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

* Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9+k3s1", GitCommit:"630bebf94b9dce6b8cd3d402644ed023b3af8f90", GitTreeState:"clean", BuildDate:"2020-09-17T19:05:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Helm: v3.3.4

On Certificate Requests:

Status:
  Conditions:
    Last Transition Time:  2020-10-07T01:54:24Z
    Message:               Waiting on certificate issuance from order wordpress/tls-demo-ingress2-svdvs-2320172803: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  16m   cert-manager  Created Order resource wordpress/tls-demo-ingress2-svdvs-2320172803
  Normal  OrderPending  16m   cert-manager  Waiting on certificate issuance from order wordpress/tls-demo-ingress2-svdvs-2320172803: ""

Logs from cert-manager:

Normal   OrderCreated  47s   cert-manager  Created Order resource wordpress/tls-demo-ingress-5jv9w-2320172803
  Normal   OrderPending  47s   cert-manager  Waiting on certificate issuance from order wordpress/tls-demo-ingress-5jv9w-2320172803: ""
  Warning  OrderFailed   12s   cert-manager  Failed to wait for order resource "tls-demo-ingress-5jv9w-2320172803" to become ready: order is in "invalid" state:

And from acme solver:

I1007 01:01:47.893291       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="demo.*****.br" "path"="/.well-known/acme-challenge/PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "expected_host"="demo.*****.br"
I1007 01:01:47.893353       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="demo.*****.br" "path"="/.well-known/acme-challenge/PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "expected_token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY"
I1007 01:01:47.893408       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="demo.****.br" "path"="/.well-known/acme-challenge/PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY" "token"="PHc2fU6RnbCrkXAYi6vshK2FnASYYpjxSbAugSrXUcY"
E1007 01:02:01.852568       1 app.go:43] cert-manager/acmesolver "msg"="error shutting down acmesolver server" "error"="context canceled"
Error: http: Server closed
Usage:
  acmesolver [flags]
Flags:
      --domain string     the domain name to verify
  -h, --help              help for acmesolver
      --key string        the challenge key to respond with
      --listen-port int   the port number to listen on for connections (default 8089)
      --token string      the challenge token to verify against
http: Server closed

Solved:
I'm using k3s with traefik, so it work changing the Cluster Issuer to use traefik:
solvers:
- http01:
ingress:
class: traefik

And I'm using LB Layer 4 on pfSense, for some reason just works when I add: serviceType: ClusterIP
- http01:
ingress:
serviceType: ClusterIP
class: traefik

bemanuel picture bemanuel on 7 Oct 2020

Can you show your manifest for ClusterIssuer & Certificate (masking credentials)

Roydon picture Roydon on 7 Oct 2020

Can you show your manifest for ClusterIssuer & Certificate (masking credentials)

Sure, I didn't create the Certificate, but this is what I did:
Created the ClusterIssuers:
-- Staging

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <email_here>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-staging
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress: 
           serviceType: ClusterIP
           class: traefik

-- Prod

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <email_here>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-production
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress: 
           serviceType: ClusterIP
           class: traefik

Ingress, put annotations and tls:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: traefik
...
tls:
  - hosts:
    - <fqdn.domain>
    secretName: <some-secretname-for-your-ing>
bemanuel picture bemanuel on 7 Oct 2020

cert-manager version 1.0.3
kubernetes version 1.19.3

I have the same problem.
But my files are looking a bit different.

# cluster-issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: issuer-account-key
    solvers:
      - http01:
          ingress:
            serviceType: ClusterIP
            class: traefik-cert-manager

My certificate definition looks "raw" a bit different, because I try to use pulumi for that.

    new certManager.certmanager.v1.Certificate('website', {
        kind: 'Certificate',
        metadata: {
            name: `website-cert`,
            labels: appLabels
        },
        spec: {
            commonName: website.com,
            secretName: `website-cert`,
            dnsNames: ['website.com'],
            issuerRef: {
                name: 'letsencrypt-prod',
                kind: 'ClusterIssuer'
            }
        }
    }, { parent: ctx });

I can deploy and serve the webapp without any problems but the certificate gets never ready:

❯ kubectl.exe get certificate
NAME          READY   SECRET        AGE
website-cert   False   website-cert   13m

When I describe the challenge I get:

Reason:      Waiting for HTTP-01 challenge propagation: did not get expected response when querying endpoint, expected "3HhDbE........" but got: <!DOCTYPE html><html lang......
matthiasbaldi picture matthiasbaldi on 27 Oct 2020

@matthiasbaldi seems some HTML page is server on your challenge... worth checking your ingress setup

meyskens picture meyskens on 28 Oct 2020

My setup worked after I added DNS record for test.example.com pointing to Ingress controller ELB.

Roydon picture Roydon on 28 Oct 2020
👍1

@meyskens thanks a lot for the hint.
I found it. You were so right. I am pretty sure that I set this argument a few days a ago, may I overwrote it somehow.
For traefik I had to set the following argument: - --providers.kubernetesIngress.ingressClass=traefik-cert-manager
Otherwise the ingress was not matching with the configuration for the ClusterIssuer.

Thank you 👍

matthiasbaldi picture matthiasbaldi on 28 Oct 2020
👍1

@meyskens thanks a lot for the hint.
I found it. You were so right. I am pretty sure that I set this argument a few days a ago, may I overwrote it somehow.
For traefik I had to set the following argument: - --providers.kubernetesIngress.ingressClass=traefik-cert-manager
Otherwise the ingress was not matching with the configuration for the ClusterIssuer.

Thank you 👍

@matthiasbaldi where did you add that line of argument? thx

koo9 picture koo9 on 28 Dec 2020

@koo9 I found it here: https://www.scaleway.com/en/docs/how-to-setup-traefikv2-and-cert-manager-on-kapsule/#-Deploying-Cert-Manager

I edited the traefik with something like kubectl edit ds traefik -n kube-system

matthiasbaldi picture matthiasbaldi on 30 Dec 2020

@matthiasbaldi excellent link. thx!

koo9 picture koo9 on 30 Dec 2020
👍1

If I have to supply ingressClass in config file (traefik.toml) can I do this way ?

    [providers]
      [providers.kubernetesCRD]
      [providers.file]
      [providers.kubernetesIngress]
        ingressClass = "traefik-cert-manager"
Roydon picture Roydon on 31 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

munnerz picture munnerz  ·  4Comments
jonathan-kosgei picture jonathan-kosgei  ·  4Comments
f-f picture f-f  ·  4Comments
apetheriotis picture apetheriotis  ·  3Comments
matthew-muscat picture matthew-muscat  ·  4Comments