Cert-manager: Error Decoding x509 Certificate

[kendricktan]

Describe the bug:
The Certificate (which is automatically generated by ingress-shim) is having issues decoding x509 certificates.

cert-manager.logs

I0829 11:00:41.315362       1 logger.go:38] Calling CreateOrder
E0829 11:00:41.315406       1 pki.go:128] cert-manager/controller/certificates/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="childchain-stress-test-cloud-tls" "related_resource_namespace"="default" "secret_key"="tls.crt" 
I0829 11:00:41.315675       1 issue.go:169] cert-manager/controller/certificates/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="childchain-stress-test-cloud-tls-2580951430" "related_resource_namespace"="default" 
E0829 11:00:41.315719       1 pki.go:91] cert-manager/metrics "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="childchain-stress-test-cloud-tls" "related_resource_namespace"="default" "secret_key"="tls.crt" 
I0829 11:00:41.319303       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/childchain-stress-test-cloud-tls" 
E0829 11:00:41.551677       1 base_controller.go:189] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="error creating new order: acme: urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" "key"="default/childchain-stress-test-cloud-tls-2580951430" 
I0829 11:00:41.693989       1 base_controller.go:187] cert-manager/controller/orders "level"=0 "msg"="syncing item" "key"="default/childchain-stress-test-cloud-tls-2580951430" 
E0829 11:00:41.694081       1 controller.go:139] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.certmanager.k8s.io \"childchain-stress-test-cloud-tls-2580951430\" not found"  
I0829 11:00:41.694104       1 base_controller.go:193] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="default/childchain-stress-test-cloud-tls-2580951430" 
I0829 11:00:41.693989       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/childchain-stress-test-cloud-tls" 
E0829 11:00:41.694289       1 pki.go:128] cert-manager/controller/certificates/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="childchain-stress-test-cloud-tls" "related_resource_namespace"="default" "secret_key"="tls.crt" 
E0829 11:00:41.704102       1 pki.go:91] cert-manager/metrics "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="childchain-stress-test-cloud-tls" "related_resource_namespace"="default" "secret_key"="tls.crt" 
I0829 11:00:41.704144       1 base_controller.go:187] cert-manager/controller/orders "level"=0 "msg"="syncing item" "key"="default/childchain-stress-test-cloud-tls-2580951430" 
I0829 11:00:41.704272       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/childchain-stress-test-cloud-tls" 
I0829 11:00:41.704318       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="default/childchain-stress-test-cloud-tls" 
I0829 11:00:41.704552       1 sync.go:89] cert-manager/controller/orders "level"=0 "msg"="creating Order with ACME server as one does not currently exist" "resource_kind"="Order" "resource_name"="childchain-stress-test-cloud-tls-2580951430" "resource_namespace"="default" 
E0829 11:00:41.704595       1 pki.go:128] cert-manager/controller/certificates/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="childchain-stress-test-cloud-tls" "related_resource_namespace"="default" "secret_key"="tls.crt" 
I0829 11:00:41.704598       1 sync.go:376] cert-manager/controller/orders "level"=0 "msg"="order URL not set, submitting Order to ACME server" "resource_kind"="Order" "resource_name"="childchain-stress-test-cloud-tls-2580951430" "resource_namespace"="default" 
I0829 11:00:41.704819       1 sync.go:382] cert-manager/controller/orders "level"=0 "msg"="build set of domains for Order" "resource_kind"="Order" "resource_name"="childchain-stress-test-cloud-tls-2580951430" "resource_namespace"="default" "domains"=["childchain-stress-test.omg.network"]
I0829 11:00:41.704848       1 logger.go:38] Calling CreateOrder
I0829 11:00:41.705099       1 issue.go:169] cert-manager/controller/certificates/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="childchain-stress-test-cloud-tls-2580951430" "related_resource_namespace"="default" 
E0829 11:00:41.705184       1 pki.go:91] cert-manager/metrics "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="childchain-stress-test-cloud-tls" "related_resource_namespace"="default" "secret_key"="tls.crt" 
I0829 11:00:41.705303       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/childchain-stress-test-cloud-tls"

Expected behaviour:
The certificate is decoded successfully.

Steps to reproduce the bug:
I followed the quickstart - https://docs.cert-manager.io/en/latest/tasks/issuing-certificates/ingress-shim.html

Anything else we need to know?:

Environment details::

• Kubernetes version (e.g. v1.10.2):

Client Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.9-gke.7", GitCommit:"b6001a5d99c235723fc19342d347eee4394f2005", GitTreeState:"clean", BuildDate:"2019-06-24T19:37:31Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.8-gke.10", GitCommit:"f53039cc1e5295eed20969a4f10fb6ad99461e37", GitTreeState:"clean", BuildDate:"2019-06-19T20:48:40Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
• cert-manager version (e.g. v0.4.0): v0.9.1
• Install method (e.g. helm or static manifests): static manifests

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
    ingress.kubernetes.io/backends: <redacted>
    ingress.kubernetes.io/forwarding-rule: <redacted>
    ingress.kubernetes.io/target-proxy: <redacted>
    ingress.kubernetes.io/url-map: <redacted>
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.global-static-ip-name: <redacted>
    kubernetes.io/tls-acme: "true"
  creationTimestamp: 2019-08-27T07:46:45Z
  generation: 13
  labels:
    component: test-ingress
    heritage: Tiller
    release: stress-test
  name: stress-test
  namespace: default
spec:
  rules:
  - host: test.domain.com
    http:
      paths:
      - backend:
          serviceName: test
          servicePort: 80
        path: /*
  tls:
  - hosts:
    - subdomain.domain.com
    secretName: test-cloud-tls
status:
  loadBalancer:
    ingress:
    - ip: <redacted>

clusterissuer.yaml

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  annotations:
  creationTimestamp: 2019-08-28T09:05:34Z
  generation: 1
  name: letsencrypt-staging
  resourceVersion: "16092254"
spec:
  acme:
    email: <redacted>
    http01: {}
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
status:
  acme:
    lastRegisteredEmail: [email protected]
    uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/10776649
  conditions:
  - lastTransitionTime: 2019-08-28T09:05:35Z
    message: The ACME account was registered with the ACME server
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

/kind bug

[munnerz]

This is a benign log message and only happens because the Certificate has not been issued yet - the order is still in progress and you should see certificates issued shortly afterwards providing you've configured your cluster correctly.

If you are having issues actually obtaining certificates, can you join the #cert-manager channel on slack.k8s.io as there's a larger body of people there able to help debug/dig in, and we find interactive back-and-forth helps uncover issues a lot quicker 😄

[arianitu]

@munnerz Hello, struggling to generate wildcard certs, config below:

Certificate:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: darwincontent
  namespace: ingress-nginx
  labels:
    use-gcloud-solver: "true"  
spec:
  secretName: wildcard-cert
  issuerRef:
    name: letsencrypt-production
  commonName: '*.domain.com'
  dnsNames:
  - '*.domain.com'

ClusterIssuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-prod

    solvers:
    - selector: {}
      http01:
        ingress:
          class: nginx

    - selector:
        matchLabels:
          use-gcloud-solver: "true"

      dns01:
        clouddns:
          # The ID of the GCP project
          # reference: https://docs.cert-manager.io/en/latest/tasks/issuers/setup-acme/dns01/google.html
          project: 'cloud-managed'

          # This is the secret used to access the service account
          serviceAccountSecretRef:
            name: gcloud-dns-service-account
            key: key.json    

Results:

Stuck on :

Normal  Requested     8m23s  cert-manager  Created new CertificateRequest resource

Logs:

E1018 01:52:23.885758       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="darwincontent-cert" "related_resource_namespace"="ingress-nginx" "resource_kind"="Certificate" "resource_name"="darwincontent" "resource_namespace"="ingress-nginx" "secret_key"="tls.crt" 
I1018 01:52:23.886027       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="ingress-nginx/darwincontent" 
E1018 01:52:23.887055       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-venafi "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"darwincontent-3947332722\": the object has been modified; please apply your changes to the latest version and try again" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:23.887092       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:23.887176       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="finished processing work item" "key"="ingress-nginx/darwincontent-3947332722" 

I1018 01:52:28.875595       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="syncing item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:28.877219       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="finished processing work item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:28.883473       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="syncing item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:28.884491       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="finished processing work item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:28.884904       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="syncing item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:28.884999       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="finished processing work item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:28.888785       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="ingress-nginx/darwincontent-3947332722" 
I1018 01:52:28.888903       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="finished processing work item" "key"="ingress-nginx/darwincontent-3947332722" 

[manhvc]

Hello @arianitu ,

My error same u...
Can u fix this? Pls tell me how to fix this.

Thanks

[rush2subbu]

Hey folks,

Anybody had any luck finding a fix for this issue yet
please do update us,
thanks and regards,

[ivanovUA]

Hey folks,

Anybody had any luck finding a fix for this issue yet
please do update us,
thanks and regards,

I found issue in my Issuer config. There was changed apiVersion
from:
certmanager.k8s.io/v1alpha1
to:
cert-manager.io/v1alpha2
It solved my issue

[arianitu]

@manhvc @rush2subbu

Hello everyone, almost always the issue is a configuration issue on your end and not cert-manager, but cert-manager is horrible with error messages, the documentation probably to blame and the authors assuming we know how it works.

The way I solved this issue was realizing that errors don't only live in the cert, they also live in all the sub objects and you should check each of them.

For example, if you go to https://docs.cert-manager.io/en/latest/reference/certificates.html, notice on the left menu there are:

• Certificate
• CertificateRequests
• Orders
• Challenges,
  etc.

You can actually find error messages in each of these, like so:

kubectl get certificaterequest
kubectl describe certificaterequest X
kubectl get order
kubectl describe order X
kubectl get challenge
kubectl describe challenge X

If you do it for all of them, you will more than likely find the real cause of your issue. In my case, it was a config issue in the ingress where it was pointing to a ClusterIssuer that did not exist (because I named it incorrectly.)

I would have never known that unless I looked at the messages in the other objects since the error messages in cert-manager were just making me look in the wrong place. Like @munnerz said, the message above is just a benign message that is pointing you in the wrong direction.

cert-manager needs some serious changes in the way it handles errors, it's been horrible for years now and I think they just need to not log any message that isn't an error, or have a guide on how to debug a cert that isn't being generated.

[matfiz]

Actually, I had the exact same problem and the advice from @ivanovUA solved the problem. It was not crystal clear following the cert-manager docs ;-)

[vfiset]

same for me ^ 💯

[johan-lejdung]

Just to document what had happened to me..

I had created a new cluster and for some reason (probably because I've been using a NodePort service instead of LoadBalancer for the nginx-ingess) the firewall rules had not automatically been set. Make sure port 80 and 443 is open!

Figured it out embarrassingly late.. :)

[suda]

I had this issue when setting commonName to a wildcard but not including this wildcard in dnsNames field. I got to the root of the problem by describing the CertificateRequest:

$ kubectl describe certificaterequest.cert-manager.io

[hmacias]

In my case describing the certificate request as advised above by @arianitu pointed me in the right direction:

1. Run: kubectl get certificaterequest -n test

2. Using the output from previous command run: kubectl describe certificaterequest -n test wild-test-domain-example-2164310232

Events:
  Type    Reason          Age                From          Message
  ----    ------          ----               ----          -------
  Normal  IssuerNotFound  13m (x5 over 13m)  cert-manager  Referenced "Issuer" not found: issuer.cert-manager.io "letsencrypt-test" not found`

1. Since the CRD for certificates changed in the new cert-manager release. I had to describe it to see why the ClusterIssuer was not being found. I was clearly missing some piece of configuration. I did this running:
   kubectl describe crd certificaterequests.cert-manager.io

2. I learned that my CRD for my certificaterequests was missing the kind attribute under spec.issuerRef as in:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: wild-test-domain-example
  namespace: test
spec:
  secretName: wild-test-domain-example-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-test
  dnsNames:
  - 'test.com'

Thank you @arianitu for the pointers.

[Legym]

Spent a lot of time trying to resolve this issue.

This is what I have installed:

Rancher | v2.4.2
Helm | v2.16.3-rancher1
Cert Manager | v0.12.0 (helm ls --all-namespaces)

In my experience, I just needed to create an issuer that was valid. This is my Issuer yaml file. I was using Cloudflare. Make sure to use ClusterIssuer kind as it's available to all namespaces. Issuer is scoped to one name space

staging_issuer.yaml

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer 
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging

    solvers:
    - selector:
        dnsNames:
        - test.com
      dns01:
        cloudflare:
          email: [email protected]
          # !! Remember to create a Rancher secret before setting up 
          apiTokenSecretRef:
            name: cloudflare-api-key-secret // Secret name
            key: api-key // Secret key

Make sure your issuer is valid and working. You can deploy a new issuer with these commands

Create new: 
kubectl create --namespace=cert-manager -f staging_issuer.yaml

Edit an existing cluster issue yaml file: 
kubectl apply -f staging_issuer.yaml

Lastly in my ingress, this was my config.

Whenever the certificate is generated, it will show up in the SSL/TLS Certificates. It will be greyed out when you first configure your ingress.

Once the issuer and your ingress controller is configured, your ingress controller will validate certificates. That's when you can run @arianitu commands to debug what the issue is.

EDIT: I also needed to add secretName: XXX under TLS. You need to edit the YAML file for your ingress controller