Cert-manager: Certificate request stuck in InProgress. v0.11.0

Created on 12 Oct 2019 · 7Comments · Source: jetstack/cert-manager

I just installed cert-manager v0.11.0 and created a clusterissuer. However certificate requests are stuck in InProgress status. It's been 8 hours now and still no valid certificate. How long does it usually take for certificate request to complete?

Thanks

ClusterIssuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cert-manager.io/v1alpha2","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-staging"},"spec":{"acme":{"email":"[email protected]","privateKeySecretRef":{"name":"letsencrypt-staging"},"server":"https://acme-staging-v02.api.letsencrypt.org/directory","solvers":[{"dns01":{"clouddns":{"project":"my_gcp_project","serviceAccountSecretRef":{"key":"clouddns-dns01-solver.json","name":"clouddns-dns01-solver-sa"}}},"selector":{"dnsNames":["*.mydomain.com","mydomain.com"]}}]}}}
  creationTimestamp: "2019-10-12T09:29:11Z"
  generation: 1
  name: letsencrypt-staging
  resourceVersion: "2819405"
  selfLink: /apis/cert-manager.io/v1alpha2/clusterissuers/letsencrypt-staging
  uid: bfedfc69-ecd2-11e9-9178-42010a800013
spec:
  acme:
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        clouddns:
          project: my_gcp_project
          serviceAccountSecretRef:
            key: clouddns-dns01-solver.json
            name: clouddns-dns01-solver-sa
      selector:
        dnsNames:
        - '*.mydomain.com'
        - mydomain.com
status:
  acme:
    lastRegisteredEmail: [email protected]
    uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/11314502
  conditions:
  - lastTransitionTime: "2019-10-12T09:29:12Z"
    message: The ACME account was registered with the ACME server
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

Certificate:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  creationTimestamp: "2019-10-12T09:29:16Z"
  generation: 1
  labels:
    app: sonatype-nexus
    chart: sonatype-nexus-1.19.0
    fullname: nexus-sonatype-nexus
    heritage: Tiller
    release: nexus
  name: nexus-tls
  namespace: nexus
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: nexus-sonatype-nexus
    uid: c29b9916-ecd2-11e9-9178-42010a800013
  resourceVersion: "2819546"
  selfLink: /apis/cert-manager.io/v1alpha2/namespaces/nexus/certificates/nexus-tls
  uid: c2b11592-ecd2-11e9-9178-42010a800013
spec:
  dnsNames:
  - nexus.mydomain.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-staging
  secretName: nexus-tls
status:
  conditions:
  - lastTransitionTime: "2019-10-12T09:29:16Z"
    message: Waiting for CertificateRequest "nexus-tls-2163909225" to complete
    reason: InProgress
    status: "False"
    type: Ready

CertificateRequest:

apiVersion: cert-manager.io/v1alpha2
kind: CertificateRequest
metadata:
  annotations:
    cert-manager.io/certificate-name: nexus-tls
    cert-manager.io/private-key-secret-name: nexus-tls
  creationTimestamp: "2019-10-12T09:29:16Z"
  generation: 1
  labels:
    app: sonatype-nexus
    chart: sonatype-nexus-1.19.0
    fullname: nexus-sonatype-nexus
    heritage: Tiller
    release: nexus
  name: nexus-tls-2163909225
  namespace: nexus
  ownerReferences:
  - apiVersion: cert-manager.io/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: nexus-tls
    uid: c2b11592-ecd2-11e9-9178-42010a800013
  resourceVersion: "2819555"
  selfLink: /apis/cert-manager.io/v1alpha2/namespaces/nexus/certificaterequests/nexus-tls-2163909225
  uid: c2f259cf-ecd2-11e9-9178-42010a800013
spec:
  csr: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2pUQ0NBWFVDQVFBd0Z6RVZNQk1HQTFVRUNoTU1Z1lZ5ZEMxdFlXNWhaMlZ5TUlJQklqQU5CZ2txaGtpRwo5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBclNpOWdmNXMxRldFL0ZkOHY5U3o4ZkVJV1FjMGVsYmdnUk9YCmlHbllNTjJzQ0o1Q0p1a2JHdGUwTVRMaU1FYk9IT3R3ZTVRdHdDVWV0NUhPTElBcmdlWVJiUjNVSmtQTG9qaHUKWUxkZGlON0ZOZStQWjhZcitFcnRMK0tya2E3NjdxaWhhZGFZSWtJNnY4dWdReVJiV1JDbnltb05PUXEwaDhDeQoydThwQXdwTHZrU0FaVFRRcgZMeklQV3FQWE1URUdkemtIeVoyREtWc3ZZWFk3Z0pZSEcySE1NN28wTU5FZm9YCm9KNEpRMFpVT3QxcHdCRnZJVzVWclM1SDZtVmVFQlRDWk5MNDhuQ3NSYnQrOHlEU24xaVFlYllYeTh3dE9uVXQKVGJ0QnV4NkIyeVBOMVlzT1hkYU82SDFZMkdwbDd6YlhHOXdIam5qeDR6MEQxN2x3M3dJREFRQUJvREV3THdZSgpLb1pJaHZjTkFRa09NU0l3SURBZUJnTlZIUkVFRnpBVmdoTnVaWGgxY3k1amFXTmtMbXAxZFd3dVkyOXRNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCVDlLSmhpT0lRYSVTNWJUTWhZMG5lbVJiS3c2RW9nS01aVFdkZmFXRUsKeUZVNWF2cy9seXovMnBhTG1FT29tUlo0TEhnVWlTRHgvemliNHdBc0ZVVUppMTJvQmpkYnpRS1RBd0NodXVmYgpoTmRrbmRvK2JzMG9FbnpRL1JwQ2U2NkdFby9nL1d4YlhCQThTL2RpTHRuMEtkNzlXbENnRk44dnBXM2VtZGVfCm9INUdVKzVQYkZ2dVk4RWtyMERMNTFhcFF2NUxkblEzRkZwTUpnRVFTQ3hkYVFld2d6bzRsdC9UM2FjVWxEajEKUFJLZUQ2MW5aR0ErK3ROMVNnbzJjcnY5VmZZbzZuV21Jc0c5NWYvaVFqWjlpOGZlSzbreGNhS0RDK3JxQVd2TwpQYXIrdjYrMkZVblBzbE0wTGNudTUzRnZmYjUzRklQc1FsMHZIQUk2ZnpjTQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-staging
status:
  conditions:
  - lastTransitionTime: "2019-10-12T09:29:16Z"
    message: 'Waiting on certificate issuance from order nexus/nexus-tls-2163909225-1210362042:
      "pending"'
    reason: Pending
    status: "False"
    type: Ready

Secret:

apiVersion: v1
data:
  ca.crt: ""
  tls.crt: ""
  tls.key: <REMOVED>
kind: Secret
metadata:
  annotations:
    cert-manager.io/certificate-name: nexus-tls
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: letsencrypt-staging
  creationTimestamp: "2019-10-12T09:29:16Z"
  name: nexus-tls
  namespace: nexus
  ownerReferences:
  - apiVersion: cert-manager.io/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: nexus-tls
    uid: c2b11592-ecd2-11e9-9178-42010a800013
  resourceVersion: "2819537"
  selfLink: /api/v1/namespaces/nexus/secrets/nexus-tls
  uid: c2e592a3-ecd2-11e9-9178-42010a800013
type: kubernetes.io/tls

Logs:

I1012 09:29:17.944580       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="nexus/nexus-tls" 
I1012 09:29:17.944835       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="syncing item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.944938       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="finished processing work item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.944997       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.945077       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="finished processing work item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.945116       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="syncing item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.945130       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="syncing item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.945197       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="finished processing work item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.945326       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="finished processing work item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.945657       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="nexus-tls-2163909225" "related_resource_namespace"="nexus" "resource_kind"="Certificate" "resource_name"="nexus-tls" "resource_namespace"="nexus" 
I1012 09:29:17.945893       1 sync.go:479] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="nexus-tls-2163909225" "related_resource_namespace"="nexus" "resource_kind"="Certificate" "resource_name"="nexus-tls" "resource_namespace"="nexus" "state"="Pending"
E1012 09:29:17.946162       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="nexus-tls" "related_resource_namespace"="nexus" "resource_kind"="Certificate" "resource_name"="nexus-tls" "resource_namespace"="nexus" "secret_key"="tls.crt" 
I1012 09:29:17.946361       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="nexus/nexus-tls" 
I1012 09:29:17.946807       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="finished processing work item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.946999       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="syncing item" "key"="nexus/nexus-tls-2163909225" 
I1012 09:29:17.947345       1 acme.go:178] cert-manager/controller/certificaterequests-issuer-acme/sign "level"=0 "msg"="acme Order resource is not in a ready state, waiting..." "related_resource_kind"="Order" "related_resource_name"="nexus-tls-2163909225-1210362042" "related_resource_namespace"="nexus" "resource_kind"="CertificateRequest" "resource_name"="nexus-tls-2163909225" "resource_namespace"="nexus" 
I1012 09:29:17.947522       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="finished processing work item" "key"="nexus/nexus-tls-2163909225"

Source

angelosanramon

👍39

Most helpful comment

Sorry for late responding. I resolved the issue. The order status was showing something about no valid solvers. I set the selector to selector: {} in my clusterissuer and that solved the issue.

Thanks for responding.

angelosanramon on 19 Oct 2019

👍12

All 7 comments

Hi, i have the same problem with issuer. I have installed via helm with GKE version 1.13

RafaelAybar on 15 Oct 2019

Hi there!

E1012 09:29:17.946162 1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="nexus-tls" "related_resource_namespace"="nexus" "resource_kind"="Certificate" "resource_name"="nexus-tls" "resource_namespace"="nexus" "secret_key"="tls.crt"

This seems odd that the returned certificate has not been able to be decoded.

Would you be able to share the status of the Order and Challenge resource related to this request?

As an aside, please don't share your private keys publicly.

JoshVanL on 15 Oct 2019

Sorry for late responding. I resolved the issue. The order status was showing something about no valid solvers. I set the selector to selector: {} in my clusterissuer and that solved the issue.

Thanks for responding.

angelosanramon on 19 Oct 2019

👍12

@angelosanramon hi, I have the same problem. Could you please show your clusterissuer yaml file.

zhanglangbravo on 24 Dec 2019

@angelosanramon hi, I have the same problem. Could you please show your clusterissuer yaml file.

@zhanglangbravo probably because the http resolver moved under solvers (I myself upgraded from 0.50). See:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
      - http01:
          ingress:
            class: nginx

This works for me :)

lebenitza on 24 Dec 2019

👍3

@angelosanramon hi, I have the same problem. Could you please show your clusterissuer yaml file.

@zhanglangbravo probably because the http resolver moved under solvers (I myself upgraded from 0.50). See:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
      - http01:
          ingress:
            class: nginx

This works for me :)

thank you a lot. It works for me too. I delete those secret whose certificates status was false and it rebuild automatically in normal status.

zhanglangbravo on 25 Dec 2019

this helps, thank you