Cert-manager: certificate never appears ready

[michaelburch]

kubectl get cert

NAME                   READY   SECRET                     AGE
example.domain.com   False   example-tls   11m

The certificate is actually working but the status is not updated.

Logs show this in a perpetual loop:

kubectl -n cert-manager logs -l app=cert-manager

I1115 23:55:41.521223       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example.domain.com-2545306439" "related_resource_namespace"="namespace1" "resource_kind"="Certificate" "resource_name"="example.domain.com" "resource_namespace"="namespace1"
I1115 23:55:41.720351       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="namespace1/example.domain.com"

[saltenhub]

I am having the same issue on v0.12 using DNS01 verification with acmeDNS and Istio v1.4. The endless loop described above is spamming our logs with about 30 lines/sec, even though the certificate is already working and valid:
I1202 17:42:08.180118 1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.180354 1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.180379 1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.180571 1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.180600 1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.372284 1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.373234 1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.374438 1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.374573 1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.374622 1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.374984 1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.375054 1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.577480 1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.577540 1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.580228 1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.580594 1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.580717 1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.581894 1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.582025 1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.775820 1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.775882 1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.777220 1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.777405 1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.777475 1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.777901 1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.777962 1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.972866 1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.972943 1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" I1202 17:42:08.974481 1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.974630 1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.974688 1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.975061 1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:08.975121 1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:09.177992 1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" I1202 17:42:09.178049 1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" I1202 17:42:09.179199 1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:09.179345 1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:09.179396 1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:09.179745 1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" I1202 17:42:09.179817 1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system"

EDIT: The issue seems to be that the controller is waiting for the CertificateRequest to complete, even though the CertificateRequest already states Certificate fetched from issuer successfully, but the Certificate itself states Waiting for CertificateRequest [...] to complete. If you need any additional info / log outputs etc. let me know, I am happy to help.

[hoangmnsd]

I am having the same issue with @saltenhub , any ideas?

[saltenhub]

Seems to be related to / same as https://github.com/jetstack/cert-manager/issues/2426

[saltenhub]

This issue still seems to exist for me on v0.13.0

[sdanbury]

Exists for me also in v0.13.0, deleting the problematic kubernetes secrets resulted in things working again. Although, for one of the secrets, I had to delete it two times for things to return back to normal.

Any information I could provide to help solve this one?

[wallrj]

Thanks for the bug report.
This was fixed in master by #2539 and in https://github.com/jetstack/cert-manager/releases/tag/v0.13.1 by #2543

Please try upgrading cert-manager and report back if you still see this issue.

The repeated log message are explained by @munnerz in https://github.com/jetstack/cert-manager/pull/2539#discussion_r370035634

/area acme
/close

[jetstack-bot]

@wallrj: Closing this issue.

In response to this:

Thanks for the bug report.
This was fixed in master by #2539 and in https://github.com/jetstack/cert-manager/releases/tag/v0.13.1 by #2543

Please try upgrading cert-manager and report back if you still see this issue.

The repeated log message are explained by @munnerz in https://github.com/jetstack/cert-manager/pull/2539#discussion_r370035634

/area acme
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[camille-rodriguez]

I am facing the same issue with the latest version (0.14). I have one certificate showing Ready:False while its CertificateRequest is Ready:True

[Romiko]

Hi,

I am having the same issue with "v0.15.1"
Allot of logs!

I0606 22:50:40.636983 1 sync.go:485] cert-manager/controller/certificates "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"

This is on minikube
minikube version
minikube version: v1.10.1
commit: 63ab801ac27e5742ae442ce36dff7877dcccb278

The CertificateRquest is successfull but the Certificate itself never updates to completed.
Here is my clusterissuer which did succeed.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: r@.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging-account-key
solvers:
- selector:
dnsZones:
- "A.com"
dns01:
route53:
region: ap-southeast-2a
accessKeyID: A*
secretAccessKeySecretRef:
name: staging-route53-credentials-secret
key: secret-access-key

[wallrj]

/reopen

[jetstack-bot]

@wallrj: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[meyskens]

Can you confirm you still see this in v0.16?

[saltenhub]

@meyskens Just updated to v0.16 a couple of minutes ago to check. So far it is looking good - no more log spam and as a bonus all certificates that had READY state on False (but have been working flawlessly) are now on True. Will keep monitoring it and update this post if anything changes.

[meyskens]

Will close this for now feel free to /reopen if you see this happening again

/close

[jetstack-bot]

@meyskens: Closing this issue.

In response to this:

Will close this for now feel free to /reopen if you see this happening again

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.