Cert-manager: Most of the metrics are missing

[philipp1992]

Hi,

ive deployed cert-manager:v0.15.0-alpha.0 with helm and enabled metrics.

There is only one metric exposed:

root@vmd51148:/# curl 10.43.29.15:9402/metrics
# HELP certmanager_controller_sync_call_count The number of sync() calls made by a controller.
# TYPE certmanager_controller_sync_call_count counter
certmanager_controller_sync_call_count{controller="clusterissuers"} 2

[meyskens]

Can you get us more information about your setup like Kubernetes version,cloud, etc.
As well as if this is a fresh install or if it is managing several certificates.
Thanks!

/triage needs-information

[Zempashi]

Hi,
Seems that I'm also affected by this bug:

$ curl http://10.128.2.116:9402/metrics
# HELP certmanager_controller_sync_call_count The number of sync() calls made by a controller.                                
# TYPE certmanager_controller_sync_call_count counter
certmanager_controller_sync_call_count{controller="certificaterequests"} 100
certmanager_controller_sync_call_count{controller="clusterissuers"} 4
certmanager_controller_sync_call_count{controller="ingress-shim"} 20
certmanager_controller_sync_call_count{controller="issuers"} 1          

My setup:

• openshift 4.3.3
• cert-manager 0.15
• by oc-applying https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml and adding --feature-gates=ExperimentalCertificateControllers=true

[Zempashi]

@meyskens Just forget to mention that logs look normal:

I0513 12:34:42.186093       1 feature_gate.go:243] feature gates: &{map[ExperimentalCertificateControllers:true]}
I0513 12:34:42.186221       1 start.go:76] cert-manager "msg"="starting controller"  "git-commit"="1d6ecc9cf8d841782acb5f3d3c28467c24c5fd18" "version"="v0.15.0"
W0513 12:34:42.186254       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0513 12:34:42.189976       1 controller.go:193] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["172.30.0.10:53"]
I0513 12:34:42.190393       1 controller.go:94] cert-manager/controller "msg"="Disabling old certificates controller"
I0513 12:34:42.190408       1 controller.go:97] cert-manager/controller "msg"="Enabling all experimental certificates controllers"
I0513 12:34:42.190424       1 controller.go:156] cert-manager/controller "msg"="starting leader election"
I0513 12:34:42.191690       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-controller...
I0513 12:34:42.192811       1 metrics.go:202] cert-manager/metrics "msg"="listening for connections on" "address"="0.0.0.0:9402"
I0513 12:35:50.960820       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-controller
I0513 12:35:50.962109       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="CertificateRequestManager"
I0513 12:35:50.962136       1 controller.go:89] cert-manager/controller/CertificateRequestManager "msg"="starting control loop"
I0513 12:35:50.962165       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="ca"
I0513 12:35:50.962532       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="selfsigned"
I0513 12:35:50.962614       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="vault"
I0513 12:35:50.963996       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="issuers"
I0513 12:35:50.964021       1 controller.go:89] cert-manager/controller/issuers "msg"="starting control loop"
I0513 12:35:50.964067       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca"
I0513 12:35:50.964085       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-ca "msg"="starting control loop"
I0513 12:35:50.964140       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim"
I0513 12:35:50.964160       1 controller.go:89] cert-manager/controller/ingress-shim "msg"="starting control loop"
I0513 12:35:50.964211       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="CertificateReadiness"
I0513 12:35:50.964220       1 controller.go:89] cert-manager/controller/CertificateReadiness "msg"="starting control loop"
I0513 12:35:50.964773       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned"
I0513 12:35:50.964795       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="starting control loop"
I0513 12:35:50.965450       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault"
I0513 12:35:50.965470       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-vault "msg"="starting control loop"
I0513 12:35:50.965489       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers"
I0513 12:35:50.965499       1 controller.go:89] cert-manager/controller/clusterissuers "msg"="starting control loop"
I0513 12:35:50.967069       1 reflector.go:175] Starting reflector *v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.767268       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="acme"
I0513 12:35:51.767418       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="venafi"
I0513 12:35:51.767437       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificates"
I0513 12:35:51.767550       1 reflector.go:175] Starting reflector *v1beta1.Ingress (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.767648       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="challenges"
I0513 12:35:51.767670       1 controller.go:89] cert-manager/controller/challenges "msg"="starting control loop"
I0513 12:35:51.767692       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="orders"
I0513 12:35:51.767702       1 controller.go:89] cert-manager/controller/orders "msg"="starting control loop"
I0513 12:35:51.767729       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme"
I0513 12:35:51.767737       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-acme "msg"="starting control loop"
I0513 12:35:51.767761       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="CertificateKeyManager"
I0513 12:35:51.767769       1 controller.go:89] cert-manager/controller/CertificateKeyManager "msg"="starting control loop"
I0513 12:35:51.767789       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="CertificateTrigger"
I0513 12:35:51.767799       1 controller.go:89] cert-manager/controller/CertificateTrigger "msg"="starting control loop"
I0513 12:35:51.767932       1 reflector.go:175] Starting reflector *v1alpha2.Challenge (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.767946       1 reflector.go:175] Starting reflector *v1alpha2.Order (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.768056       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.768186       1 reflector.go:175] Starting reflector *v1.Pod (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.768283       1 reflector.go:175] Starting reflector *v1.Service (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.768370       1 reflector.go:175] Starting reflector *v1.Secret (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.768593       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi"
I0513 12:35:51.768608       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-venafi "msg"="starting control loop"
I0513 12:35:51.768638       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="CertificateIssuing"
I0513 12:35:51.768646       1 controller.go:89] cert-manager/controller/CertificateIssuing "msg"="starting control loop"
I0513 12:35:51.768680       1 reflector.go:175] Starting reflector *v1alpha2.CertificateRequest (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.768687       1 reflector.go:175] Starting reflector *v1alpha2.Issuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.768789       1 reflector.go:175] Starting reflector *v1alpha2.ClusterIssuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0513 12:35:51.864196       1 controller.go:141] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="cert-manager/temporary-ca-djbsp"
I0513 12:35:51.864253       1 controller.go:147] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="cert-manager/temporary-ca-djbsp"
I0513 12:35:51.864266       1 controller.go:141] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="opera-poc-cert-manager/test-server-cert-2201107519"
I0513 12:35:51.864279       1 controller.go:147] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="opera-poc-cert-manager/test-server-cert-2201107519"
I0513 12:35:51.864289       1 controller.go:141] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="testing-helm-basic/client-certificate-bglxs"
I0513 12:35:51.864300       1 controller.go:147] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="testing-helm-basic/client-certificate-bglxs"

and also that I have tested (successfully) a pcks12 certificate that now lives in the cluster

[radarnex]

The issue happens when you enable the feature gate ExperimentalCertificateControllers. We have not tested v0.16 yet.

[dhiman-git]

@radarnex I am using cert-manager 0.15.1 and facing the same issue as stated above .Are you suggesting to make it "ExperimentalCertificateControllers= false" . Will work fine ??

[radarnex]

@dhiman-git Yes, if you're running the controller with the feature gate enabled just remove it, by default, it's set to false

[meyskens]

Please try 0.16, we improved it a lot in https://github.com/jetstack/cert-manager/pull/2923

[wallrj]

@radarnex @dhiman-git @philipp1992 Hopefully you've updated to a newer version of cert-manager by now. One which has the new metrics controller mentioned by @meyskens .
I'll close this, but please create new issues if you encounter issues with the metrics in recent versions of cert-manager.

/close

[jetstack-bot]

@wallrj: Closing this issue.

In response to this:

@radarnex @dhiman-git @philipp1992 Hopefully you've updated to a newer version of cert-manager by now. One which has the new metrics controller mentioned by @meyskens .
I'll close this, but please create new issues if you encounter issues with the metrics in recent versions of cert-manager.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.