cert-manager-webhook increasing CPU usage

[neszt]

Describe the bug:
Increasing CPU usage.

Expected behaviour:
Even CPU usage.

Steps to reproduce the bug:
Start the pod, watch CPU load.

Anything else we need to know?:
We are experiencing ever increasing cpu load in the webhook component. I've attached a graph of about 2 weeks:

When the pod is restarted, the CPU load will start to rise again from a low value.

Environment details::

• Kubernetes version (e.g. v1.10.2): v1.16.3
• Cloud-provider/provisioner: Own / Kubeadm.
• cert-manager version: v0.12.0
• Install method: static manifests

/kind bug

[rkojedzinszky]

I am also experiencing this issue, you can have a look at the charts:

https://grafana.adm.kojedz.in/dashboard/snapshot/mb4zqFrkZ7I4SWrXwVjdQKwEN336ZUF4

I guess it started when I upgraded from 0.11.0 to 0.12.0.

[rombert]

Similar issue here, I made an initial deployment with v0.12.0-beta.1 about two weeks ago.

[kkwojc]

The same issue with cert-manager-webhook on my cluster:

CPU usage is growing to pod deletion.
Issue started after upgrade to v0.12.0. In v0.10.0 this problem did not occur

Kubernetes version: v1.15.4
Cloud-provider/provisioner: Own / Kubeadm
cert-manager version: v0.12.0
Install method: helm chart

[liujianping]

same issue when I install cert-manager version: v0.12.0 by helm method, so I changed to use Installing with regular manifests, cpu usage became normal.

[urtho]

Same here. Running 0.12 from manifests on GKE v1.14.

[munnerz]

v0.12 includes a fairly significant refactoring of the webhook component to be a lot simpler internally. I am able to replicate this same behaviour on my own cluster when running over a few days.

2450 has added pprof debug endpoints to the webhook component which upon profiling, seem to indicate the majority of this CPU time is spent handling mutexes. I've attached a copy of a profile that I ran for 2 minutes after having run the webhook for a few days (and observing its CPU usage increasing, graph also attached).

I'm continuing to dig into this, although am struggling to trace where this is coming from so if anyone else has experience profiling Go apps, I'd really appreciate your input!

profile002.pdf

[munnerz]

Scratch that, I think I may have just found the issue! https://github.com/jetstack/cert-manager/pull/2467 should fix it 🤞

[Kugelschieber]

I've installed 0.12 through Helm after the 16th of december. Helm hub shows 0.12 has been released on 28th of november, so it does not include this bugfix. Can you publish a new release to the hub? 0.12.1 maybe?