Cert-manager: Internal error occurred: failed calling webhook

Created on 15 May 2020  路  20Comments  路  Source: jetstack/cert-manager

When testing the installation, I get "Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: context deadline exceeded"

Versions:
Kubernetes: Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

OS: CentOS 7

Update: I have removed this environment and will not be able to test any fixes.

Cert Manager: v0.15.0

Installed from Helm using official documentation on a two-node cluster.

To reproduce:

  1. Using OpenNebula, create Kubernetes service as outlined (here)[https://docs.opennebula.io/appliances/service/kubernetes.html]
  2. Install Helm on machine that has access to cluster
  3. Follow (this)[https://cert-manager.io/docs/installation/kubernetes/] to install cert-manager.
  4. Test cert-manager as outlined in section "Verifying the installation" in installation guide.

Expected results:

  • Test passes

Additional Notes:

  • I already have an ingress controller running (HAProxy).
  • I think it may be an issue with the ca-injector (https is not being served because no certificate)
  • All pods are in running state

Logs (kubectl logs -l app=cert-manager -n cert-manager):

I0515 02:41:15.606858       1 reflector.go:175] Starting reflector *v1alpha2.CertificateRequest (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606974       1 reflector.go:175] Starting reflector *v1alpha2.Order (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.607980       1 reflector.go:175] Starting reflector *v1alpha2.ClusterIssuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.608285       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.798737       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="default/[redacted]" 
I0515 02:41:15.801614       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="default" 
I0515 02:41:15.801753       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="default/[redacted]" 
I0515 02:41:15.801852       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="ingress-controller/[redacted]" 
I0515 02:41:15.802295       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="ingress-controller" 
I0515 02:41:15.802427       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="ingress-controller/[redacted]

kubectl logs -n cert-manager cert-manager-7cb75cf6b4-wjndg:

I0515 02:40:08.138598       1 start.go:76] cert-manager "msg"="starting controller"  "git-commit"="1d6ecc9cf8d841782acb5f3d3c28467c24c5fd18" "version"="v0.15.0"
W0515 02:40:08.139801       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0515 02:40:08.151968       1 controller.go:193] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.96.0.10:53"] 
I0515 02:40:08.154500       1 controller.go:156] cert-manager/controller "msg"="starting leader election"  
I0515 02:40:08.155604       1 metrics.go:202] cert-manager/metrics "msg"="listening for connections on" "address"="0.0.0.0:9402" 
I0515 02:40:08.163373       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-controller...
I0515 02:41:15.266989       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-controller
I0515 02:41:15.282469       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I0515 02:41:15.282627       1 controller.go:89] cert-manager/controller/clusterissuers "msg"="starting control loop"  
I0515 02:41:15.282805       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I0515 02:41:15.282879       1 controller.go:89] cert-manager/controller/ingress-shim "msg"="starting control loop"  
I0515 02:41:15.283158       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="issuers" 
I0515 02:41:15.283233       1 controller.go:89] cert-manager/controller/issuers "msg"="starting control loop"  
I0515 02:41:15.295088       1 reflector.go:175] Starting reflector *v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.598703       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="vault"
I0515 02:41:15.599100       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateIssuing" 
I0515 02:41:15.599165       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateKeyManager" 
I0515 02:41:15.599212       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateRequestManager" 
I0515 02:41:15.599265       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateTrigger" 
I0515 02:41:15.599495       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="ca"
I0515 02:41:15.599758       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="selfsigned"
I0515 02:41:15.600138       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="venafi"
I0515 02:41:15.600471       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateReadiness" 
I0515 02:41:15.600748       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="acme"
I0515 02:41:15.601283       1 reflector.go:175] Starting reflector *v1.Service (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.602575       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I0515 02:41:15.602640       1 controller.go:89] cert-manager/controller/challenges "msg"="starting control loop"  
I0515 02:41:15.602754       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I0515 02:41:15.602800       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-vault "msg"="starting control loop"  
I0515 02:41:15.603111       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I0515 02:41:15.603201       1 controller.go:89] cert-manager/controller/orders "msg"="starting control loop"  
I0515 02:41:15.603371       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I0515 02:41:15.603452       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-ca "msg"="starting control loop"  
I0515 02:41:15.603737       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" 
I0515 02:41:15.603832       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="starting control loop"  
I0515 02:41:15.603990       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" 
I0515 02:41:15.613745       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-venafi "msg"="starting control loop"  
I0515 02:41:15.614747       1 reflector.go:175] Starting reflector *v1.Secret (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.620672       1 reflector.go:175] Starting reflector *v1beta1.Ingress (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.623644       1 reflector.go:175] Starting reflector *v1.Pod (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606219       1 reflector.go:175] Starting reflector *v1alpha2.Issuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606446       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificates" 
I0515 02:41:15.630209       1 controller.go:89] cert-manager/controller/certificates "msg"="starting control loop"  
I0515 02:41:15.606552       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" 
I0515 02:41:15.630508       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-acme "msg"="starting control loop"  
I0515 02:41:15.606730       1 reflector.go:175] Starting reflector *v1alpha2.Challenge (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606858       1 reflector.go:175] Starting reflector *v1alpha2.CertificateRequest (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606974       1 reflector.go:175] Starting reflector *v1alpha2.Order (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.607980       1 reflector.go:175] Starting reflector *v1alpha2.ClusterIssuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.608285       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.798737       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="default/[redacted]" 
I0515 02:41:15.801614       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="default" 
I0515 02:41:15.801753       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="default/[redacted]" 
I0515 02:41:15.801852       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="ingress-controller/[redacted]" 
I0515 02:41:15.802295       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="ingress-controller" 
I0515 02:41:15.802427       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="ingress-controller/[redacted]"

kubectl logs -n cert-manager cert-manager-cainjector-759496659c-6sgkj:

I0515 02:40:11.989165       1 start.go:82] starting ca-injector v0.15.0 (revision 1d6ecc9cf8d841782acb5f3d3c28467c24c5fd18)
I0515 02:40:13.862263       1 request.go:621] Throttling request took 1.040883608s, request: GET:https://10.96.0.1:443/apis/authentication.k8s.io/v1beta1?timeout=32s
I0515 02:40:14.862445       1 request.go:621] Throttling request took 2.040553795s, request: GET:https://10.96.0.1:443/apis/policy/v1beta1?timeout=32s
I0515 02:40:14.927206       1 setup.go:81] cert-manager "msg"="unable to register injector which is still in an alpha phase. Enable the feature on the API server in order to use this injector"  "injector"="auditsink"
I0515 02:40:14.927500       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-cainjector-leader-election-core...
I0515 02:40:14.928256       1 reflector.go:175] Starting reflector *v1beta1.MutatingWebhookConfiguration (9h46m26.488119031s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:14.929974       1 reflector.go:175] Starting reflector *v1beta1.ValidatingWebhookConfiguration (10h4m16.134207791s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:14.931292       1 reflector.go:175] Starting reflector *v1beta1.APIService (10h5m42.401194473s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:14.931532       1 reflector.go:175] Starting reflector *v1beta1.CustomResourceDefinition (9h35m47.994364568s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.013269       1 setup.go:81] cert-manager "msg"="unable to register injector which is still in an alpha phase. Enable the feature on the API server in order to use this injector"  "injector"="auditsink"
I0515 02:40:15.013463       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-cainjector-leader-election...
I0515 02:40:15.014502       1 reflector.go:175] Starting reflector *v1beta1.MutatingWebhookConfiguration (9h13m40.588931011s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.016105       1 reflector.go:175] Starting reflector *v1beta1.ValidatingWebhookConfiguration (9h30m32.361430045s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.017189       1 reflector.go:175] Starting reflector *v1beta1.APIService (9h51m38.977119063s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.017855       1 reflector.go:175] Starting reflector *v1beta1.CustomResourceDefinition (9h54m34.145270079s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:30.600083       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-cainjector-leader-election-core
I0515 02:40:30.603665       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"group":"","names":{"plural":"","kind":""},"scope":""},"status":{"conditions":null,"acceptedNames":{"plural":"","kind":""},"storedVersions":null}}}
I0515 02:40:30.606195       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.607488       1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal"  "message"="cert-manager-cainjector-759496659c-6sgkj_acd9b7ea-feba-4e4b-bccf-260937b09f4c became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election-core","uid":"4a7550a8-39e2-4576-bccb-dd302185a1ae","apiVersion":"v1","resourceVersion":"702208"} "reason"="LeaderElection"
I0515 02:40:30.607694       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.608466       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.609639       1 reflector.go:175] Starting reflector *v1.Secret (9h52m51.343450322s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:30.609874       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"groupPriorityMinimum":0,"versionPriority":0},"status":{}}}
I0515 02:40:30.610176       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.610339       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.614417       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.909916       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="customresourcedefinition"
I0515 02:40:30.910146       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="customresourcedefinition" "worker count"=1
I0515 02:40:30.910811       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="mutatingwebhookconfiguration"
I0515 02:40:30.910850       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"blockaffinities.crd.projectcalico.org"}
I0515 02:40:30.910952       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="mutatingwebhookconfiguration" "worker count"=1
I0515 02:40:30.911200       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterinformations.crd.projectcalico.org"}
I0515 02:40:30.911394       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamconfigs.crd.projectcalico.org"}
I0515 02:40:30.912233       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:30.918379       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="validatingwebhookconfiguration"
I0515 02:40:30.918553       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="validatingwebhookconfiguration" "worker count"=1
I0515 02:40:30.918953       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:30.921947       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="apiservice"
I0515 02:40:30.911416       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:30.922633       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="apiservice" "worker count"=1
I0515 02:40:30.923953       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.getambassador.io"}
I0515 02:40:30.924414       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.scheduling.k8s.io"}
I0515 02:40:30.924611       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.crd.projectcalico.org"}
I0515 02:40:30.925479       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.cert-manager.io"}
I0515 02:40:30.925691       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.networking.k8s.io"}
I0515 02:40:30.925860       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.events.k8s.io"}
I0515 02:40:30.926269       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.acme.cert-manager.io"}
I0515 02:40:30.926444       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.admissionregistration.k8s.io"}
I0515 02:40:30.926594       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta1.autoscaling"}
I0515 02:40:30.926761       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.autoscaling"}
I0515 02:40:30.926950       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.apiextensions.k8s.io"}
I0515 02:40:30.927179       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authorization.k8s.io"}
I0515 02:40:30.927338       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta2.autoscaling"}
I0515 02:40:30.928849       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.acme.cert-manager.io"}
I0515 02:40:30.929405       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.policy"}
I0515 02:40:30.929635       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.rbac.authorization.k8s.io"}
I0515 02:40:30.929807       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.getambassador.io"}
I0515 02:40:30.929974       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta2.getambassador.io"}
I0515 02:40:30.941776       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.cert-manager.io"}
I0515 02:40:30.942130       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.node.k8s.io"}
I0515 02:40:30.942318       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authorization.k8s.io"}
I0515 02:40:30.942482       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.batch"}
I0515 02:40:30.942637       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.batch"}
I0515 02:40:30.942796       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2.getambassador.io"}
I0515 02:40:30.942965       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authentication.k8s.io"}
I0515 02:40:30.943207       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.networking.k8s.io"}
I0515 02:40:30.943366       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.certificates.k8s.io"}
I0515 02:40:30.943528       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.coordination.k8s.io"}
I0515 02:40:30.943687       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1."}
I0515 02:40:30.943854       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apiextensions.k8s.io"}
I0515 02:40:30.944353       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.admissionregistration.k8s.io"}
I0515 02:40:30.944531       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authentication.k8s.io"}
I0515 02:40:30.944941       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.extensions"}
I0515 02:40:30.945235       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.storage.k8s.io"}
I0515 02:40:30.945485       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apps"}
I0515 02:40:30.945653       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.rbac.authorization.k8s.io"}
I0515 02:40:30.945815       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.discovery.k8s.io"}
I0515 02:40:30.945990       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.coordination.k8s.io"}
I0515 02:40:30.946365       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.scheduling.k8s.io"}
I0515 02:40:30.946554       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.storage.k8s.io"}
I0515 02:40:30.994487       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:30.994656       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:30.995001       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:31.049958       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:31.050230       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:31.050477       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:31.133179       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:31.133342       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:31.134607       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:31.134798       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:31.160086       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" 
I0515 02:40:31.160243       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificaterequests.cert-manager.io"}
I0515 02:40:31.160538       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"felixconfigurations.crd.projectcalico.org"}
I0515 02:40:31.162777       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:31.700089       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:31.700285       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterissuers.cert-manager.io"}
I0515 02:40:31.700561       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgpconfigurations.crd.projectcalico.org"}
I0515 02:40:31.700858       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"mappings.getambassador.io"}
I0515 02:40:31.701168       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filterpolicies.getambassador.io"}
I0515 02:40:31.701363       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tcpmappings.getambassador.io"}
I0515 02:40:31.701531       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamblocks.crd.projectcalico.org"}
I0515 02:40:31.703460       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:32.410093       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:32.410261       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"issuers.cert-manager.io"}
I0515 02:40:32.410505       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"authservices.getambassador.io"}
I0515 02:40:32.410674       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimitservices.getambassador.io"}
I0515 02:40:32.410836       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tlscontexts.getambassador.io"}
I0515 02:40:32.411735       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:32.991902       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" 
I0515 02:40:32.992454       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificates.cert-manager.io"}
I0515 02:40:32.992768       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworksets.crd.projectcalico.org"}
I0515 02:40:32.993596       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"consulresolvers.getambassador.io"}
I0515 02:40:32.993849       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamhandles.crd.projectcalico.org"}
I0515 02:40:32.994179       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ippools.crd.projectcalico.org"}
I0515 02:40:32.994353       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimits.getambassador.io"}
I0515 02:40:32.994665       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.241307       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:33.241490       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"orders.acme.cert-manager.io"}
I0515 02:40:33.241736       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networkpolicies.crd.projectcalico.org"}
I0515 02:40:33.241949       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesendpointresolvers.getambassador.io"}
I0515 02:40:33.243082       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hosts.getambassador.io"}
I0515 02:40:33.243277       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tracingservices.getambassador.io"}
I0515 02:40:33.243445       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filters.getambassador.io"}
I0515 02:40:33.243615       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hostendpoints.crd.projectcalico.org"}
I0515 02:40:33.243806       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networksets.crd.projectcalico.org"}
I0515 02:40:33.244797       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"modules.getambassador.io"}
I0515 02:40:33.245120       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworkpolicies.crd.projectcalico.org"}
I0515 02:40:33.245369       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"logservices.getambassador.io"}
I0515 02:40:33.245544       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesserviceresolvers.getambassador.io"}
I0515 02:40:33.247844       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.711715       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:33.711885       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"challenges.acme.cert-manager.io"}
I0515 02:40:33.712397       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgppeers.crd.projectcalico.org"}
I0515 02:40:33.712926       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.820813       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I0515 02:40:33.823328       1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal"  "message"="cert-manager-cainjector-759496659c-6sgkj_6a3611b3-ae60-462d-88b6-3d71f9fba0d2 became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"8fad1895-8a0e-419d-a6e7-0f4603199533","apiVersion":"v1","resourceVersion":"702225"} "reason"="LeaderElection"
I0515 02:40:33.829504       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" 
I0515 02:40:33.829656       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificaterequests.cert-manager.io"}
I0515 02:40:33.832279       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"group":"","names":{"plural":"","kind":""},"scope":""},"status":{"conditions":null,"acceptedNames":{"plural":"","kind":""},"storedVersions":null}}}
I0515 02:40:33.835291       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.837113       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:33.837416       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.837805       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:33.838741       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.839460       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"groupPriorityMinimum":0,"versionPriority":0},"status":{}}}
I0515 02:40:33.840463       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.842624       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.843557       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (9h24m37.730610426s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:34.334433       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:34.334603       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterissuers.cert-manager.io"}
I0515 02:40:34.336972       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:34.978360       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:34.978487       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"issuers.cert-manager.io"}
I0515 02:40:34.978959       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:35.236502       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:35.237622       1 reflector.go:175] Starting reflector *v1.Secret (10h39m29.948285184s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:35.239889       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:35.240226       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:35.243315       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:36.038184       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="customresourcedefinition"
I0515 02:40:36.038386       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="customresourcedefinition" "worker count"=1
I0515 02:40:36.041377       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterissuers.cert-manager.io"}
I0515 02:40:36.041629       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tlscontexts.getambassador.io"}
I0515 02:40:36.041820       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filters.getambassador.io"}
I0515 02:40:36.042079       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamhandles.crd.projectcalico.org"}
I0515 02:40:36.042384       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificaterequests.cert-manager.io"}
I0515 02:40:36.042566       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterinformations.crd.projectcalico.org"}
I0515 02:40:36.042766       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamconfigs.crd.projectcalico.org"}
I0515 02:40:36.042972       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesendpointresolvers.getambassador.io"}
I0515 02:40:36.043195       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networksets.crd.projectcalico.org"}
I0515 02:40:36.043382       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimitservices.getambassador.io"}
I0515 02:40:36.043555       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hostendpoints.crd.projectcalico.org"}
I0515 02:40:36.043720       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimits.getambassador.io"}
I0515 02:40:36.043890       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworkpolicies.crd.projectcalico.org"}
I0515 02:40:36.044181       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hosts.getambassador.io"}
I0515 02:40:36.044508       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"orders.acme.cert-manager.io"}
I0515 02:40:36.044715       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"mappings.getambassador.io"}
I0515 02:40:36.044892       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworksets.crd.projectcalico.org"}
I0515 02:40:36.045912       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="apiservice"
I0515 02:40:36.046134       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="apiservice" "worker count"=1
I0515 02:40:36.046411       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authentication.k8s.io"}
I0515 02:40:36.046496       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"issuers.cert-manager.io"}
I0515 02:40:36.046575       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.events.k8s.io"}
I0515 02:40:36.046687       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgpconfigurations.crd.projectcalico.org"}
I0515 02:40:36.046727       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.batch"}
I0515 02:40:36.046857       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tcpmappings.getambassador.io"}
I0515 02:40:36.046901       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1."}
I0515 02:40:36.047114       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.cert-manager.io"}
I0515 02:40:36.047148       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"blockaffinities.crd.projectcalico.org"}
I0515 02:40:36.047258       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apps"}
I0515 02:40:36.047328       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesserviceresolvers.getambassador.io"}
I0515 02:40:36.047391       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta2.getambassador.io"}
I0515 02:40:36.047511       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"consulresolvers.getambassador.io"}
I0515 02:40:36.047690       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networkpolicies.crd.projectcalico.org"}
I0515 02:40:36.047907       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"modules.getambassador.io"}
I0515 02:40:36.048163       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ippools.crd.projectcalico.org"}
I0515 02:40:36.048337       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tracingservices.getambassador.io"}
I0515 02:40:36.049375       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificates.cert-manager.io"}
I0515 02:40:36.049577       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"felixconfigurations.crd.projectcalico.org"}
I0515 02:40:36.049788       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"authservices.getambassador.io"}
I0515 02:40:36.047528       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.storage.k8s.io"}
I0515 02:40:36.050933       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.networking.k8s.io"}
I0515 02:40:36.051120       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="mutatingwebhookconfiguration"
I0515 02:40:36.051201       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.apiextensions.k8s.io"}
I0515 02:40:36.051252       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="mutatingwebhookconfiguration" "worker count"=1
I0515 02:40:36.051388       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.admissionregistration.k8s.io"}
I0515 02:40:36.051557       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.rbac.authorization.k8s.io"}
I0515 02:40:36.051723       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.cert-manager.io"}
I0515 02:40:36.051906       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.discovery.k8s.io"}
I0515 02:40:36.052818       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:36.053334       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="validatingwebhookconfiguration"
I0515 02:40:36.053464       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="validatingwebhookconfiguration" "worker count"=1
I0515 02:40:36.053757       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:36.054862       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"challenges.acme.cert-manager.io"}
I0515 02:40:36.054991       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.scheduling.k8s.io"}
I0515 02:40:36.055193       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamblocks.crd.projectcalico.org"}
I0515 02:40:36.055239       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.rbac.authorization.k8s.io"}
I0515 02:40:36.055403       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.node.k8s.io"}
I0515 02:40:36.055451       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filterpolicies.getambassador.io"}
I0515 02:40:36.055566       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authorization.k8s.io"}
I0515 02:40:36.055706       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.networking.k8s.io"}
I0515 02:40:36.055784       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"logservices.getambassador.io"}
I0515 02:40:36.055873       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.acme.cert-manager.io"}
I0515 02:40:36.055967       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgppeers.crd.projectcalico.org"}
I0515 02:40:36.056099       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.getambassador.io"}
I0515 02:40:36.056244       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.scheduling.k8s.io"}
I0515 02:40:36.056383       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.policy"}
I0515 02:40:36.056527       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta1.autoscaling"}
I0515 02:40:36.056719       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authorization.k8s.io"}
I0515 02:40:36.056937       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.extensions"}
I0515 02:40:36.057431       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.storage.k8s.io"}
I0515 02:40:36.057608       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.admissionregistration.k8s.io"}
I0515 02:40:36.057843       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta2.autoscaling"}
I0515 02:40:36.058927       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.autoscaling"}
I0515 02:40:36.059177       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.coordination.k8s.io"}
I0515 02:40:36.059364       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.batch"}
I0515 02:40:36.059505       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authentication.k8s.io"}
I0515 02:40:36.059655       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apiextensions.k8s.io"}
I0515 02:40:36.059847       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.acme.cert-manager.io"}
I0515 02:40:36.060074       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.certificates.k8s.io"}
I0515 02:40:36.060224       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.coordination.k8s.io"}
I0515 02:40:36.060357       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2.getambassador.io"}
I0515 02:40:36.060501       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.crd.projectcalico.org"}
I0515 02:40:36.060642       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.getambassador.io"}
I0515 02:40:36.303478       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" 
I0515 02:40:36.303698       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificates.cert-manager.io"}
I0515 02:40:36.311511       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:36.471880       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:36.472601       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"orders.acme.cert-manager.io"}
I0515 02:40:36.474510       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:36.866916       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:36.867931       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"challenges.acme.cert-manager.io"}

kubectl logs -n cert-manager cert-manager-webhook-7c75b89bf6-c4fr6:

W0515 02:40:11.031377       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0515 02:40:11.033489       1 webhook.go:63]  "msg"="using dynamic certificate generating using CA stored in Secret resource"  "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I0515 02:40:11.035773       1 server.go:139]  "msg"="listening for insecure healthz connections"  "address"=":6080"
I0515 02:40:11.036697       1 server.go:152]  "msg"="listening for secure connections"  "address"=":10250"
I0515 02:40:11.036890       1 server.go:178]  "msg"="registered pprof handlers"  
I0515 02:40:11.053553       1 reflector.go:175] Starting reflector *v1.Secret (1m0s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:11.252535       1 authority.go:313]  "msg"="Generating new root CA"  
I0515 02:40:11.838994       1 authority.go:248]  "msg"="Detected change in CA secret data, notifying watchers..."  
I0515 02:40:12.043864       1 dynamic_source.go:171]  "msg"="Generating new ECDSA private key"  
I0515 02:40:12.100600       1 dynamic_source.go:186]  "msg"="Signing new serving certificate"  
I0515 02:40:12.448962       1 dynamic_source.go:192]  "msg"="Signed new serving certificate"  
I0515 02:40:12.516189       1 dynamic_source.go:198]  "msg"="Updated serving TLS certificate"
triagsupport
Source
picture sambhavsaggi
👍5

Most helpful comment

I am also running on OpenNebula k8s cluster.
Reolved it by running the cert-manager pods on the master node.

First make sure to label your master node:
kubectl label nodes <master-name> kubernetes.io/role=master

Now install cert-manager with the following helm cmd:

helm install --name cert-manager --namespace cert-manager --version v0.15.1 \
jetstack/cert-manager \
--set nodeSelector."kubernetes\.io/role"=master  \
--set cainjector.nodeSelector."kubernetes\.io/role"=master \
--set webhook.nodeSelector."kubernetes\.io/role"=master
picture ltetrel on 18 Jun 2020
👍3

All 20 comments

Can you please get us the following info:

  • version of cert-manager installed
  • state and logs of the webhook pod in namespace cert-manager

Thanks

/triage support

meyskens picture meyskens on 15 May 2020

@meyskens I have updated the question.

sambhavsaggi picture sambhavsaggi on 15 May 2020

Also seeing this on a freshly installed k8s 1.18.3 cluster with cert manager 0.15.1. Logs from the webhook pod are identical to the OP's.

thommay picture thommay on 31 May 2020
👍3

same issue with k8s 1.18.3 cluster with cert manager 0.15.1

cachak picture cachak on 8 Jun 2020

same issue with k8s 1.15.12 and cert manager 0.15.1. Is there any workaround to this issue?

zedtux picture zedtux on 12 Jun 2020

same issue with k8s 1.17.2 and cert-manager 0.15.1...

milkeryildirim picture milkeryildirim on 14 Jun 2020

same issue with k8s 1.17.5 and cert-manager 0.14.3...

estheban picture estheban on 16 Jun 2020

What OS are you using because it was working fine with cert-manager 0.13 on Debian 9, but since I moved to Ubuntu 18.04, and I can't make it working anymore (I wrote a guide at that time).

zedtux picture zedtux on 17 Jun 2020

I鈥檓 running all my nodes on CentOS 7. I鈥檝e updated my question.

sambhavsaggi picture sambhavsaggi on 17 Jun 2020
👍1

I am also running on OpenNebula k8s cluster.
Reolved it by running the cert-manager pods on the master node.

First make sure to label your master node:
kubectl label nodes <master-name> kubernetes.io/role=master

Now install cert-manager with the following helm cmd:

helm install --name cert-manager --namespace cert-manager --version v0.15.1 \
jetstack/cert-manager \
--set nodeSelector."kubernetes\.io/role"=master  \
--set cainjector.nodeSelector."kubernetes\.io/role"=master \
--set webhook.nodeSelector."kubernetes\.io/role"=master
ltetrel picture ltetrel on 18 Jun 2020
👍3

same issue with k8s 1.18.2 cluster calico with cert manager 0.15.1 and 0.13.1

0521ak47 picture 0521ak47 on 19 Jun 2020

same issue with k8s 1.17.2 and cert-manager 0.15.1

zxdyumiao picture zxdyumiao on 24 Jun 2020

same issue with openshift 4.3 and cert manager 0.15.1
the port 10250 does not seem to be listening in the cert-manager-webhook pod container, however the liveness probe is up :(

a-dawg picture a-dawg on 14 Jul 2020

same issue after upgrading k8s 1.17.x to 1.18.6 (Rancher / flannel 0.12.0) with cert-manager 0.15.0, 0.15.2 and 0.16.0

couple of observations:

  • unable to list cert-manager resources (certs, orders, etc) using kubectl, same error as it triggers the webhook
  • port-forward to webhook service and pod respond to requests
  • using curl inside the kube-apiserver pod:

    • webhook service ip takes more than a minute to respond

    • webhook pod ip responds immediately

which took me to this issue with flannel: https://github.com/coreos/flannel/issues/1243

after adding the route as proposed here things seem to start working again

sboschman picture sboschman on 31 Jul 2020

I would like to post that now I no more have any issues with cert-manager in my Kubernetes 1.15.12 cluster.

I did ran the sonobuoy tool which highlighted that master nodes were failing to communicate with worker node after the provisioning tool Chef ran due to a restart of the VPN service (that allows masters and workers to talk to each other securely).
Reboot the nodes (was a testing cluster) made the e2e tests passing, then cert-manager worked again.

I'm now checking why restarting the VPN service prevents the nodes to communicate together, but that's another topic.

To be more precise on my case, I'm running sonobuoy run -p systemd-logs && watch -n 1 sonobuoy status and in 15 seconds max all tests should be completed and marked as passed.
After Chef ran on a node, running the same makes that node test stuck.

When that happen, after grabbing the sonobuoy-systemd-logs-daemon-set-XXXX pod name of that node, looking at the sonobuoy-worker container's logs, I can see:

kubectl logs -f sonobuoy-systemd-logs-daemon-set-464c7dbf3bf84b2d-kttf4 -n sonobuoy -c sonobuoy-worker
time="2020-08-05T09:37:01Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-08-05T09:37:01Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.3.129]:8080/api/v1/progress/by-node/west-stg-mst-3/systemd-logs"
time="2020-08-05T09:37:02Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/systemd_logs
time="2020-08-05T09:39:13Z" level=error msg="error entry for attempt: 1, verb: PUT, time: 2020-08-05 09:39:13.777742167 +0000 UTC m=+131.832465314, URL: https://[10.244.3.129]:8080/api/v1/results/by-node/west-stg-mst-3/systemd-logs: Put https://[10.244.3.129]:8080/api/v1/results/by-node/west-stg-mst-3/systemd-logs: dial tcp 10.244.3.129:8080: connect: connection timed out"

All in all the sonobuoy-systemd-logs-daemon-set-XXXX pod, from the defective node, tries to reach the sonobuoy pod from the sonobuoy namespace but can't (while other sonobuoy-systemd-logs-daemon-set-XXXX pods did well) and the test is "stuck".

Update: Actually I just discovered that when restarting the VPN service, the flannel network interface disappears and is never recreated! That explains why Kubernetes pods on that node can't communicate with other pods from the cluster.
Like I said in that issue, restarting Docker allows Flannel to create its network interface and the sonobuoy tests are passing again.

zedtux picture zedtux on 5 Aug 2020

Going to close this one as it doesn't seem cert-manager related. Feel free to /reopen if needed.
If you think it could be a useful thing to document PRs to the documentations are always open!

meyskens picture meyskens on 29 Oct 2020

/close

meyskens picture meyskens on 29 Oct 2020

@meyskens: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jetstack-bot picture jetstack-bot on 29 Oct 2020

same issue when i want to create ingress service (kind: Ingress)
and i solved that by simple ping from ingress-nginx-controller to ingress-nginx-controller-admission.ingress-nginx.svc
shell to ingress-nginx-controller-ldkssk-sdjfn (this must be different in your env) and ping ingress-nginx-controller-admission.ingress-nginx.svc
i don't know why, but work

alikarimii picture alikarimii on 13 Nov 2020

@alikarimii Got any more commands for this one? Do you need to exec into the ingress-nginx-controller first?

alexpapworth picture alexpapworth on 21 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gaieges picture gaieges  路  3Comments
apetheriotis picture apetheriotis  路  3Comments
dontreboot picture dontreboot  路  3Comments
caiobegotti picture caiobegotti  路  4Comments
howardjohn picture howardjohn  路  3Comments