Project: Log flooded with: testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
While the KubeDB Operator is working fine on fresh Kubernetes 1.15 clusters, it floods its log with many
I0823 19:19:45.067465 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
messages per second on a 1.15.3 cluster upgraded from 1.13. When setting --bypass-validating-webhook-xray=false to true, the operator starts normally but requesting a Postgres database terminates with
# kubectl apply -f /tmp/y
error: error validating "/tmp/y": error validating data: [unknown object type "nil" in Postgres.metadata, unknown object type "nil" in Postgres.spec]; if you choose to ignore these errors, turn validation off with --validate=false
Increasing the log level does not provide additional information.
Database object:
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: postgres
namespace: harbor-db
spec:
version: 11.2
replicas: 2
standbyMode: Hot
streamingMode: asynchronous
storageType: Durable
storage:
storageClassName: local
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
terminationPolicy: DoNotTerminate
Cluster was deployed and upgraded with kubeadm.
# kubectl logs -n kubedb-operator kubedb-operator-c9cccd9ff-z6bcw -f
I0823 19:19:21.154788 1 log.go:172] FLAG: --alsologtostderr="false"
I0823 19:19:21.155520 1 log.go:172] FLAG: --audit-dynamic-configuration="false"
I0823 19:19:21.155541 1 log.go:172] FLAG: --audit-log-batch-buffer-size="10000"
I0823 19:19:21.155555 1 log.go:172] FLAG: --audit-log-batch-max-size="1"
I0823 19:19:21.155569 1 log.go:172] FLAG: --audit-log-batch-max-wait="0s"
I0823 19:19:21.155580 1 log.go:172] FLAG: --audit-log-batch-throttle-burst="0"
I0823 19:19:21.155595 1 log.go:172] FLAG: --audit-log-batch-throttle-enable="false"
I0823 19:19:21.155622 1 log.go:172] FLAG: --audit-log-batch-throttle-qps="0"
I0823 19:19:21.155642 1 log.go:172] FLAG: --audit-log-format="json"
I0823 19:19:21.155663 1 log.go:172] FLAG: --audit-log-maxage="0"
I0823 19:19:21.155675 1 log.go:172] FLAG: --audit-log-maxbackup="0"
I0823 19:19:21.155693 1 log.go:172] FLAG: --audit-log-maxsize="0"
I0823 19:19:21.155723 1 log.go:172] FLAG: --audit-log-mode="blocking"
I0823 19:19:21.155749 1 log.go:172] FLAG: --audit-log-path="-"
I0823 19:19:21.155761 1 log.go:172] FLAG: --audit-log-truncate-enabled="false"
I0823 19:19:21.155776 1 log.go:172] FLAG: --audit-log-truncate-max-batch-size="10485760"
I0823 19:19:21.155794 1 log.go:172] FLAG: --audit-log-truncate-max-event-size="102400"
I0823 19:19:21.155811 1 log.go:172] FLAG: --audit-log-version="audit.k8s.io/v1"
I0823 19:19:21.155823 1 log.go:172] FLAG: --audit-policy-file=""
I0823 19:19:21.155837 1 log.go:172] FLAG: --audit-webhook-batch-buffer-size="10000"
I0823 19:19:21.155852 1 log.go:172] FLAG: --audit-webhook-batch-initial-backoff="10s"
I0823 19:19:21.155899 1 log.go:172] FLAG: --audit-webhook-batch-max-size="400"
I0823 19:19:21.155912 1 log.go:172] FLAG: --audit-webhook-batch-max-wait="30s"
I0823 19:19:21.155924 1 log.go:172] FLAG: --audit-webhook-batch-throttle-burst="15"
I0823 19:19:21.155938 1 log.go:172] FLAG: --audit-webhook-batch-throttle-enable="true"
I0823 19:19:21.155974 1 log.go:172] FLAG: --audit-webhook-batch-throttle-qps="10"
I0823 19:19:21.155986 1 log.go:172] FLAG: --audit-webhook-config-file=""
I0823 19:19:21.156028 1 log.go:172] FLAG: --audit-webhook-initial-backoff="10s"
I0823 19:19:21.156039 1 log.go:172] FLAG: --audit-webhook-mode="batch"
I0823 19:19:21.156050 1 log.go:172] FLAG: --audit-webhook-truncate-enabled="false"
I0823 19:19:21.156062 1 log.go:172] FLAG: --audit-webhook-truncate-max-batch-size="10485760"
I0823 19:19:21.156099 1 log.go:172] FLAG: --audit-webhook-truncate-max-event-size="102400"
I0823 19:19:21.156113 1 log.go:172] FLAG: --audit-webhook-version="audit.k8s.io/v1"
I0823 19:19:21.156139 1 log.go:172] FLAG: --authentication-kubeconfig=""
I0823 19:19:21.156151 1 log.go:172] FLAG: --authentication-skip-lookup="false"
I0823 19:19:21.156177 1 log.go:172] FLAG: --authentication-token-webhook-cache-ttl="***REDACTED***"
I0823 19:19:21.156191 1 log.go:172] FLAG: --authentication-tolerate-lookup-failure="false"
I0823 19:19:21.156212 1 log.go:172] FLAG: --authorization-always-allow-paths="[]"
I0823 19:19:21.156224 1 log.go:172] FLAG: --authorization-kubeconfig=""
I0823 19:19:21.156238 1 log.go:172] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0823 19:19:21.156255 1 log.go:172] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I0823 19:19:21.156272 1 log.go:172] FLAG: --bind-address="0.0.0.0"
I0823 19:19:21.156285 1 log.go:172] FLAG: --burst="1000000"
I0823 19:19:21.156297 1 log.go:172] FLAG: --bypass-validating-webhook-xray="false"
I0823 19:19:21.156313 1 log.go:172] FLAG: --cert-dir="apiserver.local.config/certificates"
I0823 19:19:21.156325 1 log.go:172] FLAG: --client-ca-file=""
I0823 19:19:21.156338 1 log.go:172] FLAG: --contention-profiling="false"
I0823 19:19:21.156426 1 log.go:172] FLAG: --enable-analytics="false"
I0823 19:19:21.156440 1 log.go:172] FLAG: --enable-mutating-webhook="true"
I0823 19:19:21.156456 1 log.go:172] FLAG: --enable-status-subresource="true"
I0823 19:19:21.156467 1 log.go:172] FLAG: --enable-swagger-ui="false"
I0823 19:19:21.156480 1 log.go:172] FLAG: --enable-validating-webhook="true"
I0823 19:19:21.156503 1 log.go:172] FLAG: --governing-service="kubedb"
I0823 19:19:21.156515 1 log.go:172] FLAG: --help="false"
I0823 19:19:21.156531 1 log.go:172] FLAG: --http2-max-streams-per-connection="1000"
I0823 19:19:21.156544 1 log.go:172] FLAG: --kubeconfig=""
I0823 19:19:21.156580 1 log.go:172] FLAG: --label-key-blacklist="[app.kubernetes.io/name,app.kubernetes.io/version,app.kubernetes.io/instance,app.kubernetes.io/managed-by]"
I0823 19:19:21.156608 1 log.go:172] FLAG: --log-flush-frequency="5s"
I0823 19:19:21.156627 1 log.go:172] FLAG: --log_backtrace_at=":0"
I0823 19:19:21.156642 1 log.go:172] FLAG: --log_dir=""
I0823 19:19:21.156655 1 log.go:172] FLAG: --logtostderr="false"
I0823 19:19:21.156668 1 log.go:172] FLAG: --profiling="true"
I0823 19:19:21.156683 1 log.go:172] FLAG: --qps="1e+06"
I0823 19:19:21.156695 1 log.go:172] FLAG: --rbac="true"
I0823 19:19:21.156720 1 log.go:172] FLAG: --requestheader-allowed-names="[]"
I0823 19:19:21.156732 1 log.go:172] FLAG: --requestheader-client-ca-file=""
I0823 19:19:21.156759 1 log.go:172] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
I0823 19:19:21.156782 1 log.go:172] FLAG: --requestheader-group-headers="[x-remote-group]"
I0823 19:19:21.156802 1 log.go:172] FLAG: --requestheader-username-headers="[x-remote-user]"
I0823 19:19:21.156814 1 log.go:172] FLAG: --restrict-to-operator-namespace="false"
I0823 19:19:21.156834 1 log.go:172] FLAG: --resync-period="10m0s"
I0823 19:19:21.156848 1 log.go:172] FLAG: --secure-port="8443"
I0823 19:19:21.156863 1 log.go:172] FLAG: --stderrthreshold="0"
I0823 19:19:21.156876 1 log.go:172] FLAG: --tls-cert-file="/var/serving-cert/tls.crt"
I0823 19:19:21.156899 1 log.go:172] FLAG: --tls-cipher-suites="[]"
I0823 19:19:21.156914 1 log.go:172] FLAG: --tls-min-version=""
I0823 19:19:21.156933 1 log.go:172] FLAG: --tls-private-key-file="/var/serving-cert/tls.key"
I0823 19:19:21.156945 1 log.go:172] FLAG: --tls-sni-cert-key="[]"
I0823 19:19:21.156958 1 log.go:172] FLAG: --use-kubeapiserver-fqdn-for-aks="true"
I0823 19:19:21.156992 1 log.go:172] FLAG: --v="5"
I0823 19:19:21.157016 1 log.go:172] FLAG: --vmodule=""
I0823 19:19:21.158186 1 run.go:24] Starting kubedb-server...
I0823 19:19:21.665420 1 lib.go:112] Kubernetes version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
I0823 19:19:21.676838 1 controller.go:72] Ensuring CustomResourceDefinition...
I0823 19:19:27.059758 1 run.go:36] Starting KubeDB controller
I0823 19:19:27.063557 1 secure_serving.go:116] Serving securely on [::]:8443
I0823 19:19:34.216698 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.266907 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.319198 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.367432 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.418706 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.468974 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.517785 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.568045 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.624865 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.669029 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.718427 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.767254 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.819074 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.867753 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.918380 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:34.968322 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:35.018171 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:42.966474 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.017614 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.067211 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.118584 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.167205 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.216738 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.265386 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.317651 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.367405 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.418042 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.467325 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.518467 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.567775 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.619337 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.667269 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.717348 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.768475 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.816292 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.868107 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.915721 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:43.967653 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.016684 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.068304 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.116301 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.167209 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.216919 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.268163 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.318698 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.368303 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.418715 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.467704 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.515856 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.566234 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.617260 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.667452 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.716672 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.766436 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.818468 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.866578 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.921900 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:44.967020 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:45.017021 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0823 19:19:45.067465 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
[鈥
What does _testing ValidatingWebhook_ mean and what could cause this issue?
KubeDB Operator was deployed via Helm.
installer
stephan2012
All 9 comments
Which version of the operator are you using?
tamalsaha
on 24 Aug 2019
Operator version is 0.12
stephan2012
on 24 Aug 2019
I have the same issue, however, logs are flooded with 2 lines:
I0829 20:43:49.833486 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises
I0829 20:43:49.842134 1 log.go:172] http: TLS handshake error from 172.25.95.244:58174: remote error: tls: bad certificate
version of operator v0.13.0-rc.0
antonu17
on 29 Aug 2019
@stephan2012 @antonu17 @tamalsaha any update? We have just discovered this in our cluster. We are not sure what to do, as we have production Elasticsearch workloads that cannot be disrupted.
dacox
on 12 Jul 2020
@dacox Check if there鈥檚 proxy configured for kube-proxy or KubeDB (http_proxy, https_proxy, no_proxy)
stephan2012
on 13 Jul 2020
have same problem too.
Jamlee
on 10 Sep 2020
check your system time in every virtual machine. if time is error, ca and tls is not generated correctly.
Jamlee
on 17 Sep 2020
I have the same issue, however, logs are flooded with 2 lines:
I0829 20:43:49.833486 1 xray.go:232] testing ValidatingWebhook using an object with GVR = kubedb.com/v1alpha1, Resource=redises I0829 20:43:49.842134 1 log.go:172] http: TLS handshake error from 172.25.95.244:58174: remote error: tls: bad certificateversion of operator v0.13.0-rc.0
I fixed my kubedb-operator v0.13.0-rc.0 by updating the apiserver certificate. It seems the certificate for kubedb-operator api-server is only valid for one year (https://github.com/kubepack/onessl/blob/73986b61ba3ee905a1f112ad5f0f4f05017b6a1f/vendor/gomodules.xyz/cert/cert.go#L112) so we didn't have this error message in the beginning or in the new cluster.
I basically executed these two onessl commands: https://github.com/kubedb/installer/blob/296f10b0ca480b6478dbd656e74cb3aaef9d608b/deploy/kubedb.sh#L416 and then patched the CA certificate in validator and mutator, as well as replacing the server certificate and key in the apiserver-cert secert in my kubedb namespace.
zheli
on 17 Nov 2020
❤1
🎉1
👍1
The auto generated certificates in Helm certs are valid for 10 year. Please update / upgrade your KubeDB deployment during that period. This should resolve this.
tamalsaha
on 19 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings
Related issues
david-ecc
路
5Comments
pjebs
路
3Comments
olksdr
路
6Comments
yazfaz
路
4Comments
mauritsvdvijgh
路
5Comments
Most helpful comment
I fixed my kubedb-operator v0.13.0-rc.0 by updating the apiserver certificate. It seems the certificate for kubedb-operator api-server is only valid for one year (https://github.com/kubepack/onessl/blob/73986b61ba3ee905a1f112ad5f0f4f05017b6a1f/vendor/gomodules.xyz/cert/cert.go#L112) so we didn't have this error message in the beginning or in the new cluster.
I basically executed these two onessl commands: https://github.com/kubedb/installer/blob/296f10b0ca480b6478dbd656e74cb3aaef9d608b/deploy/kubedb.sh#L416 and then patched the CA certificate in validator and mutator, as well as replacing the server certificate and key in the apiserver-cert secert in my kubedb namespace.