K8s-config-connector: validating webhook annotation-defaulter timeout

[RoryShively]

Theres another issue thats somewhat related to this https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/406 but that issue is related to k8s 1.19 where Im getting a different error

Bug Description

GKE version: 1.17.17-gke.1101
ConfigConnector version: 1.37.0
Config Connector Mode: cluster

Followed https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall to a tee except my service account SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com doesn't have the editor role for security purposes. It has a few roles including roles/storage.admin and Im trying to create a storage bucket with config-connector

Im trying to create this resource

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  annotations:
    cnrm.cloud.google.com/force-destroy: 'false'
  labels:
    name: config-connector-test
  name: io1-sandbox-config-connector-test
spec:
  bucketPolicyOnly: true
  cors:
    - maxAgeSeconds: 3600
      method:
        - GET
        - HEAD
        - DELETE
      origin:
        - 'http://example.appspot.com'
      responseHeader:
        - Content-Type
  lifecycleRule:
    - action:
        type: Delete
      condition:
        age: 7
  versioning:
    enabled: true

Additional Diagnostic Information

The error Im getting:

 Internal error occurred: failed calling webhook "annotation-defaulter.cnrm.cloud.google.com": Post https://cnrm-validating-webhook.cnrm-system.svc:443/annotation-defaulter?timeout=30s: dial tcp 10.60.1.10:9443: i/o timeout

Copy of k -n cnrm-system logs cnrm-webhook-manager-6744f9f6c8-4246q -f:

I0308 17:54:53.561366       1 request.go:645] Throttling request took 1.046667609s, request: GET:https://10.64.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s
{"severity":"info","logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
2021/03/08 17:54:55 Registering Webhooks.
starting up webhooks
{"severity":"info","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/deny-immutable-field-updates"}
{"severity":"info","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/deny-unknown-fields"}
{"severity":"info","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/iam-validation"}
{"severity":"info","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/annotation-defaulter"}
2021/03/08 17:54:57 Starting the Cmd.
2021/03/08 17:54:57 Waiting up to 2m0s for the http server to be ready...
{"severity":"info","logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"severity":"info","logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"severity":"info","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"severity":"info","logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"severity":"info","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
2021/03/08 17:54:58 Setting container as ready...
2021/03/08 17:54:58 Container is ready
2021/03/08 17:57:01 http: TLS handshake error from 127.0.0.1:44638: EOF

[TylerRichie]

I'm running into an issue I think is related:

GKE version: 1.18.15-gke.1102
ConfigConnector version: 1.37.0
Config Connector Mode: namespaced

My config-connector service account has the Owner role.

Here is the resource I am trying to create:

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  name: external-dns
  namespace: config-connector
spec:
  displayName: external-dns

The error I get is:

Error from server (InternalError): error when creating "manifest.yaml": Internal error occurred: failed calling webhook "annotation-defaulter.cnrm.cloud.google.com": Post https://cnrm-validating-webhook.cnrm-system.svc:443/annotation-defaulter?timeout=30s: dial tcp 10.4.1.7:9443: i/o timeout

Looking at the logs for cnrm-webhook-manager I see:

Info
2021-03-09T17:32:49.493750ZThrottling request took 1.004923739s, request: GET:https://172.17.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s
Info
2021-03-09T17:32:50.539862258Z{addr: :8080, logger: controller-runtime.metrics, msg: metrics server is starting to listen}
Error
2021-03-09T17:32:50.540362760Z2021/03/09 17:32:50 Registering Webhooks.
Info
2021-03-09T17:32:50.540386472Zstarting up webhooks
Info
2021-03-09T17:32:51.587058601Z{logger: controller-runtime.webhook, msg: registering webhook, path: /deny-immutable-field-updates}
Info
2021-03-09T17:32:51.587109450Z{logger: controller-runtime.webhook, msg: registering webhook, path: /deny-unknown-fields}
Info
2021-03-09T17:32:51.587121831Z{logger: controller-runtime.webhook, msg: registering webhook, path: /iam-validation}
Info
2021-03-09T17:32:51.587134349Z{logger: controller-runtime.webhook, msg: registering webhook, path: /annotation-defaulter}
Error
2021-03-09T17:32:51.587253101Z2021/03/09 17:32:51 Starting the Cmd.
Info
2021-03-09T17:32:51.587301523Z{logger: controller-runtime.webhook.webhooks, msg: starting webhook server}
Info
2021-03-09T17:32:51.587346167Z{logger: controller-runtime.manager, msg: starting metrics server, path: /metrics}
Error
2021-03-09T17:32:51.587468779Z2021/03/09 17:32:51 Waiting up to 2m0s for the http server to be ready...
Info
2021-03-09T17:32:51.587772230Z{logger: controller-runtime.certwatcher, msg: Updated current TLS certificate}
Info
2021-03-09T17:32:51.588063979Z{host: , logger: controller-runtime.webhook, msg: serving webhook server, port: 9443}
Info
2021-03-09T17:32:51.588202600Z{logger: controller-runtime.certwatcher, msg: Starting certificate watcher}
Error
2021-03-09T17:32:51.601405476Z2021/03/09 17:32:51 Setting container as ready...
Error
2021-03-09T17:32:51.601462211Z2021/03/09 17:32:51 Container is ready
Error
2021-03-09T17:34:35.448937445Z2021/03/09 17:34:35 http: TLS handshake error from 127.0.0.1:45684: EOF
Error
2021-03-09T17:34:53.093043385Z2021/03/09 17:34:53 http: TLS handshake error from 127.0.0.1:38164: EOF
Error
2021-03-09T17:48:55.955493012Z2021/03/09 17:48:55 http: TLS handshake error from 127.0.0.1:46386: remote error: tls: unknown certificate authority
Error
2021-03-09T18:18:19.608642487Z2021/03/09 18:18:19 http: TLS handshake error from 10.4.2.10:57390: local error: tls: bad record MAC

I tried the workaround in https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/406#issuecomment-784458291. The new pods came up and recreated the secret, but I'm still running into this issue and seeing the same TLS handshake error lines in the logs for the cnrm-webhook-manager.

Also relevant -- I'm using the Config Connector add-on on GKE with a fresh cluster.

[xiaobaitusi]

Hi @RoryShively and @TylerRichie, did you install KCC on private GKE clusters? If so, the timeout could be caused by this known issue.

We have reverted the port number to 433 from version 1.39.0. If you are willing to switch manual installation, you can download version 1.39.0 or above and let us know if that fixes the problem.

Note that if you want to keep using the addon, for now you must manually create a firewall rule that allows your cluster control plane to initiate TCP connections to your nodes on port 9443. For more information, see Adding firewall rules for specific use cases. The port revert fix will be available via Addon within two weeks.

Sorry for your inconvenience.

[TylerRichie]

I am indeed running a private cluster, and opening up port 9443 fixed the issue. Thanks for the workaround @xiaobaitusi!

[RoryShively]

Worked for me!!! Thanks for the clarification