Gardener: kube-controller-manager is often not scaled down on hibernation

[ialidzhikov]

How to categorize this issue?

/area dev-productivity ops-productivity
/kind bug
/priority normal

What happened:
kube-controller-manager is often not scaled on hibernation. There might be something wrong with Scale on upstream side or dependency-watchdog can scale up kube-controller-manager again.

What you expected to happen:
kube-controller-manager to be scaled down.

How to reproduce it (as minimally and precisely as possible):

1. Hibernate a Shoot

1. Ensure that kube-controller-manager is not scaled down

$ k -n shoot--foo--bar get po
NAME                                       READY   STATUS             RESTARTS   AGE
kube-controller-manager-85489f686b-sgt4c   0/1     CrashLoopBackOff   5          4m6s

$ k -n shoot--foo--bar get deploy
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
aws-lb-readvertiser          0/0     0            0           45h
cloud-controller-manager     0/0     0            0           45h
gardener-resource-manager    0/0     0            0           45h
kube-apiserver               0/0     0            0           45h
kube-controller-manager      0/1     1            0           45h
kube-scheduler               0/0     0            0           45h
machine-controller-manager   0/0     0            0           45h

$ k -n garden-foo get shoot bar -o jsonpath='{.status.hibernated}'
true

Anything else we need to know?:

Environment:

• Gardener version: v1.10
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

[vpnachev]

I have succeed to reproduce the issue with the current master of the gardener.
The gardenlet has scaled down the kube-controller-manager but a few seconds later dependecy-watchdog scaled it up and in the next hour it was still scaled to 1 replica.

Then I have updated the dwd to the latest master that contains https://github.com/gardener/dependency-watchdog/pull/23 and it now ignores clusters that are hibernated, hibernating or awaking which is great improvement.

@amshuman-kr could you please cut a new release of dwd? The new version needs more permissions - basically

diff --git a/charts/seed-bootstrap/charts/dependency-watchdog/templates/probe-rbac.yaml b/charts/seed-bootstrap/charts/dependency-watchdog/templates/probe-rbac.yaml
index fb244bd92..e0e0d8ea3 100644
--- a/charts/seed-bootstrap/charts/dependency-watchdog/templates/probe-rbac.yaml
+++ b/charts/seed-bootstrap/charts/dependency-watchdog/templates/probe-rbac.yaml
@@ -23,6 +23,14 @@ kind: ClusterRole
 metadata:
   name: gardener.cloud:dependency-watchdog-probe:cluster-role
 rules:
+- apiGroups:
+  - extensions.gardener.cloud
+  resources:
+  - clusters
+  verbs:
+  - get
+  - watch
+  - list
 - apiGroups:
   - ""
   resources:

has to be added here https://github.com/gardener/gardener/blob/b978feafc2ea7a93b865ecc27754ba44595f0c5f/charts/seed-bootstrap/charts/dependency-watchdog/templates/probe-rbac.yaml#L25

[ialidzhikov]

/ping @amshuman-kr

[gardener-robot]

@amshuman-kr

Message

/ping @amshuman-kr

[amshuman-kr]

@amshuman-kr could you please cut a new release of dwd? The new version needs more permissions - basically

Sorry for the delay. Will cut dwd release.