Gardener: Detect webhooks that are potentially problematic for maintenance activities and place a constraint on shoots

[stoyanr]

How to categorize this issue?

/area ops-productivity
/kind enhancement
/priority normal

What would you like to be added:
We should detect webhooks that are potentially problematic for maintenance activities and place a constraint on shoots of type MaintenancePossible to forbid maintenance similarly to how we currently forbid hibernation for similar reasons. The constraint could look like this:

    constraints:
    - type: MaintenancePossible
      status: 'False'
      lastTransitionTime: '2020-11-05T02:43:34Z'
      lastUpdateTime: '2020-10-29T07:32:57Z'
      reason: ProblematicWebhooks
      message: >-
        Maintenance activities cannot be performed on this shoot because of 
        MutatingWebhookConfiguration "sen-webhooks-shoot": webhook 
        "vpn-service-webhook.sen.sap.com" with failurePolicy "Fail" will probably 
        cause the maintenance activities to fail.

Alternatively, we may refrain from forbidding maintenance, but still add a constraint and a visual cue in the dashboard based on this constraint, e.g. a red "Schedule Maintenance" button with a popup that displays the constraint message.

Why is this needed:
During DoD activities, we noticed errors where the shoot reconciliation during maintenance activities failed due to a user webhook modifying kube-system pods. If the webhook is not reachable from kube-apiserver for whatever reason (e.g. no running vpn-shoot pods, or webhook pods), calling the webhook may fail with context deadline exceeded even if the failure policy is set to Fail, or even if it's set to Ignore and the timeout is long enough (30s or more). This in turn prevents kube-system pods from being created, which may prevent calico from properly initializing and cause the nodes to fail to join the cluster due to kubelet reporting NetworkPluginNotReady.

The issue is partially mitigated by https://github.com/gardener/gardener/pull/3100, but not prevented entirely.

The solution proposed above was suggested in the DoD sync.

[rfranzke]

Also related: https://github.com/gardener/gardener/issues/3020

[timuthy]

Does this mean we skip reconciliation as long as the Shoot has a problematic webhook?

If yes, we should make sure a shoot is reconciled as soon as a new Gardener version is rolled out at the latest, even if it then fails. Otherwise we can hardly ensure that a cluster was processed by necessary migration logic which is only shipped in certain versions of Gardener.

[danielfoehrKn]

Does this mean we skip reconciliation as long as the Shoot has a problematic webhook?

I understand this to be more relevant when the Node pool is being rolled? Not for the general reconciliation.
Then when the calico system pods fails to be created the node fails to join.

[rfranzke]

In a discussion round with @gardener/gardener-maintainers we agreed to introduce this additional constraint (MaintenancePreconditionsSatisfied) with the same check as for the HibernationPossible today (however, still a separate constraint so that it can be easier extended in the future with additional checks if needed).

However, we won't block maintenance if the preconditions are not satisfied, we will still perform it. The constraint is considered as information for the user so that he is aware of potential problems caused by his configurations. It can/should be displayed in the Gardener Dashboard as warning so that end-users become aware, cc @gardener/dashboard-maintainers