Eventing: ConfigMap missing causes Webhook to fail causing ConfigMap to be uncreatable

[Harwayne]

Describe the bug
The Webhook refuses to start unless the default-ch-webhook ConfigMap is present and properly formatted.

https://github.com/knative/eventing/blob/bdc48890f0e46a95e1bd4bb253b27b3af9ed5d05/cmd/webhook/main.go#L114-L115

The Webhook is setup to validate changes to ConfigMaps. If the Webhook fails, then the create/update ConfigMap request is rejected.

https://github.com/knative/eventing/blob/bdc48890f0e46a95e1bd4bb253b27b3af9ed5d05/config/core/webhooks/config-validation.yaml#L29

Combined with

https://github.com/knative/eventing/blob/bdc48890f0e46a95e1bd4bb253b27b3af9ed5d05/vendor/knative.dev/pkg/webhook/configmaps/configmaps.go#L106-L159

If the ConfigMap does not exist, then it cannot be created:

kubectl apply -f config/400-default-ch-config.yaml
Error from server (InternalError): error when creating "config/400-default-ch-config.yaml": Internal error occurred: failed calling webhook "config.webhook.eventing.knative.dev": Post https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=2s: no endpoints available for service "eventing-webhook"

Expected behavior
I should be able to recreate the ConfigMap without having to delete the config.webhook.eventing.knative.dev ValidatingWebhookConfiguration.

To Reproduce

1. Install Knative-Eventing.
2. Verify that the Webhook came up correctly and updated the config.webhook.eventing.knative.dev ValidatingWebhookConfiguration:

kubectl get ValidatingWebhookConfiguration config.webhook.eventing.knative.dev -oyaml

• It should have the following rules:
  rules: <ul> <li>apiGroups:<br /> <ul><br /> <li>""<br /><br /> apiVersions:</li><br /> <li>v1<br /><br /> operations:</li><br /> <li>CREATE</li><br /> <li>UPDATE<br /><br /> resources:</li><br /> <li>configmaps/*<br /><br /> scope: Namespaced<br /><br />



• Delete the ConfigMap:

kubectl -n knative-eventing delete ConfigMap default-ch-webhook

1. Delete the Webhook Pod

kubectl -n knative-eventing delete Pod -l app=eventing-webhook

1. Note that the newly created Webhook will fail to start.

kubectl -n knative-eventing logs -l app=eventing-webhook

{"level":"fatal","ts":"2020-06-01T19:11:38.223Z","logger":"eventing-webhook","caller":"sharedmain/main.go:260","msg":"Failed to start configuration manager","commit":"bdc4889","error":"configmap \"default-ch-webhook\" not found","stacktrace":"knative.dev/pkg/injection/sharedmain.WebhookMainWithConfig\n\tknative.dev/[email protected]/injection/sharedmain/main.go:260\nknative.dev/pkg/injection/sharedmain.WebhookMainWithContext\n\tknative.dev/[email protected]/injection/sharedmain/main.go:207\nmain.main\n\tknative.dev/eventing/cmd/webhook/main.go:351\nruntime.main\n\truntime/proc.go:203"}



md5-2b3c30e7937aaf83d02fd3892680e208

Error from server (InternalError): error when creating "config/400-default-ch-config.yaml": Internal error occurred: failed calling webhook "config.webhook.eventing.knative.dev": Post https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=2s: no endpoints available for service "eventing-webhook"

 **Knative release version**
Master (bdc4889).

 **Additional context**
The way to get out of this is:
1. Delete the ValidatingWebhookConfiguration
```shell
kubectl delete ValidatingWebhookConfiguration config.webhook.eventing.knative.dev

1. Create the ConfigMap

kubectl apply -f config/400-default-ch-config.yaml

1. Create the ValidatingWebhookConfiguration

kubectl apply -f config/500-webhook-config-validation.yaml

[Harwayne]

I don't think this is a good-first-issue. At least, not until we decide on the semantics we want.

My opinion is that we should set the failurePolicy to Ignore. As long as the Webhook is running, it can validate the ConfigMaps. But if it isn't running for some reason (such as the ConfigMaps are already in a bad state), then it doesn't prevent mutating them. Altering knative resources should have a failurePolicy of Fail, because we rely on the defaulting and verification, but I don't feel that the ConfigMaps have the same requirements.

That may not be the right approach, but I think we need to find one that breaks this cyclic dependency.

Another possibility is to have the Webhook prevent deletion of those ConfigMaps. Then the Webhook guards all mutations of the ConfigMap, not just create and update.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[VIjayHP]

@zhongduo @Harwayne
Hi All,
Command given in the official documentation to install knative-eventing core component's is not working:

$kubectl apply --filename https://github.com/knative/eventing/releases/download/v0.19.0/eventing-core.yaml

Error from server (InternalError): error when creating "https://github.com/knative/eventing/releases/download/v0.19.0/eventing-core.yaml": Internal error occurred: failed calling webhook "config.webhook.eventing.knative.dev": Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=2s": service "eventing-webhook" not found

It takes some time to create service "eventing-webhook" after waiting for 5 seconds and if i run the same command again:
Error from server (InternalError): error when creating "https://github.com/knative/eventing/releases/download/v0.19.0/eventing-core.yaml": Internal error occurred: failed calling webhook "config.webhook.eventing.knative.dev": Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=2s": dial tcp 10.96.225.134:443: connect: connection refused

Please suggest why this behavior ?