Serving: Install knative in bare metal cluster failed

[ysjjovo]

What version of Knative?

0.15.0

Expected Behavior

Install success!

Actual Behavior

Failed!

Steps to Reproduce the Problem

kubectl apply -f serving-crds.yaml && kubectl apply -f serving-core.yaml got error below.

[root@xxx]# kubectl apply -f serving-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
namespace/knative-serving unchanged
serviceaccount/controller unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-admin configured
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-admin unchanged
image.caching.internal.knative.dev/queue-proxy unchanged
horizontalpodautoscaler.autoscaling/activator unchanged
deployment.apps/activator created
service/activator-service unchanged
deployment.apps/autoscaler created
service/autoscaler created
deployment.apps/controller created
service/controller created
deployment.apps/webhook created
service/webhook created
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-edit unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-view unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-core unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-podspecable-binding unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.serving.knative.dev unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.serving.knative.dev unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev unchanged
secret/webhook-certs unchanged
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found
Error from server (InternalError): error when creating "serving-core.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: service "webhook" not found

Command kubectl get po -A output below.

NAMESPACE         NAME                         READY   STATUS             RESTARTS   AGE
knative-serving   activator-5d89f69d68-2m788   0/1     CrashLoopBackOff   6          8m5s
knative-serving   autoscaler-c94b6fff-827r8    0/1     CrashLoopBackOff   6          8m5s
knative-serving   controller-65459c49f-htbg9   0/1     CrashLoopBackOff   6          8m5s
knative-serving   webhook-7b55469cc5-zg6gr     0/1     CrashLoopBackOff   6          8m4s
kube-system       coredns-797455887b-878b4     1/1     Running            15         323d
kube-system       coredns-797455887b-8jd76     1/1     Running            15         323d

Command kubectl logs -n knative-serving activator-5d89f69d68-2m788 output below.

[root@xxx]# kubectl logs -n knative-serving   activator-5d89f69d68-2m788
2020/06/15 09:16:32 Registering 2 clients
2020/06/15 09:16:32 Registering 3 informer factories
2020/06/15 09:16:32 Registering 3 informers
{"level":"info","ts":"2020-06-15T09:16:32.054Z","caller":"logging/config.go:111","msg":"Successfully created the logger."}
{"level":"info","ts":"2020-06-15T09:16:32.054Z","caller":"logging/config.go:112","msg":"Logging level set to info"}
{"level":"info","ts":"2020-06-15T09:16:32.155Z","logger":"activator","caller":"activator/main.go:166","msg":"Starting the knative activator","commit":"12fd001","knative.dev/controller":"activator","knative.dev/pod":"activator-5d89f69d68-2m788"}
{"level":"info","ts":"2020-06-15T09:16:32.155Z","logger":"activator","caller":"activator/main.go:195","msg":"Connecting to Autoscaler at ws://autoscaler.knative-serving.svc.cluster.local:8080","commit":"12fd001","knative.dev/controller":"activator","knative.dev/pod":"activator-5d89f69d68-2m788"}
{"level":"info","ts":"2020-06-15T09:16:32.155Z","logger":"activator","caller":"profiling/server.go:59","msg":"Profiling enabled: false","commit":"12fd001","knative.dev/controller":"activator","knative.dev/pod":"activator-5d89f69d68-2m788"}
{"level":"info","ts":"2020-06-15T09:16:32.155Z","logger":"activator","caller":"websocket/connection.go:132","msg":"Connecting to ws://autoscaler.knative-serving.svc.cluster.local:8080","commit":"12fd001","knative.dev/controller":"activator","knative.dev/pod":"activator-5d89f69d68-2m788"}
{"level":"fatal","ts":"2020-06-15T09:16:32.256Z","logger":"activator","caller":"activator/main.go:241","msg":"Failed to start configuration manager","commit":"12fd001","knative.dev/controller":"activator","knative.dev/pod":"activator-5d89f69d68-2m788","error":"configmap \"config-logging\" not found","stacktrace":"main.main\n\tknative.dev/serving/cmd/activator/main.go:241\nruntime.main\n\truntime/proc.go:203"}

Last line says it failed due to configmap config-logging not found.
But when I apply it independently by kubectl apply -f config-logging.yaml

Error from server (InternalError): error when creating "config-logging.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/config-validation?timeout=10s: context deadline exceeded

It seems apply configmap 'config-logging' depends on 'webhook' to become ready.But webhook depends on existence of configmap 'config-logging'.

[vagababov]

Do you have the logs for webhook as well?

[vagababov]

I know how to unstuck you, but we probably need to stop crashing if logging CM isn't there.
To fix the process:

1. delete the webhook deployments (knative-serving/{webhook,istio-webhook}.
2. delete the validatingwebhookconfigurations.admissionregistration.k8s.io {config.webhook.istio.networking.internal.knative.dev,config.webhook.serving.knative.dev}
3. apply the CM
4. apply knative

[mattmoor]

the order things are being applied in that output is inconsistent with the contents of serving-core.yaml in our most recent release (at least).

The webhook configurations are the last resources we create, so the config-foo resources should have already been created before the webhooks are registered.

[ysjjovo]

I know how to unstuck you, but we probably need to stop crashing if logging CM isn't there.
To fix the process:

1. delete the webhook deployments (knative-serving/{webhook,istio-webhook}.
2. delete the validatingwebhookconfigurations.admissionregistration.k8s.io {config.webhook.istio.networking.internal.knative.dev,config.webhook.serving.knative.dev}
3. apply the CM
4. apply knative

Thanks for you suggestion!I delete webooks with following commands.

kubectl delete ValidatingWebhookConfiguration  -l serving.knative.dev/release=v0.15.0
kubectl delete MutatingWebhookConfiguration  -l serving.knative.dev/release=v0.15.0

Then apply knative succeed!

[ysjjovo]

the order things are being applied in that output is inconsistent with the contents of serving-core.yaml in our most recent release (at least).

The webhook configurations are the last resources we create, so the config-foo resources should have already been created before the webhooks are registered.

Thanks for your reply.You're right.It's my fault to uninstall knative with command kubectl delete ns knative-serving.I should use command kubectl delete -f serving-core.yaml instead!