Serving: default-domain always fails on Istio sidecar enabled env

[nak3]

What version of Knative?

HEAD(443f4b233db67ac14669fe2c2c844d115b0cc6fe)

Expected Behavior

default-domain should not fail.

Steps to Reproduce the Problem

1. Deploy Serving with Istio sidecar enabled env.
2. Deploy default-domain pods.

Actual Behavior

default-domain always fails on Mesh mode

$ kubectl get pod -n knative-serving default-domain-nslk7
NAME                   READY   STATUS   RESTARTS   AGE
default-domain-nslk7   1/2     Error    0          114m

This is because it failed to get configmap.

$ kubectl -n knative-serving  logs  default-domain-nslk7 -c default-domain
W0403 08:20:18.365692       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"level":"fatal","ts":1585902018.374972,"logger":"fallback.default-domain","caller":"default-domain/main.go:173","msg":"Error getting ConfigMap","error":"Get \"https://172.20.0.1:443/api/v1/namespaces/knative-serving/configmaps/config-domain\": dial tcp 172.20.0.1:443: connect: connection refused","stacktrace":"main.main\n\tknative.dev/serving/cmd/default-domain/main.go:173\nruntime.main\n\truntime/proc.go:203"}

This is similar to https://github.com/knative/serving/issues/7486. But this needs to fix two parts - getting config(network connection error) and creating KIngress(webhook returns certs error).

[tcnghia]

@nak3 do subsequent restarts fix the problem?

[nak3]

@tcnghia Yes, I think it should fix the problem and the the best way.
I will send the PR.

/assign

[feuyeux]

I met the same issue in knative 0.13 and istio 1.5.1, wish it should be fixed soon. And as a workaround, I want to know, how to do this subsequent restart? Thanks!

[nak3]

@feuyeux You can change the restartPolicy to OnFailure from Never like this https://github.com/knative/serving/pull/7496/files.

By the way, there are some other workarounds for this issue actually like:

• a. Disable sidecar by adding sidecar.istio.io/inject: "false" annotation.
• b. Do not use default-domain but just configure it manually :smiley:

[feuyeux]

@nak3 thanks a lot! It works!

▶ sed -i "" "s#restartPolicy: Never#restartPolicy: OnFailure#g" serving-default-domain.yaml
▶ kubectl delete -f serving-default-domain.yaml
▶ kubectl apply -f serving-default-domain.yaml

▶ pwd
/Users/han/shop/knative-docs/docs/serving/samples/hello-world/helloworld-go
▶ sed "s#{username}#feuyeux#g" service.yaml > go-service.yaml
▶ ak apply --filename go-service.yaml
service.serving.knative.dev/helloworld-go created
▶ ak get po
NAME                                              READY   STATUS            RESTARTS   AGE
helloworld-go-mbfrg-deployment-54fd66bd74-jcnmt   3/3     Running   0          14s
▶ ak get ksvc helloworld-go --output=custom-columns=NAME:.metadata.name,URL:.status.url
NAME            URL
helloworld-go   http://helloworld-go.default.39.100.10.246.xip.io                                     
▶ curl http://helloworld-go.default.39.100.10.246.xip.io
Hello Go Sample v1!

[zzxwill]

I got ride of the issue by deleting and reapplying Magic DNX (see).

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.14.0/serving-default-domain.yaml

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.