Kubeflow: istio-nodeagent failed to get secret for proxy sidecar from secret cache: failed to get root cert

[pretendhigh]

kubeflow 1.1.0
pod istio-nodeagent-br5j5 image: gcr.io/istio-release/node-agent-k8s:release-1.3-latest-daily
centos7.6

When I start a deploy x-0, sidecar istio-proxy will not be ready.

kb logs -f x-0 -c istio-proxy -n nvidia-gpu
...
info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
...

I try to get the logs of the istio-nodeagent on the same kubelet node with x-0

kubectl logs -f istio-nodeagent-br5j5 -n istio-system
...
error  sdsServiceLog  CONNECTION ID: sidecar~10.253.12.32~x-0.nvidia-gpu.svc.cluster.local-7538, RESOURCE NAME： ROOTCA，EVENT：close connection. Failed to get secret for proxy "sidecar~10.253.12.32~x-0.nvidia-gpu.svc.cluster.loca" from secret cache: failed to get root cert.
...

It seems that only the node with x-0 pod has the problem. Pod distributed to other node work fine.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| kind/bug | 0.70 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[pretendhigh]

After read the article about sds of istio1.3 (which was not supported by the istio team now), I know that citadel is a import role in sds. So I check the log of citadel. the log looks like this invalid bearer token, token has expired. I restarted citadel kubectl delete po istio-citadel-*** -n istio-system . After the citadel restared, it works fine.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/istio | 0.82 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.