Istio: Sidecar injector webhook fails, injector service is unresponsive

[kvudata]

Describe the bug
A ReplicaSet is failing to be created due to the admission webhook failing to be called.

The error extracted via kubectl describe rs:

Warning  FailedCreate  11m (x19 over 33m)  replicaset-controller  Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject: dial tcp: lookup istio-sidecar-injector.istio-system.svc on 169.254.169.254:53: no such host

Expected behavior
Be able to successfully update a Deployment which in turn creates a ReplicaSet.

Steps to reproduce the bug
Unsure how to reproduce, but I'm happy to provide any debug info I can extract.
My steps were:

• Install by following https://istio.io/docs/setup/kubernetes/helm-install/
• Follow the tutorial to setup Ingress (until I got blocked)
• Try to patch a deployment in the namespace, note that deployment fails due to webhook error

Version

> istioctl version
Version: 0.8.0
GitRevision: 6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84
User: root@48d5ddfd72da
Hub: docker.io/istio
GolangVersion: go1.10.1
BuildStatus: Clean

> kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.1", GitCommit:"1883ce4eb0e057cfc2439ebeb9822da0a9d40405", GitTreeState:"clean", BuildDate:"2018-05-11T17:16:56Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}



md5-77aba666b5fb036f1b81f5f728b93212



/ # nslookup istio-sidecar-injector.istio-system.svc
Server:    10.47.240.10
Address 1: 10.47.240.10 kube-dns.kube-system.svc.cluster.local

Name:      istio-sidecar-injector.istio-system.svc
Address 1: 10.47.247.248 istio-sidecar-injector.istio-system.svc.cluster.local



md5-07be99f8879abf127ab4797233d6306a



root@gunicorn-deployment-55b4445479-7bkm6:/usr/src/app/api-flask# ping istio-sidecar-injector.istio-system.svc
PING istio-sidecar-injector.istio-system.svc.cluster.local (10.47.247.248): 56 data bytes

The istio-sidecar-injector pod doesn't have any recent logs i.e. doesn't indicate it received the webhooks.

[ayj]

The error message suggests there are no istio-sidecar-injector pods available. The equivalent k8s 1.10 error message is more informative (see below)

Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: no endpoints available for service "istio-sidecar-injector"

Can you dump the galley endpoints when the problem occurs?

kubectl -n istio-system get ep istio-sidecar-injector
kubectl -n istio-system describe deployment istio-sidecar-injector

[kvudata]

@ayj

> kubectl -n istio-system get ep istio-sidecar-injector
NAME                     ENDPOINTS       AGE
istio-sidecar-injector   10.44.1.9:443   4d

> kubectl -n istio-system describe deployment istio-sidecar-injector
Name:                   istio-sidecar-injector
Namespace:              istio-system
CreationTimestamp:      Thu, 14 Jun 2018 09:19:10 -0700
Labels:                 app=sidecarInjectorWebhook
                        chart=sidecarInjectorWebhook-0.8.0
                        heritage=Tiller
                        istio=sidecar-injector
                        release=istio
Annotations:            deployment.kubernetes.io/revision=1
                        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"sidecarInjectorWebhook","chart":"sidecarInjectorWe...
Selector:               istio=sidecar-injector
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           istio=sidecar-injector
  Service Account:  istio-sidecar-injector-service-account
  Containers:
   sidecar-injector-webhook:
    Image:  docker.io/istio/sidecar_injector:0.8.0
    Port:   <none>
    Args:
      --caCertFile=/etc/istio/certs/root-cert.pem
      --tlsCertFile=/etc/istio/certs/cert-chain.pem
      --tlsKeyFile=/etc/istio/certs/key.pem
      --injectConfig=/etc/istio/inject/config
      --meshConfig=/etc/istio/config/mesh
      --healthCheckInterval=2s
      --healthCheckFile=/health
    Liveness:     exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=2s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Readiness:    exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=2s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/istio/certs from certs (ro)
      /etc/istio/config from config-volume (ro)
      /etc/istio/inject from inject-config (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  false
   certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-sidecar-injector-service-account
    Optional:    false
   inject-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-sidecar-injector
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  istio-sidecar-injector-dbd67c88d (1/1 replicas created)
NewReplicaSet:   <none>
Events:          <none>

[ayj]

Thanks @kvudata. Can you provide the kubectl describe output for the replicaset and pod as well?

[kvudata]

Yep, @ayj:

> kubectl -n istio-system describe rs istio-sidecar-injector-dbd67c88d
Name:           istio-sidecar-injector-dbd67c88d
Namespace:      istio-system
Selector:       istio=sidecar-injector,pod-template-hash=868237448
Labels:         istio=sidecar-injector
                pod-template-hash=868237448
Annotations:    deployment.kubernetes.io/desired-replicas=1
                deployment.kubernetes.io/max-replicas=2
                deployment.kubernetes.io/revision=1
Controlled By:  Deployment/istio-sidecar-injector
Replicas:       1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           istio=sidecar-injector
                    pod-template-hash=868237448
  Service Account:  istio-sidecar-injector-service-account
  Containers:
   sidecar-injector-webhook:
    Image:  docker.io/istio/sidecar_injector:0.8.0
    Port:   <none>
    Args:
      --caCertFile=/etc/istio/certs/root-cert.pem
      --tlsCertFile=/etc/istio/certs/cert-chain.pem
      --tlsKeyFile=/etc/istio/certs/key.pem
      --injectConfig=/etc/istio/inject/config
      --meshConfig=/etc/istio/config/mesh
      --healthCheckInterval=2s
      --healthCheckFile=/health
    Liveness:     exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=2s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Readiness:    exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=2s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/istio/certs from certs (ro)
      /etc/istio/config from config-volume (ro)
      /etc/istio/inject from inject-config (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  false
   certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-sidecar-injector-service-account
    Optional:    false
   inject-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-sidecar-injector
    Optional:  false
Events:        <none>

> kubectl -n istio-system describe po istio-sidecar-injector-dbd67c88d-j68pw
Name:           istio-sidecar-injector-dbd67c88d-j68pw
Namespace:      istio-system
Node:           gke-istio-test-2-default-pool-7f649c6c-cv1v/10.128.0.5
Start Time:     Thu, 14 Jun 2018 09:19:10 -0700
Labels:         istio=sidecar-injector
                pod-template-hash=868237448
Annotations:    <none>
Status:         Running
IP:             10.44.1.9
Controlled By:  ReplicaSet/istio-sidecar-injector-dbd67c88d
Containers:
  sidecar-injector-webhook:
    Container ID:  docker://e1938d8f4b03fa6395ec8914b6e44845c785236ff799362b1a2fe72700cdbd73
    Image:         docker.io/istio/sidecar_injector:0.8.0
    Image ID:      docker-pullable://istio/sidecar_injector@sha256:ecb37af05435bc25f754419af432c5f07788e70b93f495cca41d7cb6d0d4f211
    Port:          <none>
    Args:
      --caCertFile=/etc/istio/certs/root-cert.pem
      --tlsCertFile=/etc/istio/certs/cert-chain.pem
      --tlsKeyFile=/etc/istio/certs/key.pem
      --injectConfig=/etc/istio/inject/config
      --meshConfig=/etc/istio/config/mesh
      --healthCheckInterval=2s
      --healthCheckFile=/health
    State:          Running
      Started:      Thu, 14 Jun 2018 09:19:14 -0700
    Ready:          True
    Restart Count:  0
    Liveness:       exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=2s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Readiness:      exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=2s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /etc/istio/certs from certs (ro)
      /etc/istio/config from config-volume (ro)
      /etc/istio/inject from inject-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-sidecar-injector-service-account-token-7xmc2 (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  false
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-sidecar-injector-service-account
    Optional:    false
  inject-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-sidecar-injector
    Optional:  false
  istio-sidecar-injector-service-account-token-7xmc2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-sidecar-injector-service-account-token-7xmc2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

[yacut]

Same issue with Istio 0.8.0 on minikube v0.27.0.

Workaround:

• Delete affected deployments
• Delete the istio-sidecar-injector pod
• Wait for a new istio-sidecar-injector pod is ready
• Create your deployments again

[yacut]

More information about this issue.

kubectl describe replicaset details-v1-7b97668445 
Name:           details-v1-7b97668445
Namespace:      default
Selector:       app=details,pod-template-hash=3653224001,version=v1
Labels:         app=details
                pod-template-hash=3653224001
                version=v1
Annotations:    deployment.kubernetes.io/desired-replicas=1
                deployment.kubernetes.io/max-replicas=2
                deployment.kubernetes.io/revision=1
Controlled By:  Deployment/details-v1
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=details
           pod-template-hash=3653224001
           version=v1
  Containers:
   details:
    Image:        istio/examples-bookinfo-details-v1:1.5.0
    Port:         9080/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                From                   Message
  ----     ------        ----               ----                   -------
  Warning  FailedCreate  37s (x15 over 2m)  replicaset-controller  Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority

kubectl --namespace istio-system logs -f istio-sidecar-injector-645fddc6db-ng2sv
2018-06-21T06:05:27.236479Z info    version [email protected]/istio-0.8.0-6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84-Clean
2018-06-21T06:05:27.323186Z info    New configuration: sha256sum bd448e4daf82e86695233d789fbb6b52e2779859f2f2642e7912cfd5e3469e5c
2018-06-21T06:05:27.325870Z info    Policy: enabled
2018-06-21T06:05:27.325913Z info    Template: |
  initContainers:
  - name: istio-init
    image: docker.io/istio/proxy_init:0.8.0
    args:
    - "-p"
    - [[ .MeshConfig.ProxyListenPort ]]
    - "-u"
    - 1337
    - "-m"
    - [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    - "-i"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges"  ]]"
    [[ else -]]
    - "*"
    [[ end -]]
    - "-x"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges"  ]]"
    [[ else -]]
    - ""
    [[ end -]]
    - "-b"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts"  ]]"
    [[ else -]]
    - [[ range .Spec.Containers -]][[ range .Ports -]][[ .ContainerPort -]], [[ end -]][[ end -]][[ end]]
    - "-d"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts" ]]"
    [[ else -]]
    - ""
    [[ end -]]
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    restartPolicy: Always

  containers:
  - name: istio-proxy
    image: [[ if (isset .ObjectMeta.Annotations "sidecar.istio.io/proxyImage") -]]
    "[[ index .ObjectMeta.Annotations "sidecar.istio.io/proxyImage" ]]"
    [[ else -]]
    docker.io/istio/proxyv2:0.8.0
    [[ end -]]
    args:
    - proxy
    - sidecar
    - --configPath
    - [[ .ProxyConfig.ConfigPath ]]
    - --binaryPath
    - [[ .ProxyConfig.BinaryPath ]]
    - --serviceCluster
    [[ if ne "" (index .ObjectMeta.Labels "app") -]]
    - [[ index .ObjectMeta.Labels "app" ]]
    [[ else -]]
    - "istio-proxy"
    [[ end -]]
    - --drainDuration
    - [[ formatDuration .ProxyConfig.DrainDuration ]]
    - --parentShutdownDuration
    - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
    - --discoveryAddress
    - [[ .ProxyConfig.DiscoveryAddress ]]
    - --discoveryRefreshDelay
    - [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]
    - --zipkinAddress
    - [[ .ProxyConfig.ZipkinAddress ]]
    - --connectTimeout
    - [[ formatDuration .ProxyConfig.ConnectTimeout ]]
    - --statsdUdpAddress
    - [[ .ProxyConfig.StatsdUdpAddress ]]
    - --proxyAdminPort
    - [[ .ProxyConfig.ProxyAdminPort ]]
    - --controlPlaneAuthPolicy
    - [[ .ProxyConfig.ControlPlaneAuthPolicy ]]
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: ISTIO_META_INTERCEPTION_MODE
      value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    imagePullPolicy: IfNotPresent
    securityContext:
        privileged: false
        readOnlyRootFilesystem: true
        [[ if eq (or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String) "TPROXY" -]]
        capabilities:
          add:
          - NET_ADMIN
        [[ else -]]
        runAsUser: 1337
        [[ end -]]
    restartPolicy: Always
    resources:
      limits:
        cpu: 0.05
        memory: 32Mi
      requests:
        cpu: 0.05
        memory: 16Mi
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      optional: true
      [[ if eq .Spec.ServiceAccountName "" -]]
      secretName: istio.default
      [[ else -]]
      secretName: [[ printf "istio.%s" .Spec.ServiceAccountName ]]
      [[ end -]]
2018-06-21T06:05:27.326789Z warn    Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018-06-21T06:36:43.122282Z info    http: TLS handshake error from 10.132.0.3:38194: remote error: tls: bad certificate
2018-06-21T06:36:43.127524Z info    http: TLS handshake error from 10.132.0.4:36286: remote error: tls: bad certificate
2018-06-21T06:36:43.138065Z info    http: TLS handshake error from 10.132.0.3:38196: remote error: tls: bad certificate
2018-06-21T06:36:43.226830Z info    http: TLS handshake error from 10.16.2.1:54328: remote error: tls: bad certificate
2018-06-21T06:36:43.229958Z info    http: TLS handshake error from 10.132.0.4:36288: remote error: tls: bad certificate
2018-06-21T06:36:43.256326Z info    http: TLS handshake error from 10.16.2.1:54330: remote error: tls: bad certificate

kubectl --namespace istio-system logs -f istio-citadel-58fc7dc4fb-k6wnz 
2018-06-21T06:30:24.653877Z warn    Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018-06-21T06:30:24.753388Z info    Use self-signed certificate as the CA certificate
2018-06-21T06:30:25.354842Z info    adding registry entry "k8s.cluster.local" -> "k8s.cluster.local"
2018-06-21T06:30:28.053310Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/certificate-controller" -> "spiffe://cluster.local/ns/kube-system/sa/certificate-controller"
2018-06-21T06:30:28.155120Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/endpoint-controller" -> "spiffe://cluster.local/ns/kube-system/sa/endpoint-controller"
2018-06-21T06:30:28.254235Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"
2018-06-21T06:30:28.653253Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-cleanup-old-ca-service-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-cleanup-old-ca-service-account"
2018-06-21T06:30:28.754209Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/prometheus" -> "spiffe://cluster.local/ns/istio-system/sa/prometheus"
2018-06-21T06:30:28.853370Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/node-controller" -> "spiffe://cluster.local/ns/kube-system/sa/node-controller"
2018-06-21T06:30:28.953651Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/persistent-volume-binder" -> "spiffe://cluster.local/ns/kube-system/sa/persistent-volume-binder"
2018-06-21T06:30:28.953815Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/replication-controller" -> "spiffe://cluster.local/ns/kube-system/sa/replication-controller"
2018-06-21T06:30:28.953901Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/service-account-controller" -> "spiffe://cluster.local/ns/kube-system/sa/service-account-controller"
2018-06-21T06:30:28.953977Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/default" -> "spiffe://cluster.local/ns/istio-system/sa/default"
2018-06-21T06:30:28.954030Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-egressgateway-service-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-egressgateway-service-account"
2018-06-21T06:30:28.954101Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/clusterrole-aggregation-controller" -> "spiffe://cluster.local/ns/kube-system/sa/clusterrole-aggregation-controller"
2018-06-21T06:30:28.954187Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/namespace-controller" -> "spiffe://cluster.local/ns/kube-system/sa/namespace-controller"
2018-06-21T06:30:28.954274Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/pvc-protection-controller" -> "spiffe://cluster.local/ns/kube-system/sa/pvc-protection-controller"
2018-06-21T06:30:28.954359Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/horizontal-pod-autoscaler" -> "spiffe://cluster.local/ns/kube-system/sa/horizontal-pod-autoscaler"
2018-06-21T06:30:28.954414Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/ttl-controller" -> "spiffe://cluster.local/ns/kube-system/sa/ttl-controller"
2018-06-21T06:30:28.954436Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/service-controller" -> "spiffe://cluster.local/ns/kube-system/sa/service-controller"
2018-06-21T06:30:28.954458Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/resourcequota-controller" -> "spiffe://cluster.local/ns/kube-system/sa/resourcequota-controller"
2018-06-21T06:30:28.954495Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-citadel-service-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-citadel-service-account"
2018-06-21T06:30:28.954513Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-mixer-post-install-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-mixer-post-install-account"
2018-06-21T06:30:28.954536Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/cronjob-controller" -> "spiffe://cluster.local/ns/kube-system/sa/cronjob-controller"
2018-06-21T06:30:28.954555Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/deployment-controller" -> "spiffe://cluster.local/ns/kube-system/sa/deployment-controller"
2018-06-21T06:30:28.954600Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/kube-dns-autoscaler" -> "spiffe://cluster.local/ns/kube-system/sa/kube-dns-autoscaler"
2018-06-21T06:30:28.954623Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/attachdetach-controller" -> "spiffe://cluster.local/ns/kube-system/sa/attachdetach-controller"
2018-06-21T06:30:29.053211Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/default" -> "spiffe://cluster.local/ns/kube-system/sa/default"
2018-06-21T06:30:29.053248Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-sidecar-injector-service-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-sidecar-injector-service-account"
2018-06-21T06:30:29.053272Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
2018-06-21T06:30:29.053289Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/cloud-provider" -> "spiffe://cluster.local/ns/kube-system/sa/cloud-provider"
2018-06-21T06:30:29.053302Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/replicaset-controller" -> "spiffe://cluster.local/ns/kube-system/sa/replicaset-controller"
2018-06-21T06:30:29.053313Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/pod-garbage-collector" -> "spiffe://cluster.local/ns/kube-system/sa/pod-garbage-collector"
2018-06-21T06:30:29.053323Z info    adding registry entry "spiffe://cluster.local/ns/default/sa/default" -> "spiffe://cluster.local/ns/default/sa/default"
2018-06-21T06:30:29.053335Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/heapster" -> "spiffe://cluster.local/ns/kube-system/sa/heapster"
2018-06-21T06:30:29.053344Z info    adding registry entry "spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account" -> "spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"
2018-06-21T06:30:29.053353Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/pv-protection-controller" -> "spiffe://cluster.local/ns/kube-system/sa/pv-protection-controller"
2018-06-21T06:30:29.053363Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/statefulset-controller" -> "spiffe://cluster.local/ns/kube-system/sa/statefulset-controller"
2018-06-21T06:30:29.053373Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/generic-garbage-collector" -> "spiffe://cluster.local/ns/kube-system/sa/generic-garbage-collector"
2018-06-21T06:30:29.053383Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/disruption-controller" -> "spiffe://cluster.local/ns/kube-system/sa/disruption-controller"
2018-06-21T06:30:29.053392Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/daemon-set-controller" -> "spiffe://cluster.local/ns/kube-system/sa/daemon-set-controller"
2018-06-21T06:30:29.053401Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/job-controller" -> "spiffe://cluster.local/ns/kube-system/sa/job-controller"
2018-06-21T06:30:29.053411Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/route-controller" -> "spiffe://cluster.local/ns/kube-system/sa/route-controller"
2018-06-21T06:30:29.053421Z info    adding registry entry "spiffe://cluster.local/ns/kube-public/sa/default" -> "spiffe://cluster.local/ns/kube-public/sa/default"
2018-06-21T06:30:29.053430Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/kube-dns" -> "spiffe://cluster.local/ns/kube-system/sa/kube-dns"
2018-06-21T06:30:29.053441Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/metadata-proxy" -> "spiffe://cluster.local/ns/kube-system/sa/metadata-proxy"
2018-06-21T06:30:29.053450Z info    adding registry entry "spiffe://cluster.local/ns/kube-system/sa/metrics-server" -> "spiffe://cluster.local/ns/kube-system/sa/metrics-server"
2018-06-21T06:30:36.853794Z info    Citadel monitor has started.
2018-06-21T06:30:36.853820Z info    Citadel has started
2018-06-21T06:30:36.853857Z info    Monitor server started.
2018-06-21T06:30:36.853883Z info    Starting GRPC server on port 8060
2018-06-21T06:31:37.755674Z error   Failed to create secret in attempt 1/3, (error: secrets "istio.certificate-controller" already exists)
2018-06-21T06:31:38.960098Z error   Failed to create secret in attempt 2/3, (error: secrets "istio.certificate-controller" already exists)
2018-06-21T06:31:39.967939Z error   Failed to create secret in attempt 3/3, (error: secrets "istio.certificate-controller" already exists)
2018-06-21T06:31:40.968109Z error   Failed to create secret for service account "certificate-controller"  (error: secrets "istio.certificate-controller" already exists), retries 3 times

I hope it will be helpful...

[ostromart]

Seems to be the same issue as caBundle being overwritten - would you agree @ayj ?

[ayj]

Yes. x509: certificate signed by unknown authority suggests caBundle is being overwritten.

[ostromart]

Marking as duplicate of https://github.com/istio/istio/issues/6069.