Kiali: All apps are degraded due to unsynced proxies

[HazemElAgaty]

General remarks

Describe the bug
All my apps are degraded due to unsynced proxies, although istioctl proxy-status shows all proxies are synced.

Versions used
Kiali: 1.27.0
Istio: 1.7.3
Kubernetes: 1.16.13

Installation
Installed using helmfile

  - name: kiali-operator
    namespace: kiali-operator
    chart: kiali/kiali-operator
    version: 1.27.0
    values:
      - cr:
          create: false
          namespace: istio-system
          spec:
            deployment:
              accessible_namespaces:
              - '**'

Kiali CR
kubectl get kiali -n istio-system kiali -o yaml>

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  annotations:
    ansible.operator-sdk/verbosity: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kiali.io/v1alpha1","kind":"Kiali","metadata":{"annotations":{"ansible.operator-sdk/verbosity":"1"},"name":"kiali","namespace":"istio-system"},"spec":{"deployment":{"accessible_namespaces":["**"],"namespace":"kiali-operator"},"external_services":{"istio_namespace":"istio-system","prometheus":{"url":"http://prometheus-chart-server.monitoring.svc.cluster.local"}}}}
  creationTimestamp: "2020-12-07T15:06:25Z"
  finalizers:
  - finalizer.kiali
  generation: 1
  name: kiali
  namespace: istio-system
  resourceVersion: "52180939"
  selfLink: /apis/kiali.io/v1alpha1/namespaces/istio-system/kialis/kiali
  uid: 3fb8a2a9-f634-4dbb-b419-260b12e42e44
spec:
  deployment:
    accessible_namespaces:
    - '**'
    namespace: kiali-operator
  external_services:
    istio_namespace: istio-system
    prometheus:
      url: http://prometheus-chart-server.monitoring.svc.cluster.local
status:
  accessibleNamespaces:
  - '**'
  conditions:
  - ansibleResult:
      changed: 1
      completion: 2020-12-07T15:08:55.280506
      failures: 0
      ok: 83
      skipped: 76
    lastTransitionTime: "2020-12-07T15:06:25Z"
    message: Awaiting next reconciliation
    reason: Successful
    status: "True"
    type: Running

Kiali ClusterRole
kubectl get clusterrole -n istio-system kiali -o yaml >

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    operator-sdk/primary-resource: istio-system/kiali
    operator-sdk/primary-resource-type: Kiali.kiali.io
  creationTimestamp: "2020-12-07T15:07:05Z"
  labels:
    app: kiali
    app.kubernetes.io/name: kiali
    app.kubernetes.io/part-of: kiali
    app.kubernetes.io/version: v1.27.0
    version: v1.27.0
  name: kiali
  resourceVersion: "52180047"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/kiali
  uid: 04d06607-c7d3-407b-8138-0570ab5b4055
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - namespaces
  - nodes
  - pods
  - pods/log
  - pods/proxy
  - replicationcontrollers
  - services
  verbs:
  - get
  - list
  - patch
  - watch
- apiGroups:
  - extensions
  - apps
  resources:
  - deployments
  - replicasets
  - statefulsets
  verbs:
  - get
  - list
  - patch
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - get
  - list
  - patch
  - watch
- apiGroups:
  - networking.istio.io
  - security.istio.io
  resources:
  - '*'
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - watch
- apiGroups:
  - monitoring.kiali.io
  resources:
  - monitoringdashboards
  verbs:
  - get
  - list
- apiGroups:
  - iter8.tools
  resources:
  - experiments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - watch

[lucasponce]

Please, could you describe how did you update/upgrade Kiali in your platform ?

Thanks

[HazemElAgaty]

Heyy @lucasponce,

I made a fresh install this time rather than updating it. I removed the chart and made sure all kiali resources are removed and then installed it again as described above.

[lucasponce]

Thanks @HazemElAgaty, I cc @xeviknal if he can take a look.

[HazemElAgaty]

@lucasponce You are welcome, thanks for the quick response guys

[xeviknal]

@HazemElAgaty what authentication strategy are you using? I guess that is the token one right?

I think I can reproduce this one now in my environment. Let me get back to you later and probably with a PR covering this one.
Appreciate your time, comments and patience. Thanks.

Update: btw, if you use the auth strategy anonymous you shouldn't get any problem. If you want to try it and let us know, it would be helpful too.

[HazemElAgaty]

@xeviknal Yes, I am using token . I will try out anonymous and get back to you.

[HazemElAgaty]

@xeviknal unfortunately I am still seeing the same behavior with anonymous strategy. I also slept on it so that Id give it time to cache all pods status, but still no luck. :/

istioctl proxy-status >

[CathalC2011]

I'm seeing the same behaviour my side.

Kubernetes v1.19.2
Kiali UI/Server v1.27.0
Prometheus 2.20.0

❯ istioctl version
client version: 1.7.2
control plane version: 1.7.3-9686754643d0939c1f4dd0ee20443c51183f3589
data plane version: 1.7.3 (350 proxies)

All pods show as synced when running istioctl ps, but in Kiali all workloads are degraded.

[xeviknal]

@HazemElAgaty I see one suspicious misconfiguration in your kiali cr. It is pretty tiny but it might effect everything.

spec:
  deployment:
    accessible_namespaces:
    - '**'
    namespace: kiali-operator
  external_services:
    istio_namespace: istio-system
    prometheus:
      url: http://prometheus-chart-server.monitoring.svc.cluster.local

The istio_namespace field shouldn't be under external_services. It should go at first level, under spec otherwise.
Therefore it should be:

spec:
  istio_namespace: istio-system
  deployment:
    accessible_namespaces:
    - '**'
    namespace: kiali-operator
  external_services:
    prometheus:
      url: http://prometheus-chart-server.monitoring.svc.cluster.local

This property is specially important when you are installing kiali in a different namespace than the default.

Hope this tiny bit puts the big ones in place :)

[xeviknal]

@CathalC2011 hello there!

Since it is pretty improbable that you've go the same problem as @HazemElAgaty, I am adding a couple of questions for me better understand which point are you at.

• Could you tell me how did you install kiali? In case you recently updated it, which method did you follow?
• Do you mind pasting your kiali clusterrole? kubectl get clusterrole -n istio-system kiali -o yaml
• Do you also mind pasting your kiali cr? k get kiali -n istio-system kiali -o yaml

Let's try with this intuition: there has been a change in the kiali permissions at pod level. Therefore, in the case of updating the kiali version just changing the image version number, it won't work. You'd need to re-install kiali (and kiali-operator) to give those permissions.

related to https://github.com/kiali/kiali/issues/3318

[CathalC2011]

Hey @xeviknal , so we're actually using the upstream kiali-server chart found here: https://github.com/kiali/helm-charts/tree/master/kiali-server

Pretty vanilla setup, minus the openshift and ingress resources.

So unfortunately not using the operator, but the end result should be identical.

We install Istio in istio-system namespace, and kiali in monitoring namespace.

apiVersion: v1
data:
  config.yaml: |
    auth:
      openid: {}
      openshift:
        client_id_prefix: kiali
      strategy: anonymous
    createSecret: false
    dashboard:
      grafanaURL: https://grafana.monitoring.svc.cluster.local:9965
    deployment:
      additional_service_yaml: {}
      affinity:
        node: {}
        pod: {}
        pod_anti: {}
      image_name: private.com/kiali
      image_pull_policy: Always
      image_pull_secrets: []
      image_version: v1.27-25fa611
      ingress_enabled: false
      logger:
        log_format: text
        log_level: debug
        sampler_rate: "1"
        time_field_format: 2006-01-02T15:04:05Z07:00
      namespace: monitoring
      node_selector: {}
      override_ingress_yaml:
        metadata: {}
      pod_annotations: {}
      pod_labels: {}
      priority_class_name: ""
      replicas: 1
      resources: {}
      secret_name: kiali
      service_account_name: kiali-service-account
      service_annotations: {}
      service_type: ""
      tolerations: []
      version_label: v1.27-25fa611
      view_only_mode: false
    external_services:
      prometheus:
        url: http://istio-prometheus.monitoring:9090
    identity:
      cert_file: ""
      private_key_file: ""
    istio_namespace: prometheus
    login_token:
      signing_key: <redacted>
    prometheusAddr: http://prometheus.monitoring.svc.cluster.local:9090
    secret_name: kiali
    server:
      metrics_enabled: true
      metrics_port: 9090
      port: 20001
      web_root: /kiali

We use Kiali in viewer-only mode. Here is our kiali-viewer clusterrole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: kiali
  name: kiali-viewer
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - namespaces
  - nodes
  - pods
  - pods/log
  - pods/proxy
  - replicationcontrollers
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  - apps
  resources:
  - deployments
  - replicasets
  - statefulsets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.istio.io
  - security.istio.io
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - monitoring.kiali.io
  resources:
  - monitoringdashboards
  verbs:
  - get
  - list
- apiGroups:
  - iter8.tools
  resources:
  - experiments
  verbs:
  - get
  - list

Currently using anonymous auth, if that matters!

[HazemElAgaty]

istio_namespace

@xeviknal Great catch, this fixed everything since yes I am installing kiali in a different namespace. Thanks a lot.

[CathalC2011]

Actually, our problem was extremely similar to above! istio_namespace was pointing to the namespace Kiali itself was running inside, rather than istio 🤦 Thank you for the help!

[jshaughn]

@xeviknal nice find. Not sure if there is anything we can do as a follow-up to help indicate to the user the potential issue.

[xeviknal]

@jshaughn good point.

What about adding an extra notification like the ones for jaeger and grafana when are disabled? (should play around a bit though)
Perhaps an entry in the FAQ?

[denniseffing]

I have the same issue because I changed the istio_labels config parameter to use app.kubernetes.io/name instead of app for the app label key.

The issue occurs because Kiali now expects the istiod pod to have the app.kubernetes.io/name=istiod label. However, istiod is deployed with app=istiod and there is no way to change that configuration using the Istio operator.

Is there a workaround for this?

[xeviknal]

hi @denniseffing,

Thanks for letting us know about this one. Unfortunately, there is no workaround this one.
I am preparing a PR approaching this bug. I guess it won't be available until 2021 though.

[jshaughn]

@xeviknal Shall we open a new issue? This is really an issue with the Istio operator, I think, and maybe we should open an issue there as well. But I guess we need to workaround it for now.

[jmazzitelli]

Is there no way to add additional labels to the istiod pod via istio operator? Maybe we can't remove or change the app one, but could we add a new one?

[denniseffing]

@jmazzitelli There is no way to do that. Afaik the operator uses the istio-discovery Helm chart internally and it doesn't provide a config parameter to inject labels to the istiod pod:

https://github.com/istio/istio/blob/master/manifests/charts/istio-control/istio-discovery/templates/deployment.yaml#L34-L46

[xeviknal]

@denniseffing kiali team has put a minor release to approach this bug: 1.28.1. It is ready to download now. If you are using kiali v1.28, you only need to re-download that one and the bug should be fixed.