Kiali: Error rendering dashboard on GKE with Istio add-on

[majelbstoat]

Describe the bug

Kiali fails to render graphs and charts following the quick start instructions on a vanilla GKE cluster with istio add-on.

Two errors seems important:

1) Cannot load the graph: Get http://prometheus.istio-system:9090/api/v1/query?query=round%28sum%28rate%28istio_requests_total%7Breporter%3D%22destination%22%2Csource_workload%3D%22unknown%22%2Cdestination_service_namespace%3D%22default%22%7D+%5B60s%5D%29%29+by+%28source_workload_namespace%2Csource_workload%2Csource_app%2Csource_version%2Cdestination_service_namespace%2Cdestination_service_name%2Cdestination_workload_namespace%2Cdestination_workload%2Cdestination_app%2Cdestination_version%2Crequest_protocol%2Cresponse_code%2Cresponse_flags%29%2C0.001%29&time=2019-06-21T19%3A15%3A23Z: dial tcp: lookup prometheus.istio-system on 10.181.0.10:53: no such host

2) Unknown Istio implementation version [email protected]/istio-b2915d396d159e54a88a259c971940348d8a9082-b2915d396d159e54a88a259c971940348d8a9082-Clean is not recognized, thus not supported.

The GKE add-on is quite opaque, but it seems that Grafana, Prometheus, and Istio are either not installed, not installed where expected, or the wrong version. If there are extra steps that need to be done, it would be great to have those in the documentation.

Versions used
Kiali: 0.21
Istio: 1.1.3-gke.0 (presumed, from GKE cluster version)
Kubernetes flavour and version: GKE 1.13.6-gke.6

To Reproduce
Quick install, then go to the dashboard.

Expected behavior
Graphs demonstrating service to service communication

[jmazzitelli]

@lucasponce was just running tests on GKE - he can probably help. He had it working. Though I will say this:

Unknown Istio implementation version [email protected]/istio-b2915d396d159e54a88a259c971940348d8a9082-b2915d396d159e54a88a259c971940348d8a9082-Clean is not recognized, thus not supported.

Means you aren't using a released version of Istio. That's a version string from probably a self-built Istio or a snapshot release or something.

[majelbstoat]

I think I am maybe on an early access version (to workaround a different istio/k8s bug 😂) so that would make sense. but, i think GKE always does slightly bespoke builds, hence the -gke.X designation they add, so i'm curious whether this will be an ongoing problem. thanks for the clarification.

[jmazzitelli]

BTW: any errors you get like that about graph not able to render, look in the kiali pod's logs - usually you'll see error messages that can be used to help diagnose the problem.

[lucasponce]

Oh, apologizes, I didn't see the notification of this issue.

@majelbstoat I'm using these draft notes for installing Istio, exposing Telemetry addongs and upgrading Kiali on GKE:

https://gist.github.com/lucasponce/ac4de4ad5a2994ec7e5a291746db4c28

I want to prepare a more ellaborated document once it's tested but let me know if those help to you.

As there are several steps, I tried to summarize them into a single document.

If correct I can create a medium post with all the details.

Thanks

[majelbstoat]

Thanks @lucasponce, I'll give this a crack. One notable difference will be that because the cluster has Cloud Run with GKE enabled, it already has a specific version of istio installed as part of GKE's istio add-on. But, i'll see if that causes issues :)

[lucasponce]

@majelbstoat yeah, the specific integration between GKE and the Istio addon can be the issue here.
In my notes I basically follow the Istio recommendation:
https://istio.io/docs/setup/kubernetes/platform-setup/gke/
But with all the steps in a single place to install/configure all pieces.

Let me know if it works for you.

[lucasponce]

@majelbstoat I updated the steps used to install Istio on GKE and upgrade Kiali to v1.0.0 here
https://lucasponce.github.io/presentations/install-istio-kiali.html

We have tested this with some group of users and no problems so far, please, let me know if it also works for your environment, or if there is some scenario not covered and we can update/work on it.

Thanks

[shotat]

Same issue.
Is there any way to run Kiali with managed Istio on GKE?
It would be nice if we could run Kiali without reinstalling Istio manually.

[israel-hdez]

@shotat Do you get both the unknown version message and the error loading the graph? or just one of them?

[lucasponce]

@shotat I will try to give a try to see if I can reproduce your errors.

[lucasponce]

Ok, I've installed GKE with Istio addon and these are my pods and services:

[lponce@attilan ~]$ kubectl get pods -n istio-system
NAME                                     READY     STATUS    RESTARTS   AGE
istio-citadel-859cbd6bc7-hr6r4           1/1       Running   0          3m
istio-galley-6f7fd75474-kcqgz            1/1       Running   0          3m
istio-ingressgateway-6c77f9bfb4-k7gkb    1/1       Running   0          3m
istio-pilot-9ddcf649-xkb9q               2/2       Running   0          3m
istio-policy-7bf9cf67b7-62wxk            2/2       Running   3          3m
istio-sidecar-injector-94c485dbb-zdptm   1/1       Running   0          3m
istio-telemetry-5954dc5ddc-szh44         2/2       Running   2          3m
promsd-76f8d4cff8-df2rq                  2/2       Running   1          3m

[lponce@attilan ~]$ kubectl get services -n istio-system
NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                                                                                                                                      AGE
istio-citadel            ClusterIP      10.72.0.233    <none>           8060/TCP,15014/TCP                                                                                                                           6m
istio-galley             ClusterIP      10.72.3.32     <none>           443/TCP,15014/TCP,9901/TCP                                                                                                                   6m
istio-ingressgateway     LoadBalancer   10.72.13.104   104.197.116.26   15020:31361/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31594/TCP,15030:31780/TCP,15031:31299/TCP,15032:30835/TCP,15443:30460/TCP   6m
istio-pilot              ClusterIP      10.72.4.73     <none>           15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                       6m
istio-policy             ClusterIP      10.72.9.194    <none>           9091/TCP,15004/TCP,15014/TCP                                                                                                                 6m
istio-sidecar-injector   ClusterIP      10.72.10.80    <none>           443/TCP                                                                                                                                      6m
istio-telemetry          ClusterIP      10.72.10.29    <none>           9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                                       6m
promsd                   ClusterIP      10.72.1.238    <none>           9090/TCP                                                                                                                                     6m

So, by default prometheus service is diffferent and it seems that is using the "promsd", so when installing Kiali, the Kiali CR should customize this, because by default Istio uses "http://prometheus.istio-system:9090" and in this case, it sounds it should be used as "http://promsd.istio-system:9090".

This is the reference:
https://github.com/kiali/kiali/blob/master/operator/deploy/kiali/kiali_cr.yaml#L207

That would be a first step, we should check if there are further changes.
I don't see other addons, so Jaeger tracing integration won't be available as well.

Let me know if that helps.

P.S.: I will try to prepare a custom CR today to check if there are more changes to apply, but just wanted to share this first.

[lucasponce]

Ok, confirmed.

I used this Kiali CR:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
annotations:
  ansible.operator-sdk/reconcile-period: "0s"
spec:
  deployment:
    image_version: "v1.0"
    accessible_namespaces: [ "**" ]  
  external_services:
    grafana:
      url: 
    tracing:
      service: tracing
      url: 
    prometheus:
      url: "http://promsd.istio-system:9090"      
  identity:
    cert_file: ""
    private_key_file: ""

with the operator update:

bash <(curl -L https://git.io/getLatestKialiOperator) -oiv v1.0.0 --kiali-cr $HOME/update-kiali-cr.yaml

To install Kiali 1.0.0 and update the prometheus service in the kiali-cr.

If you have installed a current version you can edit the Kiali CR online using:

 [lponce@attilan ~]$ kubectl edit kiali kiali -n kiali-operator

and adding your changes there. The operator should re-start Kiali with the good url.

Please, let me know if this helps.

[lucasponce]

cc @majelbstoat @shotat

[shotat]

@israel-hdez

Do you get both the unknown version message and the error loading the graph? or just one of them?

Both of them.

@lucasponce

Thank you for the quick investigation!
I'd like to try it later. ( I'm sorry but it may take for a few days because of my another work to do. )

[majelbstoat]

Making these changes seems to get it working :)

Stlil have this error:

Unknown Istio implementation version [email protected]/istio-b2915d396d159e54a88a259c971940348d8a9082-b2915d396d159e54a88a259c971940348d8a9082-Clean is not recognized, thus not supported.

But pages load, services are listed etc.

Thanks!

[lucasponce]

Yes, Kiali should be updated to recognize other Istio distributions.

Thanks for the feedback.

[lucasponce]

@majelbstoat @shotat
by the way, to recognize this version in kiali should be an easy "goot first bug" as the changes are located just in this area:
https://github.com/kiali/kiali/blob/master/status/versions.go#L25

We'd love to get more contributors in the project :-) so I invite you if you can help us to grow our community members !

Thanks again.

[slve]

based on @lucasponce 's fix, but in order to avoid overriding other settings I ended up using a patch on the custom resource

kubectl -n kiali-operator patch kiali.kiali.io -p '{"spec": {"external_services": {"prometheus": {"url": "http://promsd.istio-system:9090"}}}}' kiali --type=merge

[lucasponce]

Closing this issue.
Please, don't hesitate to re-open, if there is more actions to perform around the topic.

[taisph]

As per the Istio on GKE release notes, promsd is for internal metrics only. Instead, use this guide to install Prometheus for Istio on GKE.