Kiali: [Operator/UI] istio_namespace doesn't work with istio_component_namespaces

[PatrickDuncan]

General remarks

https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L47-L65

Describe the bug

istio_namespace is ignored when istio_component_namespaces is present.

Versions used
Kiali: 1.23.0
Istio: 1.7.0
Kubernetes flavour and version: 1.17.5

To Reproduce

istiod -> istio-system
istio-ingressgateway -> istio-system
kiali -> istio-observe
prometheus -> istio-observe

istio_component_namespaces:
  prometheus: istio-observe
istio_namespace: istio-system

Expected behavior

The namespace for istiod and istio-ingressgateway should be inherited from istio_namespace since it's not defined in istio_component_namespaces. It shouldn't use Kiali's deploy namespace.

Minor bug, the workaround is just to define the istiod and istio-ingressgateway namespaces separately

[jmazzitelli]

FWIW: I think we want to deprecate the istio_component_namespace stuff - that was needed for a-now-obsolete Istio installation mechanism where things like pilot, citadel, etc would be in different places. Now there is one monolith (istiod) there is no really need for this and in fact creates confusion.

IIRC, the thinking was if you had anything in istio_component_namespace, that would be assumed to have everything. istio_namespace wasn't a fallback. I can't remember exactly.

But, be that as it may, there is no real need for istio_component_namespace anymore. and we will most likely deprecate it/remove it at some point soon.

[jmazzitelli]

Here's one of the PRs where we were going to do this - but pulled back. We'll probably do this in the future soon, though. https://github.com/kiali/kiali-operator/pull/54

[PatrickDuncan]

@jmazzitelli Thanks for the reply. If you guys deprecate istio_component_namespace how would you go about something like this?

istiod -> istio-system
istio-ingressgateway -> istio-system
kiali -> istio-observe
prometheus -> istio-observe

I think it's still a reasonable use case to have istio core and istio addon resources in separate namespaces. I'm currently doing separating them so I can (easily) tear down the observability without affecting istio/istio-ingressgateway/sidecars

[jmazzitelli]

You tell the system where Prometheus is here: https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L452

Kiali is deployed here: https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L286

You indicate where the core istio components are in the istio_namespace setting.

[PatrickDuncan]

Since prometheus is not in istio_namespace it shows up as not found:

istio_namespace: istio-system
external_services:
  prometheus:
    custom_metrics_url: http://prometheus.istio-observe:9090
    url: http://prometheus.istio-observe:9090

(Also custom_metrics_url is not in kiali_cr.yaml)

[jmazzitelli]

custom_metrics_url got removed/moved. We should probably create an FAQ for this. The status stuff changed:

https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L509-L513

I didn't' write that stuff - so I'm not sure how it works. I'll see if we can get this documented a little better.

Here's the prometheus component status settings: https://github.com/kiali/kiali-operator/blob/master/deploy/kiali/kiali_cr.yaml#L541-L543

[PatrickDuncan]

I saw a ticket regarding component statuses, but it only makes it a warning with:

external_services:
  prometheus:
    component_status:
      app_label: "prometheus123"
      is_core: false

I could _hide_ the warning by changing app_label to "ignore" or something, but that's not ideal. I'd also rather not disable the health of anything either.

Thanks for the help, I'll leave the issue open for now

[xeviknal]

@PatrickDuncan The istio component status takes the namespaces from the istio_component_namespaces. If you add there the prometheus: istio-observe namespace and you have the proper "app=prometheus" label in the p8s deployment, you should be able to see the real prometheus status.
In addition, the is_core field let you define the severity of the message: true => error severity, false => warning severity.

+1 to add more documentation about this in kiali.io

[PatrickDuncan]

@xeviknal Hey, only using istio_component_namespaces does work, but @jmazzitelli said it will be removed eventually

With it removed the only option is the component status, but that would just make Prometheus outside of the istio namespace a warning, when in fact it's healthy.

[jmazzitelli]

@xeviknal should we keep the istio_component_namespaces ? We were going to remove it because it didn't seem to provide any value-add and it was only here because it was supporting an obsolete way of installing istio.

[xeviknal]

@jmazzitelli If this field is removed, I'd need another field for users to specify where they istio components are installed in. Otherwise, we will be auto-discovering and guessing values which I don't think is a good idea.

I am fine also by moving this namespace into each component_status object in grafana, prometheus, jaeger and istio. Does this makes more sense? Just to specify the namespace next to each component?

[jmazzitelli]

I am fine also by moving this namespace into each component_status object in grafana, prometheus, jaeger and istio. Does this makes more sense? Just to specify the namespace next to each component?

I think that would make sense - otherwise, we are requiring people to put values in the istio-component-namespaces field and they have to know to enter (and not have any typos in) the key names "prometheus" and "grafana", etc.

[xeviknal]

@jmazzitelli good! We are aligned then. I'll prepare a PR approaching this.

[xeviknal]

@PatrickDuncan a solution is in the way in here: https://github.com/kiali/kiali/pull/3211
Feel free to add any review or comment in there.

[xeviknal]

@PatrickDuncan thank you for opening the issue an showing you interest in it. The pull request previously referenced is now merged. If you'd like to try it right now, the latest version of Kiali already includes those changes. If not, kiali 1.24 is expected to go live next monday.

[PatrickDuncan]

@xeviknal Thanks, I'll try out 1.24 when it's out

[pslobo]

Sorry to bump a closed issue, but I'm still seeing this behaviour in 1.24 and 1.26. How exactly are we supposed to configure Kiali when components such as Prometheus, Grafana and Jaeger are in other namespaces?

[jmazzitelli]

How exactly are we supposed to configure Kiali when components such as Prometheus, Grafana and Jaeger are in other namespaces?

Prometheus: https://github.com/kiali/kiali-operator/blob/v1.26/deploy/kiali/kiali_cr.yaml#L481-L520
Grafana: https://github.com/kiali/kiali-operator/blob/v1.26/deploy/kiali/kiali_cr.yaml#L395-L449
Jaeger: https://github.com/kiali/kiali-operator/blob/v1.26/deploy/kiali/kiali_cr.yaml#L522-L567

[pslobo]

Thanks @jmazzitelli but I already have that and still see the issue:

Currently, I have:

• istiod, istio-[ingress|egress]gateway in istio-system NS
• Prometheus, kiali, grafana, jaeger in observability NS

This is what I have configured (relevant sections)

 external_services:
      grafana:
        auth:
          ca_file: ""
          insecure_skip_verify: false
          password: "*****"
          token: ""
          type: basic
          use_kiali_token: false
          username: "*****"
        dashboards:
        - name: Istio Service Dashboard
          variables:
            namespace: var-namespace
            service: var-service
        - name: Istio Workload Dashboard
          variables:
            namespace: var-namespace
            workload: var-workload
        enabled: true
        namespace: "observability"
        in_cluster_url: http://grafana.observability:3000
        url: "http://localhost:3000"
      istio:
        istio_identity_domain: svc.cluster.local
        istio_sidecar_annotation: sidecar.istio.io/status
        istio_status_enabled: true
        url_service_version: http://istiod.istio-system:15014/version
      prometheus:
        auth:
          ca_file: ""
          insecure_skip_verify: false
          password: ""
          token: ""
          type: none
          use_kiali_token: false
          username: ""
        namespace: "observability"
        custom_metrics_url: http://prometheus-k8s.observability:9090
        url: http://prometheus-k8s.observability:9090
      tracing:
        auth:
          ca_file: ""
          insecure_skip_verify: false
          password: ""
          token: ""
          type: none
          use_kiali_token: false
          username: ""
        enabled: true
        namespace: "observability"
        in_cluster_url: http://jaeger-query.observability:16686
        namespace_selector: true
        url: ""
        whitelist_istio_system:
        - jaeger-query
        - istio-ingressgateway

[jmazzitelli]

I can't say what is wrong - but I can say, briefly looking, at minimum the grafana setting looks wrong. Why is "url" a localhost URL? That can't possibly work, can it?. From the link I provided, the docs say:

url: The URL that Kiali uses when integrating with Grafana. This URL must be accessible to clients external to
      the cluster in order for the integration to work properly. If empty, an attempt to auto-discover it is made

My suggestion is to read the docs (the ones I linked to earlier) very carefully and review all your "url" and "in_cluster_url" settings (and look at the "component_status" settings that you do not set - I believe you are going to need them if your components are in other namespaces - the defaults won't be correct for you).

If you still can't get it, feel free to start a conversation on the kiali-users google group or the Istio slack #kiali room - see: https://kiali.io/contribute/#_talk_with_us ... you will want to provide folks with what you are seeing (error messages, screen shots, etc) and what the Kiali pod logs show (if it has any errors or warnings related to this).

[pslobo]

Thanks @jmazzitelli. It was my understanding the the url in the grafana settings was just to add a link to access the grafana instance directly from the Kiali via, and Kiali uses in_cluster_url for the rest. I'll continue to tinker with this and review my settings a couple more times, but if I'm still stuck I'll reach ou on the group.