Helm-charts: [kube-prometheus-stack] Make the chart work with Istio

[irizzant]

Describe the bug
With Istio 1.7.3 there are different issues with this chart.

• Alertmanager cluster is not forming up:
  level=warn ts=2020-10-08T07:48:13.572Z caller=cluster.go:438 component=cluster msg=refresh result=failure addr=10.233.77.22:9094 err="1 error occurred:\n\t* Failed to join 10.233.77.22: EOF\n\n"

Even if I enable listenLocal on both Prometheus and Alertmanager the latter is started exposing the mesh port 9094 to the pod ip address.
There should be an option to enable listenLocal behaviour for the mesh port too.

• Conflicting listeners:

Conflicting inbound listener:9093. existing: kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local, incoming: kube-prometheus-stack-alertmanager-0.monitoring.svc.cluster.local
Conflicting inbound listener:9093. existing: kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local, incoming: kube-prometheus-stack-alertmanager-1.monitoring.svc.cluster.local
Conflicting inbound listener:9090. existing: kube-prometheus-stack-prometheus.monitoring.svc.cluster.local, incoming: kube-prometheus-stack-prometheus-1.monitoring.svc.cluster.local
Conflicting inbound listener:9090. existing: kube-prometheus-stack-prometheus.monitoring.svc.cluster.local, incoming: kube-prometheus-stack-prometheus-0.monitoring.svc.cluster.local

This is because there are multiple services pointing to the same port.

alertmanager-operated                            ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   11h
prometheus-operated                              ClusterIP   None            <none>        9090/TCP,10901/TCP           11h
kube-prometheus-stack-alertmanager               ClusterIP   10.233.62.114   <none>        9093/TCP                     11h
kube-prometheus-stack-prometheus                 ClusterIP   10.233.1.147    <none>        9090/TCP                     11h

• Service's port names don't respect Istio conventions.
  I was able to fix some wherever possible using portNames but we still have:
  Info [IST0118] (Service kube-prometheus-stack-prometheus-node-exporter.monitoring) Port name metrics (port: 9100, targetPort: 9100) doesn't follow the naming convention of Istio port.

• Service for Thanos is incorrectly defined

spec:
  clusterIP: None
  ports:
  - name: http-web
    port: 9090
    protocol: TCP
    targetPort: http-web
  - name: grpc
    port: 10901
    protocol: TCP
    targetPort: grpc
  selector:
    app: prometheus

it defines only the 10901 but not the 10902 port so Istio doesn't create any listener for it.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Which chart: prometheus-kube-stack
Which version of the chart: 9.4.9

What happened:
Chart is not working properly with Istio

What you expected to happen:
Chart should work properly with Istio

How to reproduce it (as minimally and precisely as possible):
Deploy the chart with Istio 1.7.3 and Istio auto injection enabled.
Observe the issues listed above.

Anything else we need to know:

[bismarck]

@irizzant Alertmanager requires UDP for clustering to work and Istio does not currently support UDP.

[irizzant]

@bismarck actually the fact that Istio does not support UDP at the moment does not mean that the Alertmanager cluster is impossible to form up.
You can avoid Istio to proxy the UDP port, which is exactly the same approach taken in the upcoming Istio 1.8 release:

Non-TCP based protocols, such as UDP, are not proxied. These protocols will continue to function as normal, without any interception by the Istio proxy but cannot be used in proxy-only components such as ingress or egress gateways.

This means that we have a possible workaround for point

• Alertmanager cluster is not forming up

others are still pending

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[irizzant]

activity!

[monotek]

Would be nice on of you guys could create a pr, keeping the old config as default.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[bryankaraffa]

Istio provides a mechanism to mark certain pods exempt from pod injection based on their labels and that has worked well for the admission-webhook jobs. It seems alertmanager still has issues, and I am working on this here and there.....

[monotek]

Would be nice you could share your solution :)

[bryankaraffa]

Here's the example of how to exempt pods from the istio sidecar injection. This configures the neverInjectSelector on istiod to never inject the sidecar into pods with label app=kube-prometheus-stack-admission-create or app=kube-prometheus-stack-admission-patch, which resolves the issue with the kube-prometheus-stack admission webhook jobs that can't exit as expected with the istio sidecar.

sidecarInjectorWebhook:
  neverInjectSelector:
  # kube-prometheus-stack
  ## Admission Webhook jobs do not terminate as expected with istio-proxy
  - matchExpressions:
    - {key: app, operator: In, values: [kube-prometheus-stack-admission-create,kube-prometheus-stack-admission-patch]}

Official docs for this mechanism: Istio / More control: adding exceptions

Some of the other issues I saw with istio sidecar injection enabled was related to prometheus <-> alert manager not reaching each other...

For now I have disabled istio pod injection on the monitoring namespace / kube-prometheus-stack and plan to investigate it later when we want Prometheus to use mTLS as well