Helm-charts: [kube-prometheus-stack] tlsConfig support for servicemonitors and default alertingEndpoints to work with istio strict mtls

[infa-ddeore]

Is your feature request related to a problem? Please describe.
prometheus stack installed with istio sidecar with STRICT mtls policy
1) prometheus cant scrape grafana, alertmanager, kube-state-metrics and prometheus-operator
2) prometheus cant connect to alertmanager

Describe the solution you'd like
1) Provide scheme and mtls options in grafana, alertmanager, kube-state-metrics and prometheus-operator servicemonitor definition
2) provide an option to update alertmanager's scheme and tlsconfig

Describe alternatives you've considered
1) manually update the default servicemonitors with https scheme and tlsConfig using istio tls certs
2) use prometheus. prometheusSpec.alertingEndpoints https and tlsconfig

Additional context
below is the override values (considering "prom" is the installed helm chart name, eg. helm install prom) that makes istio write the tls cert on the memory volume and prometheus mounts that volume
NOTE:

Prerequisites:
1) install istio 1.6.1 or higher
2) enable istio sidecar injection (eg. label the namespace with istio-injection=enabled)

Istio 1.6.1 and higher needs to be installed to make it work, prometheus will have istio certs mounted under /etc/istio-certs/ dir, once PeerAuthentication is set to STRICT, prometheus will fail to scrape connect to other components like grafana, operator, alertmenager.

prometheusOperator:
  podAnnotations:
    sidecar.istio.io/rewriteAppHTTPProbers: "true"
  admissionWebhooks:
    patch:
      podAnnotations:
        sidecar.istio.io/inject: "false"

grafana:
  podAnnotations:
    sidecar.istio.io/rewriteAppHTTPProbers: "true"

prometheus:
  prometheusSpec:
    logLevel: info
    alertingEndpoints:
    - name: prom-prometheus-operator-alertmanager
      namespace: default
      port: web
      scheme: https
      pathPrefix: /
      apiVersion: v2
      tlsConfig:
        caFile: /etc/istio-certs/root-cert.pem
        keyFile: /etc/istio-certs/key.pem
        certFile: /etc/istio-certs/cert-chain.pem
        insecureSkipVerify: true    
    volumeMounts:
    - name: istio-certs
      mountPath: /etc/istio-certs/
      readOnly: true
    podMetadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
        sidecar.istio.io/userVolume: '[{"name": "istio-certs", "emptyDir": {"medium": "Memory"}}]'
        sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-certs/"}]'

        proxy.istio.io/config: |-
          proxyMetadata:
            OUTPUT_CERTS: /etc/istio-certs/     

alertmanager:
  alertmanagerSpec:
    podMetadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"

  serviceMonitor:
    tlsConfig:
      caFile: /etc/istio-certs/root-cert.pem
      keyFile: /etc/istio-certs/key.pem
      certFile: /etc/istio-certs/cert-chain.pem
      insecureSkipVerify: true

kube-state-metrics:
  podAnnotations:
    sidecar.istio.io/rewriteAppHTTPProbers: "true"

edit servicemonitor to make prometheus scrape the target over https with tlsconfig as below kubectl edit servicemonitors.monitoring.coreos.com prom-prometheus-operator-alertmanager, now prometheus can scrape the target over mtls.

spec:
  endpoints:
  - path: /metrics
    port: web
    scheme: https
    tlsConfig:
      caFile: /etc/istio-certs/root-cert.pem
      certFile: /etc/istio-certs/cert-chain.pem
      insecureSkipVerify: true
      keyFile: /etc/istio-certs/key.pem
  namespaceSelector:

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[infa-ddeore]

@bismarck @gianrubio @gkarthiks @scottrigby @vsliouniaev @Xtigyro
could you take a look at this feature request?

it will help many who want to use promeththeus operator with istio STRICT mTLS between prometheus component

[bismarck]

@infa-ddeore We would be happy to accept a PR implementing this feature.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[stevehipwell]

Is nobody still interested in this?

[RenaudDemarneffe]

Yes, I'm very interested in this.

[uvw]

I end up using the Helm post-renderer approach with kustomize to patch the chart on the fly. Here is a distilled example repo: https://github.com/uvw/kube-prometheus-stack-istio.

[stevehipwell]

Looking into the _Istio_ code and docs further it looks like Metric merging is the supported way of handling metrics with MTLS set to strict. I've asked for clarification on the correct pattern to implement this.