Telegraf: Support Prometheus Operator

[rawkode]

Feature Request

Currently, Telegraf supports the Prometheus annotations for service discovery in Kubernetes. With CRDs hitting GA, there's growing support for this workflow within the Kubernetes community; particularly around Prometheus.

The Prometheus Operator provides CRDs for directing Prometheus servers to metric endpoints, which Telegraf should support too.

Proposal:

Support the ServiceMonitor CRD, with the same behaviour as the annotation approach.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web

Current behavior:

Not supported

Desired behavior:

Scrape metrics as per instructions in ServiceMonitor CRD

Use case:

The Prometheus Operator is becoming the standard for Kubernetes monitoring and Telegraf could provide a drop-in replacement

[danielnelson]

High level sounds good, I'm assuming you are referring to the CoreOS Prometheus Operator? Can you pseudocode the calls we would make and what example responses would look like?

[M0rdecay]

@danielnelson, you need to support the definition of the destination port, getting them not from labels, but from spec.ports[%PORT_NAME%]

Example service manifest:

apiVersion: v1
kind: Service
metadata:
  name: pili
  labels:
    app: pili
spec:
  type: ClusterIP
  ports:
  - name: uwsgi
    protocol: TCP
    port: 8080
    targetPort: uwsgi
  selector:
    app: pili-web
    tier: backend

Then the configuration of Prometheus is something like this:

[[inputs.prometheus]]
...

kubernetes_lablel_selector = "appl=pili"
kubernetes_monitor_endpoint = "uwsgi"
kubernetes_monitor_path = "/metrics"
kubernetes_monitor_scheme = "http"
...

In this form, it will be necessary to create one input line per service, but this is seen as a plus rather than a minus.
It might be worth using an array of endpoints:

[[inputs.prometheus]]
  [[inputs.prometheus.endpoints]]
    selector = "appl=pili"
    endpoint = "uwsgi"
    path = "/metrics"
    scheme = "http"
...

But, as it seems to me, such a refinement will cost more.

ServiceMonitor configuration example for the above service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: monitoring-pili
  namespace: monitoring
  labels:
    app: pili-service-monitor
spec:
  selector:
    matchLabels:
      # Target app service
      app: pili
  endpoints:
  - interval: 15s
    path: /metrics
    port: uwsgi
    scheme: http
  namespaceSelector:
    matchNames:
    - default

This logic allows you to use the same principle of collecting goals as ServiceOperator.

[M0rdecay]

Or you can use the API of the prometheus operator - https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#servicemonitor

But this will require a deployed prometheus operator in the cluster, which, in some scenarios, I would like to avoid.

[rawkode]

@M0rdecay work on this support actually started last week! We'll keep you updated with this issue 👍

[vavdoshka]

hey guys! any news on that effort?