Kube-prometheus: How to use other scrape_configs than kube_sd_config

[grobie]

Instead of adding a short static configuration, the current setup creates a dummy service just so that all services get discovered via Kubernetes.

- job_name: etcd
  static_configs:
  - targets: [172.17.4.51:2379]

This indirection seems to be just overhead. I assume it's the result of trying to completely generate prometheus.yml at some point. I think there should still be some way to provide non-kubernetes_sd_configs in that case.

[brancz]

The idea is that by knowing kubernetes objects, you know how to configure monitoring for it. If there is an endpoints list, for it, it will be monitored.

However, I can see how it might be good (especially during migration to kubernetes for example) to be able to extend the configuration. Maybe however it is enough that someone can then skip the configmap being generated and just specify the whole config for themselves when that is what they need. Wdyt @fabxc ?

[fabxc]

I don't find this unidiomatic at all. Wrapping something into a headless service is an absolutely straightforward and idiomatic way to make something outside of the cluster discoverable on the inside.
That is aligned with the idea to make monitoring available to the user within Kubernetes concepts.

That said, what we are doing in kube-prometheus is collecting all the boilerplate and examples to get end-to-end monitoring of your cluster components running. In the end it relies on a static config, which the user can modify however he likes. Instead of headless services he can use your snippet above any time he wants.

[grobie]

From a Prometheus configuration point of view, having to create a service and endpoints manifests just to create a static endpoint doesn't feel straightforward. The idiomatic Prometheus way is to add the three lines outlined above, now users need to be able to write such manifests on their own.

While the endpoints manifest approach can be used for static_configs, it doesn't work out-of-the-box for other dynamic sd setups. Will users only be able to benefit from ServiceMonitors and the full feature set of the prometheus-operator if all their monitoring targets can be discovered via the Kubernetes API, or are there plans to support non-kubernetes jobs as well?

[fabxc]

I think the vast majority of cases will be discovering things inside of the cluster.
The rest will be stuff that hasn't been migrated yet or isn't intended to. For those cases I think it's fine to have users specify their own configurations, which is totally supported by the operator. It's just not realistic or feasible to rebuild all Prometheus SD integrations as Kuberentes TPRs.

[brancz]

Targets that are not running as a Pod within Kubernetes can always be discovered by adding those targets to an Endpoints object. Anything that requires more customization can be achieved by not selecting ServiceMonitors in the Prometheus spec and specifying a completely custom configuration.

Will close this here. Feel free to reopen if you think further discussion is necessary.

[dayglojesus]

For anybody else down the "How do I add more scrape_configs?" rabbit hole, I leave this breadcrumb ...
https://github.com/coreos/prometheus-operator/blob/master/Documentation/additional-scrape-config.md