Prometheus: Macros for PromQL

[jutley]

Proposal

There are many patterns for rather elaborate PromQL queries that I use across many metrics and at many points in time. For example, I've written some very complex queries based on this article:
https://about.gitlab.com/blog/2019/07/23/anomaly-detection-using-prometheus/

These queries can be very hard to reproduce, and it is nearly impossible for new users in our org to be able to understand these. Using a Prometheus recording rule doesn't quite make sense, as we don't necessarily know where or when we'll want to use the pattern, and it is too expensive to apply the rule to every metric.

To handle these use cases, it would be nice as a Prometheus user to be able to configure Prometheus with macros. These can be parameterize, and expand to a new piece of PromQL. This can make is significantly easier for complex queries to get used more often.

I realize this could get pretty complicated, and could have performance issues as macros are stacked. Nonetheless, I think it would be really nice!

Example

As an example, say I want to alert if results for a query go absent at some point. Let's say the query is this:

sum by (a,b) (rate(my_metric_count{foo="bar"}[5m]))

To detect when this goes missing, I can use this query:

max_over_time((timestamp(
  sum by (a,b) (rate(my_metric_count{foo="bar"}[5m]))
))[24h:]) 
unless
max_over_time((timestamp(
  sum by (a,b) (rate(my_metric_count{foo="bar"}[5m]))
))[10m:]) 

I'd like to be able to define this pattern as a macro, which may look something like this:

macro query_result_becomes_absent(query, search_window, absent_window):
max_over_time((timestamp(
  query
))[search_window:]) 
unless
max_over_time((timestamp(
  query
))[absent_window:]) 

Then, I may invoke the macro like this:

query_result_becomes_absent(
  sum by (a,b) (rate(my_metric_count{foo="bar"}[5m])), 
  24h, 
  10m)

[brian-brazil]

I don't see how these examples are saving you anything - and in fact for me only make the query harder to understand. In general this is something to handle on top of Prometheus, such as via configuration management or Grafana templates.

I also note that those queries are much more complex than they need to be.

[bwplotka]

Hi and thanks for this proposal. I totally agree with the benefits that Macros gives, but in the same time, you can already make the same macros outside of PromQL in any language of your choice as @brian-brazil mentioned. As an example in the Thanos project we use jsonnet mixin to generate resources like alerts, rules or dashboards e.g. this awesome HTTP jsonnet lib with useful HTTP queries (out of the box!) that generates our dashboard like this: https://github.com/thanos-io/thanos/blob/master/examples/dashboards/query.json

I also note that those queries are much more complex than they need to be.

I think so, too.

[jutley]

Handling this in a separate language is a good suggestion. I'll explore that more.

I agree that the example I shared is a bit ridiculous. The anomaly detection article has much more realistic examples, though it would make this a significantly larger post.