Loki: Extract metrics in promtail

[SuperQ]

mtail is extremely powerful for extracting metrics from logs for systems that don't have good whitebox options. It would be nice if promtail could integrate mtail's parsing system and produce metrics based on the logs.

[ventris]

Another thing would be to also expose metrics about the promtail exporter. Such as input records, output errors etc.

[SuperQ]

@ventris promtail already includes metrics about itself.

[tomwilkie]

@ventris we've exposed a bunch of metrics from promtail already, and have ideas for more (see #327). If you have any suggestions, let me know! We also have example dashboards from promtail here: https://github.com/weaveworks/common/pull/146

@SuperQ we're starting to think about where to expose metrics from, and supporting mtail like usecase is top of our list. We're also looking at https://github.com/fstab/grok_exporter, have you seen that?

Open questions:

• where is the best place to do this? promtail seems obvious, but we also support fluentd-based ingestion, and would be nice to grab metrics from that too. This implied doing it in Loki might be preferred. Perhaps we can do it in both?
• how to do this? were starting to think about adding a query language, and we could consider supporting prometheus-style recording rules in this language, and use them to record metrics from logs. For instance, recording count(logs{job="foo"}[1m] | "^bar$") as a recording rule of the number of records which match "^bar$" in the last 5mins, correctly propagating labels etc...

@slim-bean is thinking about this too.

[SuperQ]

@tomwilkie Yes, I've seen the grok_exporter as well.

My usual suggestion for where to produce metrics from logs is at the closest point to ingestion. To me it makes sense to do this in promtail, as it is reading, tagging, and compressing logs. I don't think it would be good to waste CPU time decompressing logs just to process them for metrics. Also doing things in promtail makes things scale better. Doing things in the Loki service means having to scale it up much more to handle the processing.

I personally like the mtail way of being able to define extractions via regexp. There are lots of cases where the log lines also contain numbers to extract.

[slim-bean]

With the 0.1.0 release there is included a pipeline which allows for extracting data from log messages using regex/json and further creating metrics from this data