Prometheus: promtool rules migration produces unreadable rule files

[grobie]

The current rules files migration tool produces output files which are unusable for us.

Smaller issues:

• order of annotations is not preserved, but alphabetical order enforced
• regular expression anchoring is printed verbatim "^(?:ext.|xfs)$" instead of "ext.|xfs"

Bigger issue:

• Any sequence of whitespace is truncated to a single space, the resulting expression is written as one long string to the file. We have longer expressions which spawn 8 lines without any kind of formatting and therefore, are completely unreadable for a human now. Either the original formatting or a new promtool fmt formatting should be used here.

[fabxc]

First one seems hard to fix given how the Go maps and probably the YAML lib work.
Second on is fixable.
Third one probably not either with the YAML lib alone.

How are we feeling about our decision overall? So far it seems there's some migration pain and YAML isn't free of issues either but generally I've heard several people welcome the decision as it's easier to work with in general.

[brian-brazil]

The indentation issue was unexpected, but doesn't seem to have caused problems overall so I think YAML is still the way to go.

[alin-amana]

FYI, I migrated my (handful of) rules manually and reached some conclusions regarding how to combine YAML, go templates and Prometheus' rule language to get something that's (1) easy to migrate and (2) easy to understand and thus maintain. I still find the combination of the 3 different formats/languages with wildly different newline, quoting and whitespace requirements a huge and unnecessary PITA, but that's not the point.

This approach may be useful for writing an automated tool (way too much for my own needs) or for aiding in a manual migration. I'll start off with a rather complex sample migrated rule so you can appreciate whether it's worth the effort or not:

- alert: TasksMissing
  expr: |
    (
      # Compare with a matching job-specific threshold, keeping all labels on the left
      job_env:up:ratio
        <= on(job) group_left()
      job:up:ratio_critical_threshold
    ) or (
      # Or select all jobs without a matching job-specific threshold
      (
        job_env:up:ratio
          unless on(job)
        job:up:ratio_critical_threshold
      )
      # And compare with the default threshold, keeping all labels on the left
        <= on() group_left()
      job:up:ratio_critical_threshold{job=""}
    )
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Tasks missing for {{ $labels.job }} in {{ $labels.env }}
    description:
     '{{ with printf `job_env:up:count{job="%s",env="%s"} - job_env:up:sum{job="%s",env="%s"}` $labels.job $labels.env $labels.job $labels.env | query }}
        {{- . | first | value -}}
      {{ end }}
      of
      {{ with printf `job_env:up:count{job="%s",env="%s"}` $labels.job $labels.env | query }}
        {{- . | first | value -}}
      {{ end }}
      {{ $labels.job }} instances are missing in {{ $labels.env }}:
      {{ range printf `up{job="%s",env="%s"}==0` $labels.job $labels.env | query }}
        {{- .Labels.instance }}
      {{ end }}'
    debug: '{{ externalURL }}{{ printf `up{job="%s",env="%s"}==0` $labels.job $labels.env | graphLink }}'

Note how it allows you to keep indentation, newlines and comments even though YAML tries really hard to prevent it. It basically uses literal block notation and single quoted flow notation in different places and limits specific quotation characters to specific uses.

The basic ideas are

1. Use literal block notation (the expr: | bit) for expressions and indent the whole block past the expr. This allows you to use whatever formatting/indentation and add comments that will be parsed and dropped by Prometheus (this is why I don't really buy the selling point of transitioning to YAML -- you still have a domain specific language with comments and all).
2. Use single quoted flow notation (e.g. debug: ' or description:\n ') for labels and annotations, and limit single quotes to YAML use, double quotes for Prometheus strings (as in job="foo") and backticks for Go template strings (e.g. the printf bits). This will convert all whitespace (including newlines) to single blanks, and if you combine it with Go template trimming (using {{- and -}} instead of {{ and }} where needed, you can again format stuff (e.g. loops) the way you please.

All in all very simple and natural.\

But if anyone was willing (and had gobs of time) it would be possible to automate at least parts of it.

[beorn7]

This is obsolete as the rule migration tool is deprecated by now and taken out of current releases.

However, pretty formatting of rules is still a thing, see #5370 . Still, closing this one to avoid confusion.