Node_exporter: alert annotations: message or description?

[mmallet-digipok]

What did you do that produced an error?

Not really an error but I think it would be worth it to have details in the documentation. I noticed that alerts do not use the same annotation to hold the alert content. I don't know whether it is normal design.

What did you expect to see?

Check the online documentation example about alert annotations: https://www.prometheus.io/docs/alerting/latest/notifications/

{
  summary: "alert summary",
  description: "alert description",
}

The documentation suggests that the alert description is set in the description attribute.

What did you see instead?

In the alerts.libsonnet, alerts use description and also message:

            annotations: {
              summary: 'Filesystem is predicted to run out of space within the next 24 hours.',
              description: 'Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left and is filling up.',
            },

            annotations: {
              summary: 'Clock skew detected.',
              message: 'Clock on {{ $labels.instance }} is out of sync by more than 300s. Ensure NTP is configured correctly on this host.',
            },

[discordianfish]

It should be description, will be fixed by https://github.com/prometheus/node_exporter/pull/1764

[adinhodovic]

@discordianfish Quick question, the kubernetes-mixin project just uses message and no description or summary. Will there be an unified approach and if so, which one will be the default approach?

[roidelapluie]

cc @brancz

[brancz]

Long time dispute. I and many others prefer message, but there are others who prefer description and summary. I would also prefer if we can just agree on one of them. I personally have never found the description and summary distinction useful, as most people tend to just write the same things with little value in duplicating.

[adinhodovic]

Long time dispute. I and many others prefer message, but there are others who prefer description and summary. I would also prefer if we can just agree on one of them. I personally have never found the description and summary distinction useful, as most people tend to just write the same things with little value in duplicating.

I agree and prefer message too. Is there an issue I can track this discussion/progress on the approach?

[brancz]

I think there was a google doc at one point where this was being discussed. @brian-brazil @beorn7 @RichiH @gouthamve @codesome @bwplotka thoughts? :)

[brian-brazil]

I'm not sure we've really discussed this, historically this comes from Borgmon where these were first class fields and subsequently also in Prometheus for a good while too.

Even if this were a good idea, changing what is now a pattern that has existed for 7+ years within the community seems like causing needless confusion and breakage.

[brancz]

Maybe I'm misremembering and mistaking it for discussions around the severity label.

The divide already exists, for the community converging on one at least in the mixins would be helpful I believe. This is may be subjective, but it appears to me that message is much less contentious, as there is no dilemma of choice of what's appropriate to put in and what not, which is my observation with summary and description.

[kfdm]

I believe AM tool defaults to showing summary in it's output amtool alert

❯ amtool alert 
Alertname                 Starts At                Summary  
...

I'm not aware of any other preference for message vs description in any other tools

[beorn7]

The outcome of the discussion long ago (about which fields to make first class and which not) ended with the result that it's impossible to find an agreement because organizations just have too different ideas and requirements.

Having said that, aligning the mixin behind one common practice makes sense.

What I have evangelized with some success (but I guess never codified publicly) is the summary/description divide with the rule of thumb that summary should be the same for all alerts typically grouped into a notification while description contains the parts that may vary. In that way, the summary can serve as some kind of headline (e.g. "Disk close to capacity") while the description is something you can enumerate per alert in a list (e.g. list all the individual disks that are close to capacity).

[brancz]

At this point, I'm happy with whatever as long as we're consistent :)

Do you want to add that to the mixin design and then we can iterate over the existing mixins and align?

[brian-brazil]

Then it sounds like once #1764 is merged we'll be good here.

[paulfantom]

Please remember that alert name is also meant to carry some information and as such having discussion about summary/description vs message is essentially a discussion of how many layers of context information an alert should have (3 vs 2).

From my personal experience having a meaningful alert name causes summary field to be unnecessary and confusing (this confusion can be seen when looking at already present monitoring mixins). Simplifying that to alert name + description/message is creating a framework that is much easier to comprehend and reason with.

Additionally, since all mixins are written in jsonnet, it is always possible to extend annotations to match what users have in the organization. What we need is a generic framework that can be easy to understand.

[beorn7]

From my personal experience having a meaningful alert name causes summary field to be unnecessary and confusing (this confusion can be seen when looking at already present monitoring mixins). Simplifying that to alert name + description/message is creating a framework that is much easier to comprehend and reason with.

That's certainly a valid opinion. The problem is that others have other opinions, and a lot of discussions in the past along exactly the lines you are bringing up here haven't created a consensus. In the end, some people prefer what you said, others prefer the name/summary/descriptions hierarchy. And then there are still other ideas around that some other people keep preferring.

Additionally, since all mixins are written in jsonnet, it is always possible to extend annotations to match what users have in the organization. What we need is a generic framework that can be easy to understand.

Following that line of argument, I'd err on the side of having more annotations. Pruning annotations I don't want is much easier than creating annotations I do want out of thin air.

I'll create a PR against https://github.com/monitoring-mixins/docs with a suggestion. We can then take it from there.

[beorn7]

Done, see
https://github.com/monitoring-mixins/docs/pull/10