Alertmanager: match_re matches missing labels with '.*' regexp

[mouzfun]

Giving the configuration possibilities, the most straightforward method to test for the missing label to me is to test it like this

# match if label exists
match_re:
  label: '.*'

But currently we treat empty labels as empty strings, so regex.match("") will be true in this case.
Yes, this can be worked around easily with an \S+ regexp, but i spent 1 hour of my time hunting this :)

To me it seems unintuitive and wrong from the user perspective - how does the missing label can possibly be matched with anything.

I threw together a quick dirty patch which fixes the issue with couple of simple tests here.
Not sure which path forward is better thought, something along the lines of my patch, maybe new match_exists config param or something else entirely.

I would actually happily try to implement this given a little mentoring and contribute to the project:) Or if you awesome folk wish to grab this one this is no problem also.

My use case is simple and goes like:

route:
  receiver: 'alerta'
  repeat_interval: 5m
  routes:
    - match_re:
        notify: '.*'
      receiver: 'zulip'
# omitted for brevity
receivers:

I want to send all alerts with the notify label to the zulip, and everything else to the alerta receiver, as a user i expect that missing label wont be matched with any regexp, and i always use '.*' for such cases, which i guess is a habit.

Also i'm not sure if alertmanager allows labels with empty string, this also would probably factor in.

[simonpasquier]

Labels with empty value are discarded by the Alertmanager's API so there's no way that an empty label value exists ever.
https://github.com/prometheus/alertmanager/blob/e4437ab54f26dbcd66b90eaf4d5701fee98a1da6/api/v2/api.go#L321-L330
You can check against ".+" if you want to match any alert containing a given label without caring about the value.

[mouzfun]

I understand that there are workarounds, but as i said originally:
To me it seems unintuitive and wrong from the user perspective - how does the missing label can possibly be matched with anything.

I think that its clearly a bug.

[beorn7]

As unintuitive as it might appear to you, it's one of the very fundamentals of the Prometeheus data model: A label with an empty label value is considered equivalent to a label that does not exist. If you consider that a bug, you'd need many many PRs in many different repos to "fix" it, and new breaking major releases of many Prometheus projects.

[mouzfun]

Okay, i did not realize that this behaviors is not limited to the alertmanager.

I tested prometheus queries, and there are indeed same thing happening

series{missing_label=""} returns all series, even without label present.

Is there any reason for this, though?
Apart from the fact that the go map implementation returns empty strings for missing keys? :)

[beorn7]

We tried various things during the evolution of the data model. We had to settle on one way, and the one we settled on appeared as the most suitable. (Or in other word: It was a (very) informed decision, not just an ad-hoc one ore one out of coding convenience.)

[mouzfun]

Very well. Perhaps we can still improve user experience by mentioning it in the alertmanager docs? I don't think that that's a such a rare case to test for label presence.

By the way, docs for the data model also do not mention that we treat missing labels as empty string

[beorn7]

It's sprinkled over the documentation in various places. PRs to clarify things will certainly be welcome.