Kapacitor: Implement Rate limiting for sending alerts

[nathanielc]

If a Kapacitor task starts to create a siginificant volume of alerts its not helpful to just keep spamming the alerting integration endpoints. Kapacitor should have some kind of rate limit per alert integration where we don't send alerts if Kapacitor is being to spamy.

[elvarb]

The possiblity of grouping alerts into a single alert would be helpful instead of just rate limiting the output.

[nathanielc]

@elvarb I am looking for feedback on how you would like the grouping of alerts to behave. Do you have any suggestions on how you would want this to behave? I have a few ideas ...

• Simplest but probably not that useful would be to concatenate all the alert messages into one large message.
• Use a template to allow defining the message yourself. For example: {{ .NumAggregated }} alerts are firing\nAlerts {{ range .Alerts }} {{ .Message}} {{end}}
• Something else entirely?

[bfgoodrich]

It would be nice if you could aggregate based on time period and tags in the alert. (e.g. create a single aggregate alert for a count of all alerts of type foo on host type bar within the last minute)

[imcom]

I think it would be nice to set a threshold for each alert targets in tick definition and then when whichever alert exceeds its threshold then kapacitor can start being lazy to send out the repeated alerts (all in the same level) by increasing the interval (simply swallow repeated alerts in between)

[lblt]

Hi, I perfectly agree with @bfgoodrich
Have new features been implemented in 1.3? @nathanielc

[m4ce]

@nathanielc, is Kapacitor still not offering a way of rate limiting alerts? I am trying to dig it up in the docs but can't find anything that indicates that..

[zabudskyi]

@m4ce, have you found any workaround?
It would be great if it worked like @bfgoodrich suggested

[crazy-canux]

Any updates?

[scottwilson312]

I'm also interested in rate limiting alerts

[badhop]

It will be great to have such mechanism. Instead of storm message send aggregate statistic with number of suppressed alert messages grouped by AlertName every N minutes.

[poojabms]

Any updates on this? I Am also interested in :
I think it would be nice to set a threshold for each alert targets in tick definition and then when whichever alert exceeds its threshold then kapacitor can start being lazy to send out the repeated alerts (all in the same level) by increasing the interval (simply swallow repeated alerts in between)

[gh94uk]

This feature would be very useful for us!