Elastalert: Unique run_every value per elastalert rule

[rashmichoudhary07]

@Qmando
Hello @Qmando ,
I read in the documentation and a few issues at github that run_every property, when specified per rule overwrites this property in elastalert's global config.yaml.
However in practice the run_every in config.yaml is considered by elastalert for every rule that i create.
I am using python 3.5 , ES 7.4 and elastalert 0.2.4
Is there something that i am missing. I need to enable the unique value of run_every property in each rule such that it is considered. Is this feature already available per rule file? If not what must be done.

Please assist!
Thanks in advance.

[rashmichoudhary07]

@nsano-rururu @Qmando
Looked into it. Still dont know why the run_every per rules aint working. :(

[nsano-rururu]

The common settings that cross the individual monitoring settings are config.yaml. As for the setting value, the individual monitoring setting has priority, and if it is not in the individual monitoring setting, it will be the relationship referenced from config.yaml of global config.

[rashmichoudhary07]

@nsano-rururu
Thanks!
So this means if i specify unique run_every value in my rules it should override the one in config.yaml right?

@Qmando
Is the understanding correct? Please provide your inputs too.

[rashmichoudhary07]

i came across limit_execution property. Is it available in the 0.2.4 Version and handles all timestamps.
i still can't figure out a way for run_every to work per rule. It would be really helpful if we have anything to schedule rules other than using run_every in config.yaml.

[nsano-rururu]

@rashmichoudhary07

I thought it would be better to use a third party ElastAlert Kibana Plugin or Praeco instead of using ElastAlert directly.

By the way, the development of ElastAlert Kibana Plugin and Praeco seems to be stopped, so it is difficult to actively recommend it.

[nsano-rururu]

@rashmichoudhary07

Where is the alert notification destination?
https://elastalert.readthedocs.io/en/latest/ruletypes.html?highlight=hours#alerts

[nsano-rururu]

@rashmichoudhary07

ElastAlert Kibana Plugin

Kibana 7.5.0 is officially supported. If you want to use it in a higher version, there is a workaround that I will support
https://github.com/bitsensor/elastalert-kibana-plugin

Example

Praeco

For the full term, only Slack, Email, Telegram, Jira or an HTTP POST can be set on the screen
https://github.com/johnsusek/praeco
https://medium.com/@john_8166/praeco-walkthrough-5aada7e078a9

Example

[nsano-rururu]

@rashmichoudhary07

What information is stored in Elasticsearch as long as it can be published (for example, the log of your own application is sent to Elasticsearch with Fluentd etc.). I'm not so familiar with ElastAlert because it was the level I investigated during surveillance for work for about half a year, but if you have the information, you can check the operation and investigate. Is it possible to tell us about the content of the alert rules? If it is not possible to publish it, there is no problem with saying that it is impossible.

The following is an example of run_every

Example 1) 1 second

run_every:
     seconds: 1

Example 2) 1 minute

run_every:
     minutes: 1

Example 3) 1 hour

run_every:
   hours: 1

[rashmichoudhary07]

@nsano-rururu
Thanks for the inputs. Kibana plugin looks good.
However, in my case i am bound to use elastalert directly.
PFB my rule file for reference.

alert: post
attach_related: true
category: Critical
filter:
term:
log_json.response.statusCode: '500'
http_post_payload:
conversationId: xyz
relatedEvents: related_events
http_post_static_payload:
name: OrderManagement 500
http_post_url:
include:
xyz
index: xyz
name: xyz
num_events: 1
run_every:
minutes: 1
timeframe:
minutes: 30
type: frequency

However, the run_every property in my rule is not getting picked up. It is still picking up the run_every from config.yaml(global config). A few posts mention that run_every is allowed per rule now. However, it does seem to work. Am i missing something?

Thanks!

[nsano-rururu]

@rashmichoudhary07

Is that so. I might have misunderstood then. I'm sorry.
I guess the settings in config.yaml are stronger.
By the way, I have heard that run_every is an interval (monitoring interval) at which queries are sent to Elasticsearch, so if you set it too short, it will consume CPU resources. It is better to monitor the CPU resource and adjust the setting value.

[MakoWish]

I would also like the ability to specify a different run_any setting for each rule. I have several different rules that should be run at different intervals, but it does not seem that is currently possible. Some rules are okay with only running like every 10 minutes, and I have a couple rules I would like to run every 30 seconds.

@Qmando Is this something that could be done?