Elastalert: How to specify different run_every: values on rules

[MatthewVines]

I have some rules I want to run every 10 seconds, and some that can run only once a day. But I can't seem to find a way to override the run_every: value in the base config file. Is this currently supported?

[Qmando]

Not currently supported. You have to run multiple instances with two separate rules folders and two separate config files. They could both use the same elastalert_status index without problems.

[MatthewVines]

Thanks, is this a core philosophy issue? or something that I should look into adding to the project?

[Qmando]

If you want to add that, that would be awesome. The main looping code is pretty dumb now, it just runs everything and then sleeps. You could basically turn it into an async loop with each rule having it's own timing. I would definitely accept a pull request for that, it's just something I am unlikely to put the effort into at this point.

[MatthewVines]

Cool. It may not come soon, but I think I will be working with this tool a lot over the next few months. As I get more comfortable with it, I hope to contribute a bit.

[Qmando]

Excellent. I'm mostly in maintenance mode for this project at the moment but always happy to help and get pull requests. Ask in https://gitter.im/Yelp/elastalert if you have any questions.

[djtecha]

+1 This would be very helpful vs having a flood of checks then nothing.

[amilsyb]

Would like that feature!

[bharatm89]

Do we have this feature now ?

[Qmando]

It's in the beta releases, which I'm very very close to merging with master.

[essodjolo]

Hello @Qmando
Thanks for merging the beta into the master branch.
Please, can you share with us how we can actually configure the feature mentionned ealry in this thread per rule?
Thanks.

[Qmando]

Add a different run_every to each rule's yaml. That's it.

[essodjolo]

Thanks.

Essodjolo KAHANAM 

---- On jeu., 08 août 2019 02:33:16 +0200 [email protected] wrote ----

Add a different run_every to each rule's yaml. That's it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[essodjolo]

Hello @Qmando

I've made a fresh installation of elastalert (had toubles with python 2 so did it with python3).
I'm able to start the program and my rule is loaded but I'm getting an exception after a while, and the program exits with: AttributeError: 'ElastAlerter' object has no attribute 'current_es'
Full trace

root@myserver:/opt/elastalert# python3 -m elastalert.elastalert --verbose
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999926 seconds
INFO:elastalert:Queried rule My rule name from 2019-08-08 13:42 CEST to 2019-08-08 13:57 CEST: 9 / 9 hits
ERROR:root:Traceback (most recent call last):
  File "/opt/elastalert/elastalert/elastalert.py", line 1259, in handle_rule_execution
    num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
  File "/opt/elastalert/elastalert/elastalert.py", line 900, in run_rule
    if not self.run_query(rule, rule['starttime'], endtime):
  File "/opt/elastalert/elastalert/elastalert.py", line 660, in run_query
    self.current_es.clear_scroll(scroll_id=scroll_id)
AttributeError: 'ElastAlerter' object has no attribute 'current_es'

ERROR:root:Uncaught exception running rule My rule name: 'ElastAlerter' object has no attribute 'current_es'

Could you please provide any help?
Thanks.

[essodjolo]

@Qmando
Found a solution.
Changed line 660 of elastalert/elastalert/elastalert.py to

self.thread_data.current_es.clear_scroll(scroll_id=scroll_id)

I was able to get my rule triggered and received a Slack alert.

[essodjolo]

But I would have tought that run_every directive per rule will override the one in config.yml but this is not the case for me.
Is there anything I'm missing?

run_every:
  seconds: 5

[yuriipolishchuk]

@Qmando I've faced the same issue. run_every in a rule doesn't override default one in config.
I'm using the latest version of elastalert available: 0.2.1.

Here're my config

    rules_folder: /opt/rules
    scan_subdirectories: false
    run_every:
      minutes: 3
    buffer_time:
      minutes: 10
    es_host: eshost
    es_port: 443
    writeback_index: elastalert_status
    use_ssl: True
    verify_certs: True
    alert_time_limit:
      minutes: 2880

and rules:

    name: TEST Error
    type: frequency
    index: "logs-*"
    num_events: 1
    attach_related: true
    timeframe:
      minutes: 10
    run_every:
      minutes: 10
    alert_subject: "TEST Error Occured at {0}"
    alert_subject_args:
    - "@timestamp"
    alert_text: "*Chart:* {0} *Component:* {1} *Pod:* {2}\n\n{3}"
    alert_text_args:
    - "kubernetes.labels.chart"
    - "kubernetes.labels.component"
    - "kubernetes.pod_name"
    - "log"
    filter:
      - query:
          query_string:
            query: "query_goes_here"
    alert:
      - "slack"
    slack:
    slack_webhook_url: slack_webhook_url
    slack_channel_override: "#slack-channel"

I have a list of similar rules with different queries and run_every parameter.

What's wrong here?

@essodjolo Did you manage to find the workaround except running another instance of elastalert?

[akshaygupta2208]

Is this issue resolved ?

[Qmando]

Sorry this was open for so long... I've just opened a PR to fix the issue.