Elastalert: Deleting a rule does not stop querying or alerts when running elastalert as kubernetes deployment

[Shashankft9]

Hi,
I am running elastalert as a kubernetes deployment in my system. It is working well with CREATE and UPDATE operations, but seems to be failing on DELETE OPERATION, the problem being, once I delete the rule, the logs show "Rule file test.yaml not found, stopping rule execution" but the elasticsearch logstash index still keeps on being queried as per the rule and that too with the alerts being posted even after the deletion of the rule.
I saw that you have added a provision of is_enabled: False as a way to disable rules but that too apparently doesn't work here, saying: "Rule file check.yaml is now disabled" but the querying still goes on and the alerts are posted.

Any suggestions or workarounds?

this is my configuration file:

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /opt/rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 10

max_query_size: 10000

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  days: 3

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: elasticsearch-logging.xyz.svc.cluster.local

# The Elasticsearch port
es_port: 9200

# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
#  version: 1
#  incremental: false
#  disable_existing_loggers: false
#  formatters:
#    logline:
#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
#    handlers:
#      console:
#        class: logging.StreamHandler
#        formatter: logline
#        level: DEBUG
#        stream: ext://sys.stderr
#
#      file:
#        class : logging.FileHandler
#        formatter: logline
#        level: DEBUG
#        filename: elastalert.log
#
#    loggers:
#      elastalert:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch.trace:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      '':  # root logger
#        level: WARN
#          handlers:
#            - console
#            - file
#        propagate: false

[RandolphCYG]

Hi，my team has confronted with this question half mouth ago. According to my testing, I found that this could be easy to solve if you delve into file——elastalert/elastalert/elastalert.py.
func load_rule_changes() aims to syncs the running rules to match the files in rules_folder. However, I found it's deleting and modify part may have small defects——when the program reday to syncs the running rules, or rather when it step into func load_rule_changes, the rule files has been operated already. But it not use func scheduler to remove_job/modify_job. So the already deleted/modified rules still running.
I am ready to solve it, and my strategy is based on my study ahead——sacrifice few coupling to delay file operations，move file operations into func load_rule_changes's delete/modify part.
For now, I am trying to dig up the program's file operations with yaml configuration files.

[Shashankft9]

Hi Randolph,
you are absolutely correct with the problem in the func load_rule_changes()
I just found out that the issue has been acknowledged and fixed in September in the master branch, you can have a look here: https://github.com/Yelp/elastalert/pull/2484
This change hasn't been released yet, but you can test it by cloning the branch and installing elastalert again.
My initial testing shows that its working, would be nice if you can test and get back?

[RandolphCYG]

Hi，dear friend. Glad to receive your reply. And I just completed the rule deleting test. It works well. Thank you very much[cid:_Foxmail.1@a8d2dd94-7a5a-0519-ef3a-15198ffc4282]
Cause my careless not to fetch the value of the deleted rule's name. Actually it's in job_id=rule['name'].
Wish you success in your work!

---

[email protected]

From: [e-mail]
Date: 2019-12-06 18:12
To: Yelp/[e-mail]
CC: Randolph蔡迎港[e-mail]; [e-mail]
Subject: Re: [Yelp/elastalert] Deleting a rule does not stop querying or alerts when running elastalert as kubernetes deployment (#2589)

Hi Randolph,
you are absolutely correct with the problem in the func load_rule_changes()
I just found out that the issue has been acknowledged and fixed in September in the master branch, you can have a look here: #2484https://github.com/Yelp/elastalert/pull/2484
This change hasn't been released yet, but you can test it by cloning the branch and installing elastalert again.
My initial testing shows that its working, would be nice if you can test and get back?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/Yelp/elastalert/issues/2589?email_source=notifications&email_token=AHQ6HPEYLRIFCHV5EDWUCWLQXIQQRA5CNFSM4JUXVRPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGDUR6I#issuecomment-562514169, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHQ6HPD4VUVSG342PMKP42LQXIQQRANCNFSM4JUXVRPA.