elastalert.util.EAException: Error loading file /opt/elastalert/rules/.snapshot/daily.2019-04-03_0010/rule.yaml: Duplicate rule named first-rule

[RamjiVE]

I have swarm enviornment and there is no symlinks defined.
But still I could see this error . Could you pls tell me how to solve?

Starting Alerting

Container timezone not modified

Elastalert index already exists in ES.

Traceback (most recent call last):

File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main

"__main__", fname, loader, pkg_name)
File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code

exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in

sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1923, in main

client = ElastAlerter(args)
File "/opt/elastalert/elastalert/elastalert.py", line 130, in init

self.conf = load_rules(self.args)
File "elastalert/config.py", line 488, in load_rules

raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file /opt/elastalert/rules/.snapshot/daily.2019-04-03_0010/rule.yaml: Duplicate rule named first-rule

My rules file

---

alert:

• email
  alert_subject: "ElastAlert Failed in {0}"
  alert_subject_args:
• deployment_environment_id
  alert_text_type: alert_text_only
  alert_text: " import attempted at {0} failed. I"
  alert_text_args:
• "@timestamp"
  email:
• [email protected]
  email_reply_to: [email protected]
  filter:
  -
  query_string:
  query: "message: import_failed"
  from_addr: [email protected]
  index: logstash-%Y.%m.%d*
  name: import-failed
  smtp_host: s97mhsn.com
  type: any
  use_strftime_index: true

[RamjiVE]

Any updates on this issue???

[Qmando]

Duplicate rule named first-rule
You can't have two rules with the same name.

[RamjiVE]

I have only one rule... I don't have any duplicates

[RamjiVE]

I'm using docker swarm environment, i even don't have symlinks defined.. So how it is showing as a duplicate?

[Qmando]

All subdirectories are included too.
You could use this diff to debug

--- a/elastalert/config.py
+++ b/elastalert/config.py
@@ -402,6 +402,7 @@ def get_file_paths(conf, use_rule=None):
                 if use_rule and use_rule != filename:
                     continue
                 if isyaml(filename):
+                    print filename
                     rule_files.append(os.path.join(root, filename))
     else:

[diversario]

Elastalert shouldn't be picking up directories starting with ., IMO. This is what a mounted ConfigMap looks like in Kubernetes:

/opt/elastalert $ cd rule_templates/
/opt/elastalert/rule_templates $ ls -la
total 4
drwxrwsrwx    3 root     node          4096 Aug  6 22:16 .
drwxr-xr-x    1 node     node            24 Jul 21 17:29 ..
drwxr-sr-x    2 root     node           309 Aug  6 22:16 ..2019_08_06_22_16_48.074554588
lrwxrwxrwx    1 root     root            31 Aug  6 22:16 ..data -> ..2019_08_06_22_16_48.074554588
lrwxrwxrwx    1 root     root            30 Aug  6 22:16 detection_template.yaml -> ..data/detection_template.yaml
lrwxrwxrwx    1 root     root            31 Aug  6 22:16 error_jira_template.yaml -> ..data/error_jira_template.yaml
lrwxrwxrwx    1 root     root            40 Aug  6 22:16 integration_started_template.yaml -> ..data/integration_started_template.yaml
lrwxrwxrwx    1 root     root            28 Aug  6 22:16 no_data_template.yaml -> ..data/no_data_template.yaml
lrwxrwxrwx    1 root     root            36 Aug  6 22:16 relevant_attack_template.yaml -> ..data/relevant_attack_template.yaml
lrwxrwxrwx    1 root     root            25 Aug  6 22:16 spike_template.yml -> ..data/spike_template.yml
lrwxrwxrwx    1 root     root            38 Aug  6 22:16 successful_attack_template.yaml -> ..data/successful_attack_template.yaml
lrwxrwxrwx    1 root     root            29 Aug  6 22:16 threshold_template.yml -> ..data/threshold_template.yml
lrwxrwxrwx    1 root     root            37 Aug  6 22:16 volumetric_alert_template.yaml -> ..data/volumetric_alert_template.yaml
/opt/elastalert/rule_templates $ ls -la ..data/
total 40
drwxr-sr-x    2 root     node           309 Aug  6 22:16 .
drwxrwsrwx    3 root     node          4096 Aug  6 22:16 ..
-rwxrwxrwx    1 root     node          1458 Aug  6 22:16 detection_template.yaml
-rwxrwxrwx    1 root     node          1546 Aug  6 22:16 error_jira_template.yaml
-rwxrwxrwx    1 root     node          1158 Aug  6 22:16 integration_started_template.yaml
-rwxrwxrwx    1 root     node           802 Aug  6 22:16 no_data_template.yaml
-rwxrwxrwx    1 root     node          2083 Aug  6 22:16 relevant_attack_template.yaml
-rwxrwxrwx    1 root     node           679 Aug  6 22:16 spike_template.yml
-rwxrwxrwx    1 root     node          1152 Aug  6 22:16 successful_attack_template.yaml
-rwxrwxrwx    1 root     node          1465 Aug  6 22:16 threshold_template.yml
-rwxrwxrwx    1 root     node          1625 Aug  6 22:16 volumetric_alert_template.yaml
/opt/elastalert/rule_templates $ ls -la ..2019_08_06_22_16_48.074554588/
total 40
drwxr-sr-x    2 root     node           309 Aug  6 22:16 .
drwxrwsrwx    3 root     node          4096 Aug  6 22:16 ..
-rwxrwxrwx    1 root     node          1458 Aug  6 22:16 detection_template.yaml
-rwxrwxrwx    1 root     node          1546 Aug  6 22:16 error_jira_template.yaml
-rwxrwxrwx    1 root     node          1158 Aug  6 22:16 integration_started_template.yaml
-rwxrwxrwx    1 root     node           802 Aug  6 22:16 no_data_template.yaml
-rwxrwxrwx    1 root     node          2083 Aug  6 22:16 relevant_attack_template.yaml
-rwxrwxrwx    1 root     node           679 Aug  6 22:16 spike_template.yml
-rwxrwxrwx    1 root     node          1152 Aug  6 22:16 successful_attack_template.yaml
-rwxrwxrwx    1 root     node          1465 Aug  6 22:16 threshold_template.yml
-rwxrwxrwx    1 root     node          1625 Aug  6 22:16 volumetric_alert_template.yaml

ConfigMaps cannot be used because Elastalert goes through those .. directories and thinks it found duplicate rules.

[kylezs]

For anyone else getting this issue using kubernetes and ConfigMaps, set scan_subdirectories: false in your config.yaml. https://github.com/Yelp/elastalert/issues/1012