Elastalert: Don't look for rules in subdirectories that start with a period

[micahhausler]

I'm using Kubernetes to mount Elastalert rules into a directory, and the way that Kuberentes mounts files is with links, like so:

/ # ls -alR /rules/
/rules/:
total 12
drwxrwxrwx    3 root     root          4096 Apr 12 13:33 .
drwxr-xr-x    1 root     root          4096 Apr 12 13:33 ..
drwxr-xr-x    2 root     root          4096 Apr 12 13:33 ..4984_12_04_13_33_37.139875337
lrwxrwxrwx    1 root     root            31 Apr 12 13:33 ..data -> ..4984_12_04_13_33_37.139875337
lrwxrwxrwx    1 root     root            17 Apr 12 13:33 rule1.yaml -> ..data/rule1.yaml
lrwxrwxrwx    1 root     root            17 Apr 12 13:33 rule2.yaml -> ..data/rule2.yaml

/rules/..4984_12_04_13_33_37.139875337:
total 16
drwxr-xr-x    2 root     root          4096 Apr 12 13:33 .
drwxrwxrwx    3 root     root          4096 Apr 12 13:33 ..
-rw-r--r--    1 root     root           598 Apr 12 13:33 rule1.yaml
-rw-r--r--    1 root     root           555 Apr 12 13:33 rule2.yaml

When I try to point my rule directory at /rules/, I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert", line 11, in <module>
    load_entry_point('elastalert==0.1.8', 'console_scripts', 'elastalert')()
  File "/usr/local/lib/python2.7/site-packages/elastalert/elastalert.py", line 1578, in main
    client = ElastAlerter(args)
  File "/usr/local/lib/python2.7/site-packages/elastalert/elastalert.py", line 95, in __init__
    self.conf = load_rules(self.args)
  File "/usr/local/lib/python2.7/site-packages/elastalert/config.py", line 425, in load_rules
    raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file /rules/..4984_12_04_13_30_12.887977537/rule2.yaml: Duplicate rule named Admin Privilege Gain

Can Elastalert ignore rules in subdirectores that begin with a period .?

[micahhausler]

I just found the scan_subdirectories flag is available and that solves my use case, but I'd still like to know if this is a feature worth providing. I'd be willing to implement it if so.

[FestivalBobcats]

Just ran into the same issue myself. Thanks for pointing out that flag @micahhausler

[Qmando]

Yeah this sounds reasonable.

[rdtr]

@micahhausler You saved my day... very good catch. I've been struggling why ElastAlert fails due to "Duplicated Name" while I mounting only one rule.

[greg-cook]

I'm having this exact issue right now. But unfortunately when I set scan_subdirectories: false elastalert just doesn't start (no output to stdout).
Have there been any recent changes to this?

[Qmando]

I don't believe so. You definitely have .yaml or .yml files in the rules_folder?

[greg-cook]

Sorry, should have run it in --verbose. It is in fact working, it's just very silent 😄
Thanks @micahhausler for the tip and @Qmando for responding!