Elastalert: Include blacklist and whitelist using files

[ghost]

Include blacklist and whitelist using files
rule
blacklist:

• include : /path/file.yaml

Will it be realized?

[Qmando]

No, this is not a feature right now.

[bean5]

An interesting idea. That would allow rules to share portions of what they match. I can see value in this. But formatting of the file would have be agreed upon.

[SOCSIEM]

Need this feature too. ASAP. :)
Will be really useful to handle long list of malicious domains, URLs, IPs

[SherifEldeeb]

I did a small modification to allow for this (still testing before submitting a pull request) ... it works for both blacklist and whitelist.
This will do two things:

• Add a new option to include entries from files using a special directive - "!file /path/to/list"
• use a set instead of a list in the blacklist/whitelist entries, which is much faster with larger sets and deduplicates automatically.
  For example:

blacklist:
- "evil.com"
- "hacker.org"
- "!file /tmp/black.txt"
- "!file /tmp/black2.txt"

You can mix between the list entry definition formats, or use either one.
Feedback is welcome :)

please find it here:
https://github.com/SherifEldeeb/elastalert/commit/f3df37a052e55f246e9f402bd5a64ecd866969a0

[ghost]

Thank you very much!

[sathishdsgithub]

@pushkarevds @SherifEldeeb

I would like to use the above method to whitelist IP address and specific string. Currently, I have a cardinality rule and I would like to do whitelist on few strings and IP address by combining the rule as shown below. The rule runs without any error but it's not working as expected. Can someone assist me is this a right method to combine the rule for whitelist?

Main Rule

type: cardinality
filter:
 - terms:
     rule_name:
         -  whitelist-zone  --> Rule name for whitelisting zone
         -  whitelisti-ip  --> Rule name for whitelisting IP address

whitelist-zone

root@ubuntu:/tmp/elastalert# more whitelist-zone.yaml
# ElastAlert Rule Name

name: whitelist-zone

# ElasticSearch host and port details

es_host: 192.168.96.141

es_port: 9200

# ElasticSearch Index Name

index: graylog_0

# Timestamp added to overcome ES graylog parsing issue

timestamp_field: timestamp
timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%f'
timestamp_format_expr: 'ts[:23] + ts[26:]'


doc_type: message

#writeback_index: elastalert_status

# Network Port scan aggregation Rule 50 counts in 60 Seconds

type: whitelist

compare_key: srx-source-zone-name

whitelist:
 - "!file /opt/whitelist-zone.txt"

whitelisti-ip

root@ubuntu:/tmp/elastalert# more whitelist-ip.yaml
# ElastAlert Rule Name

name: whitelisti-ip

# ElasticSearch host and port details

es_host: 192.168.96.141

es_port: 9200

# ElasticSearch Index Name

index: graylog_0

# Timestamp added to overcome ES graylog parsing issue

timestamp_field: timestamp
timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%f'
timestamp_format_expr: 'ts[:23] + ts[26:]'


doc_type: message

#writeback_index: elastalert_status

# Network Port scan aggregation Rule 50 counts in 60 Seconds

type: whitelist

compare_key: srx-source-address

whitelist:
  - "!file /opt/whitelist-IP.txt"

Files

root@ubuntu:/tmp/elastalert# cat /opt/whitelist-IP.txt
10.18.109.59
10.10.10.10
20.20.20.20
30.30.30.30
root@ubuntu:/tmp/elastalert# cat /opt/whitelist-zone.txt
EXTERNAL
Internet
root@ubuntu:/tmp/elastalert#

[bean5]

Maybe I don't know if this feature, but you have "-->" in rule names. Do
you think that is causing problems. In yaml you use # to start a comment.
Correct me if I'm wrong.

On Fri, Jul 14, 2017, 5:50 AM sathishdsgithub [e-mail]
wrote:

@pushkarevds https://github.com/pushkarevds @SherifEldeeb
https://github.com/sherifeldeeb

I would like to use the above method to whitelist IP address and specific
string. Currently, I have a cardinality rule and I would like to do
whitelist on few strings and IP address by combining the rule as shown
below. The rule runs without any error but it's not working as expected.
Can someone assist me is this a right method to combine the rule for
whitelist?

Main Rule

type: cardinality
filter:

• terms:
  rule_name:
  - whitelist-zone --> Rule name for whitelisting zone
  - whitelisti-ip --> Rule name for whitelisting IP address

whitelist-zone

root@ubuntu:/tmp/elastalert# more whitelist-zone.yaml

ElastAlert Rule Name

name: whitelist-zone

ElasticSearch host and port details

es_host: 192.168.96.141

es_port: 9200

ElasticSearch Index Name

index: graylog_0

Timestamp added to overcome ES graylog parsing issue

timestamp_field: timestamp
timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%f'
timestamp_format_expr: 'ts[:23] + ts[26:]'

doc_type: message

writeback_index: elastalert_status

Network Port scan aggregation Rule 50 counts in 60 Seconds

type: whitelist

compare_key: srx-source-zone-name

whitelist:

• "!file /opt/whitelist-zone.txt"

whitelisti-ip

root@ubuntu:/tmp/elastalert# more whitelist-ip.yaml

ElastAlert Rule Name

name: whitelisti-ip

ElasticSearch host and port details

es_host: 192.168.96.141

es_port: 9200

ElasticSearch Index Name

index: graylog_0

Timestamp added to overcome ES graylog parsing issue

timestamp_field: timestamp
timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%f'
timestamp_format_expr: 'ts[:23] + ts[26:]'

doc_type: message

writeback_index: elastalert_status

Network Port scan aggregation Rule 50 counts in 60 Seconds

type: whitelist

compare_key: srx-source-address

whitelist:

• "!file /opt/whitelist-IP.txt"

Files

root@ubuntu:/tmp/elastalert# cat /opt/whitelist-IP.txt
10.18.109.59
10.10.10.10
20.20.20.20
30.30.30.30
root@ubuntu:/tmp/elastalert# cat /opt/whitelist-zone.txt
EXTERNAL
Internet
root@ubuntu:/tmp/elastalert#

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Yelp/elastalert/issues/711#issuecomment-315351632,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB9SJstl1tw8nwGk76J9LFTDHDrFZjWyks5sN2QKgaJpZM4J_FP8
.

[sathishdsgithub]

@bean5 I have just mentioned that for understanding ,My actual rule doesn't have that

[bean5]

Okay. Hopefully someone else more experienced can locate your real issue.

On Fri, Jul 14, 2017, 8:10 AM sathishdsgithub [e-mail]
wrote:

@bean5 https://github.com/bean5 I have just mentioned that for
understanding ,My actual rule doesn't have that

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/Yelp/elastalert/issues/711#issuecomment-315384693,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB9SJt9PmjsYLFVYcqWtj-Qq93Rkmb3Uks5sN4T8gaJpZM4J_FP8
.