Elastalert: Basic New_Term Rule for newb

[senatorhotchkiss]

Trying to accomplish a very simple task. Show SHA256 checksum value of executed files, which has not been seen before.

But I am flooded with alerts by already existing SHA256 values. It's not working as expected.
What am I doing wrong?

name: New Process Detected
type: new_term
index: "*:logstash-*"
fields:
 - "sha256"
terms_window_size:
  days: 90
filter:
- term:
    event_type: "sysmon"
alert:
#- "debug"
- "email"
# Email options
alert_subject: "New Process Found!"
email:
- "[email protected]"
smtp_host: "myserver"
smtp_port: 25
from_addr: "[email protected]"

[senatorhotchkiss]

I am having better luck when fields is set to another field (image_name) then it will trigger as expected. Is it something to do with the field type?

Json example code looks like this in my Kibana. But it is not working for sha256. Any guidance would be appreciated.

"image_path": "C:\Program Files (x86)\Microsoft Office\Word.exe",
"sha256": "A46D7D24C7E31A332CE29FDD6A639163BA3D4BC31840176CEC01510183D2C10",

[Qmando]

Hmmm.. They should act roughly the same. Did you get a similar message when you start it up saying "Found ____ existing values for sha256/image_path"? You should see that when you first start elastalert if you are using --verbose or --debug mode. Maybe it's possible that one query timed out or returned bad results for some reason.

I believe it will also tried to add ".keyword" to the field so that it gets the full value instead of analyzed tokens. If you've got a mapping where sha256.keyword doesn't exist, maybe it would have weird behavior.

[senatorhotchkiss]

I think I resolved some problems by looking at "md5" instead. But I am still getting alerts on already existing values. It's like ElastAlert needs to see everything once before being able to filter it.

INFO:elastalert:Ignoring match for silenced rule New Process Detected
INFO:elastalert:Ran New Process Detected from 2018-06-29 08:56 UTC to 2018-06-29 08:57 UTC: 75 query hits (0 already seen), 24 matches, 1 alerts sent
INFO:elastalert:Sleeping for 59.05537 seconds
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 08:57 UTC to 2018-06-29 08:58 UTC: 25 / 25 hits
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 08:58 UTC to 2018-06-29 08:58 UTC: 25 / 0 hits
INFO:elastalert:Ignoring match for silenced rule New Process Detected
INFO:elastalert:Ignoring match for silenced rule New Process Detected
INFO:elastalert:Ignoring match for silenced rule New Process Detected
INFO:elastalert:Ran New Process Detected from 2018-06-29 08:57 UTC to 2018-06-29 08:58 UTC: 25 query hits (0 already seen), 3 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.678049 seconds
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 08:58 UTC to 2018-06-29 08:59 UTC: 24 / 24 hits
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 08:59 UTC to 2018-06-29 08:59 UTC: 24 / 0 hits
INFO:elastalert:Sent email to ['[email protected]']
INFO:elastalert:Ran New Process Detected from 2018-06-29 08:58 UTC to 2018-06-29 08:59 UTC: 24 query hits (0 already seen), 1 matches, 1 alerts sent
INFO:elastalert:Sleeping for 59.716847 seconds
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 08:59 UTC to 2018-06-29 09:00 UTC: 30 / 30 hits
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 09:00 UTC to 2018-06-29 09:00 UTC: 30 / 0 hits
INFO:elastalert:Sent email to ['[email protected]']
INFO:elastalert:Ran New Process Detected from 2018-06-29 08:59 UTC to 2018-06-29 09:00 UTC: 30 query hits (0 already seen), 1 matches, 1 alerts sent
INFO:elastalert:Sleeping for 59.731614 seconds
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 09:00 UTC to 2018-06-29 09:01 UTC: 56 / 56 hits
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 09:01 UTC to 2018-06-29 09:01 UTC: 56 / 0 hits
INFO:elastalert:Sent email to ['[email protected]']
INFO:elastalert:Ignoring match for silenced rule New Process Detected
INFO:elastalert:Ignoring match for silenced rule New Process Detected
INFO:elastalert:Ran New Process Detected from 2018-06-29 09:00 UTC to 2018-06-29 09:01 UTC: 56 query hits (0 already seen), 3 matches, 1 alerts sent
INFO:elastalert:Sleeping for 59.4066 seconds
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 09:01 UTC to 2018-06-29 09:02 UTC: 32 / 32 hits
INFO:elastalert:Queried rule New Process Detected from 2018-06-29 09:02 UTC to 2018-06-29 09:02 UTC: 32 / 0 hits
INFO:elastalert:Ran New Process Detected from 2018-06-29 09:01 UTC to 2018-06-29 09:02 UTC: 32 query hits (0 already seen), 0 matches, 0 alerts sent

At the moment I only get one mail, but if I enable realert: 0 I will be overwhelmed.

Is there any way to see the index of existing values, so I can see values already known?

[senatorhotchkiss]

Could this have any relation to the raw vs. keyword aggr issue?

https://github.com/Yelp/elastalert/issues/1662

[Qmando]

The linked issue was fixed several months ago, so if you are using the latest version it's fine. Do you get an expected number in the log "Found ____ unique values for sha256/image_path"?

you can print them all out using

--- a/elastalert/ruletypes.py
+++ b/elastalert/ruletypes.py
@@ -732,6 +732,7 @@ class NewTermsRule(RuleType):
                         elastalert_logger.info('Found no values for %s' % (field))
                     continue
                 self.seen_values[key] = list(set(values))
+                print set(values)
                 elastalert_logger.info('Found %s unique values for %s' % (len(set(values)), key))

[senatorhotchkiss]

I just checked a run from 16:09 - 16:10 - it triggered an e-mail.
But that trigger is not a true "new_term", I can find six occurences within the last 24 hours in Kibana, but it is the first occurence since the rule was created.

Actually the numbers does not add up - the ElastAlert says 14 query hits, where a Kibana query with same filters applied gives me 80 hits.

INFO:elastalert:Ran New Process Detected from 2018-06-29 16:09 UTC to 2018-06-29 16:10 UTC: 14 query hits (0 already seen), 1 matches, 1 alerts sent

I am getting something, but apparently not everything.

The ElastAlert I am using is part of the SO distribution.

[Qmando]

What does part of the SO distribution mean? What version are you using?

Again, does the number reported in the log "Found ____ unique values for sha256/image_path" match your expectations?

[senatorhotchkiss]

My BAD - the numbers in the query hits are OK. Kibana showed me the non-unique hits, where Elastalert will show me the unique hits.

But I still have the problem about alerting on false positives.

[senatorhotchkiss]

Qmando - sorry I meant the Security Onion distribution. I think they made a docker for ElastAlert.

[senatorhotchkiss]

I am not quite sure where to find the logs, that you are talking about, since it is running as a docker image.

The only file I can find is this one (probably customized by team SO) : /var/log/elastalert/elastalert_stderr.log

[Qmando]

The same log you posted from several comments ago, the stderr one. When you start elastalert, new_term rules will report this line with the unique count. The print line I added would like show up in /var/log/elastalert/elastalert_stdout.log

[senatorhotchkiss]

Oh, I am rebooting the system now and will state the line. Sorry for the confusion. :-)
Appreciate your assistance here.

I find it a bit hard grasp the whole docker idea, at the moment, but I will probably have to get up to speed on that. Then I can probably access the main configuration for the Elastalert.

[senatorhotchkiss]

I am not seeing that line anywhere. I included the logfile. It shows the entire boot cycle time periodlog_after_reboot.txt
.

[senatorhotchkiss]

INFO:elastalert:Found no values for md5

This line is seen after reboot. Is this the evidence, as to that no md5 baseline is returned?
Btw. not sure, you noticed that I changed my new_term field from "sha256" to "md5", during my efforts to fix the problem.

[senatorhotchkiss]

It doesn't matter what field I try to set as new_term field. I will see the error below on all cases. So that is why my initial index is never built. Hmmm...

INFO:elastalert:Found no values for hostname
INFO:elastalert:Found no values for md5
INFO:elastalert:Found no values for image_path

[Qmando]

This definitely looks like the issue I mentioned about no .keyword field existing. Hmm, this is currently hard coded. I don't want you to have to re-index everything...

You can verify this by looking at the mapping of your fields to see if md5.keywor exists. To bypass this, you'd have to make more code changes, which I feel really bad about using as a bug fix for people 😢 But, if you feel like trying to get past this:

--- a/elastalert/ruletypes.py
+++ b/elastalert/ruletypes.py
@@ -678,7 +678,7 @@ class NewTermsRule(RuleType):
                 level = query_template['aggs']
                 # Iterate on each part of the composite key and add a sub aggs clause to the elastic search query
                 for i, sub_field in enumerate(field):
-                    level['values']['terms']['field'] = add_raw_postfix(sub_field, self.is_five_or_above())
+                    level['values']['terms']['field'] = sub_field #add_raw_postfix(sub_field, self.is_five_or_above())
                     if i < len(field) - 1:
                         # If we have more fields after the current one, then set up the next nested structure
                         level['values']['aggs'] = {'values': {'terms': copy.deepcopy(field_name)}}
@@ -686,7 +686,7 @@ class NewTermsRule(RuleType):
             else:
                 self.seen_values.setdefault(field, [])
                 # For non-composite keys, only a single agg is needed
-                field_name['field'] = add_raw_postfix(field, self.is_five_or_above())
+                field_name['field'] = field #add_raw_postfix(field, self.is_five_or_above())

[senatorhotchkiss]

Funny thing is that the term md5.keyword is visible in Kibana. I can filter on it just fine from in there.

The code you shared. I would need to apply that 'inside' the docker instance, right? At the current time my skill level, is rookie. So I need to find my way to edit those files.

[senatorhotchkiss]

I gained access to the docker container and found the attached in ruletypes.py.
Class_NewTerm.txt

[senatorhotchkiss]

This is quite a journey for a man who has been living in powershell for the last 5 years :) I am really positive over the things, that I have encountered. The Docker concept is so powerful and elegant.

My docker image was somehow reset (file reverted to original) on every boot, by an external script. That took me some hours to figure out.

Now I am making some progress. After changing the code in ruletypes.py, I get this, when I power on EA.

I am a little sceptic about the number of unique values found. That number sounds more like the total hits (non-unique)?

INFO:elastalert:Found 2739626 unique values for md5.keyword

Any feedback would be highly appreciated.

[senatorhotchkiss]

Oh - I appear to have a new issue now. The alerts for new "new_term" occurence are not firing.

As you can see in the attached file, after restart the unique values went from 88 to 89, yet no alert was triggered.
no_alert.txt

[senatorhotchkiss]

Qmando, I really hope, that you can help me. I feel I am so close to the solution.

It's like changing those two lines of code, made the alert not trigger. If I revert back, I get the alerts again, but then it's back to having no hits for the initial baseline. :(

This thread really is my last hope.

[senatorhotchkiss]

So to make it absolutely clear:

1) Will trigger match/alerts, but not build the initial index:

fields:

• [image_path]

Result:
INFO:elastalert:Found no values for image_path
UTC: 1 query hits (0 already seen), 1 matches, 1 alerts sent

2) Will build the index (only if I modify the two lines in ruletypes.py), but not trigger any match or alerts:

fields:

• [image_path.keyword]

Result:
INFO:elastalert:Found 112 unique values for image_path.keyword
UTC: 7 query hits (0 already seen), 0 matches, 0 alerts sent

What am I doing wrong here??

[Qmando]

So, the 88 to 89 thing might be a Elasticsearch being imprecise in it's unqiue value aggregation results.

Counts are approximated
Computing exact counts requires loading values into a hash set and returning its size. This doesn’t scale when working on high-cardinality sets and/or large values as the required memory usage and the need to communicate those per-shard sets between nodes would utilize too many resources of the cluster.

This cardinality aggregation is based on the HyperLogLog++ algorithm, which counts based on the hashes of the values with some interesting properties:

Though normally, I only see this number vary if the cardinality is in the 1000s, not <100. If you stop and start elastalert multiple times very quickly, you might see the number jump back and force.

As for why you get no alerts for new values when using the .keyword, it's possible there was a bug in the hotfix I gave you. Let me attempt to reproduce this and verify myself, and I'll report back asap.

[senatorhotchkiss]

Thanks Qmando :)

But I need to say that appending .keyword to any search query using fields: will stop matching/alerting. It's not only when using your "hotfix" - it exhibits the same behavior using the original code.

[senatorhotchkiss]

I tried to determine the version in use and found this in the head of changelog.md inside the docker.
Not sure if that is an outdated version?

I think the Docker container is part of the Security Onion distribution. I am not sure, that it can be updated? Maybe I can just replace the files updated?

Change Log

v0.1.29

#

• Added a feature forget_keys to prevent realerting when using flatline with query_key
• Added a new alert_text_type, aggregation_summary_only

[ccao93]

I have this same issue, got lots of hits but no matches or alerts. The following is the detail I got:
$ python elastalert/elastalert.py --rule new_devices.yaml --debug INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent. To send them but remain verbose, use --verbose instead. INFO:elastalert:Found 42 unique values for mac.keyword INFO:elastalert:Starting up INFO:elastalert:Queried rule New Devices Detected in the Network from 2018-07-02 13:34 CDT to 2018-07-02 13:42 CDT: 46 / 46 hits INFO:elastalert:Skipping writing to ES: {'hits': 46, 'matches': 0, '@timestamp': '2018-07-02T18:42:26.757630Z', 'rule_name': 'New Devices Detected in the Network', 'starttime': '2018-07-02T18:34:18.705115Z', 'endtime': '2018-07-02T18:42:26.144182Z', 'time_taken': 0.613422155380249} INFO:elastalert:Ran New Devices Detected in the Network from 2018-07-02 13:34 CDT to 2018-07-02 13:42 CDT: 46 query hits (0 already seen), 0 matches, 0 alerts sent

[senatorhotchkiss]

Yeah that looks like the problem I am fighting. Are you on the latest release?

[ccao93]

@senatorhotchkiss I am not, the version I am using is 0.1.30. I used to have this issue and I fixed by changing elastalert/util.py. I am now able to get INFO:elastalert:Found 42 unique values for mac.keyword but not able to have any matches.

[senatorhotchkiss]

OK. I think we have to wait for Qmando. He is such a smart guy ;)

[ccao93]

I added use_terms_query, doc_type, and query_key to my rule, and I can get alerts properly now. However, when I tried to send alerts to slack, I got lots of MISSING VALUES in the alerts.
My setting is like the following:

use_terms_query: true
query_key: ["mac.keyword", "message.keyword", "assigned_addr", "client_addr", "hostname", "server_addr", "requested_addr", "msg_types"]
doc_type: "doc"
filter:
- query:
    query_string:
      query: "<some search query>"
alert_text_type: alert_text_only
alert_text: "Found a new device in the network:\n timestamp: {}\n mac_address: {}\n message: {}\n assigned_address: {}\n client_address: {}\n hostname: {}\n server_address: {}\n requested_address: {}\n message_types: {}\n"
alert_text_args:
  - "@timestamp"
  - mac.keyword
  - message.keyword
  - assigned_addr
  - client_addr
  - hostname
  - server_addr
  - requested_addr
  - msg_types

Apart from timestamp and mac.keyword, everything else in the alert text is a MISSING VALUE. I would really appreciate if anyone know what might be the issue.

[Qmando]

Ok, I did some research and I did find some bugs.

1. If using use_terms_query, the "search for existing values" query will add .keyword and get the full text, whereas each subsequent query will only return the keywords themselves, causing false positives. IE
   Query1: terms of username.keyword ["John Doe", etc]
   Query2: terms of username ["john", "doe"]
   Two false positives would be generated

2. If you applied my patch (or use the new option i'm adding), and aren't using use_terms_query, then the problem is roughly the opposite. The first query will get the keywords and then they won't match the full document's values.
   Query1: terms of username: ["john", "doe"]
   Query2: Full doc={"username": "John Doe"}
   And a single false positive is generated

3. You're not supposed to be able to use use_terms_query with multiple query_keys. It's supposed to stop you, but it doesn't. This results in lots of values seen in the first query, and nothing returned after that.

@ccao93 @senatorhotchkiss
It looks like you both DO have ".keyword" fields, because of the "Found ... for field.keyword" lines
I think the following should work, after removing the patch I suggested

fields: ["mac"]
use_terms_query: false

OR

fields: ["mac.keyword"]
use_terms_query: true

Also please note that if you use terms query, you cannot use other fields in the alert_text. In order to use the other fields, you have to use the normal, default query method.

I'm pushing a new feature to optionally disable the .keyword addition soon.

[ccao93]

@Qmando It helps a lot, and thank you for the quick response and contribution to this project!

[Qmando]

PR #1778 should address having this option, as well as fixing some warnings/error conditions.

[ccao93]

@senatorhotchkiss Actually, https://github.com/Yelp/elastalert/blob/master/elastalert/util.py#L261-L268 this function would add .keyword automatically for you if you are using new version of ES, and seems using you cannot just use mac directly in the field.

python elastalert/elastalert.py --rule new_devices.yaml --debug
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
WARNING:elasticsearch:GET http://localhost:9200/b-ids-*/_search?ignore_unavailable=true&timeout=50s [status:400 request:0.278s]
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 1867, in <module>
    sys.exit(main(sys.argv[1:]))
  File "elastalert/elastalert.py", line 1861, in main
    client = ElastAlerter(args)
  File "elastalert/elastalert.py", line 130, in __init__
    self.conf = load_rules(self.args)
  File "/etc/elastalert/elastalert/config.py", line 484, in load_rules
    raise EAException('Error loading file %s: %s' % (rule_file, e))
util.EAException: Error loading file /etc/elastalert/rules/new_devices.yaml: Error initializing rule New Devices Detected in the Network: Error searching for existing terms: RequestError(400, u'search_phase_execution_exception', {u'status': 400, u'error': {u'failed_shards': [{u'node': u'-eSLamAkQRy_t2DVpxgiYA', u'index': u'b-ids-2018.04.26', u'reason': {u'reason': u'Fielddata is disabled on text fields by default. Set fielddata=true on [mac] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.', u'type': u'illegal_argument_exception'}, u'shard': 0}], u'root_cause': [{u'reason': u'Fielddata is disabled on text fields by default. Set fielddata=true on [mac] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.', u'type': u'illegal_argument_exception'}], u'grouped': True, u'reason': u'all shards failed', u'phase': u'query', u'type': u'search_phase_execution_exception'}})

The only option here might be use mac.keyword and turn on use_terms_query.

[senatorhotchkiss]

I must admit, that I still have no luck getting both baseline results and new matches. I tried using the rule below.

index: "*:logstash-beats*"
fields:
 - ["computer_name"]
use_terms_query: false

Throws this and will not built the baseline.
WARNING:elastalert:No results were found from all sub-aggregations. This can either indicate that there is no baseline data OR that a non-primitive field was used in a composite key.

I am starting to think that bulding something in bash using the diff command, is a more simple option. But this is also because I am no expert on Python. Just a simple guy. :)

[Qmando]

@senatorhotchkiss
Can you try switching

fields:
 - ["computer_name"]

to

fields:
 - "computer_name"

[senatorhotchkiss]

I tried that with the same result Qmando :(

I have only been able to retrieve initial results "baseline", if I change the code in ruletypes.py, as you suggested and appending .keyword to the field.

[senatorhotchkiss]

I made a "curl json aggr query script" to run against my ES. It solved my needs. But I wish the EA could do the job. But thanks for all the insight. Made me a lot smarter :)