Elastalert: How to access attach_related field?

[MaxFlanders]

Near the end of the documentation for the Frequency alert type, there is an option called attach_related which states:

attach_related: Will attach all the related events to the event that triggered the frequency alert. For example in an alert triggered with num_events: 3, the 3rd event will trigger the alert on itself and add the other 2 events in a key named related_events that can be accessed in the alerter.

How do we access the "related_events" key in our alerter (right now we are focusing on slack alerts). Specifically, if I would like to display a list of all values of a specific field from all events that triggered the alert, is there a way to do that??

[Qmando]

You can use top_count_keys if you just want to get a list of values for the field. You could also potentially use aggregation + summary table fields (http://elastalert.readthedocs.io/en/latest/ruletypes.html#summary-table-fields).

If you have a custom alerter, you can access the related events with match['related_events']. I'm not sure if that's what you are asking or not..

[jocooler]

@Qmando - I'm running into the same issue. I'm trying to find a way to get a field out of a related event for the alert text. You can see some of the things I've tried below.

beat.hostname and metricset.module work, but all the permutations of related_events below show <MISSING VALUE>

Is there a way to get to these fields to interpolate them into the alert text?

type: frequency
index: metricbeat-*
num_events: 50
timeframe:
  minutes: 15
attach_related: true
email_format: html
alert_text_type: exclude_fields
alert_text: "<h3>Quoted ElastAlert Notification for {2}</h3><h4>{1}</h4><p>Triggering events ({7} total): <ul><li>{3}: {4}</li><li>{5}: {6}</li><li>{8}: {9}</li><li>{10}: {11}</li></ul></p><p>For complete information, consult the <a href='{0}'>Dashboard</a>.</p>"
alert_text_args:
  - kibana_link
  - description
  - name
  - beat.hostname 
  - metricset.module
  - match['related_events'][0].beat.hostname
  - match['related_events'][0].metricset.module
  - num_matches
  - "related_events[1].beat.hostname"
  - "related_events[1].metricset.module"
  - related_events[2].beat.hostname
  - related_events[2].metricset.module

[Qmando]

This should be possible, though I haven't tested it myself. You'll want to add something like {[1][beat][hostname]} (or maybe {[1][beat.hostname]}) in the alert_text and then just add related_events in alert_text_args.

ElastAlert is basically doing alert_text.format(*[match[field] for field in alert_text_args]), so you can take advantage of Python's string formatting language.

[jocooler]

Thanks so much for the quick reply Qmando. This format ended up working for me was:

{argument#[index][property]}

Like this:

alert_text: "<h3>Notification for {2}</h3><h4>{1}</h4><p>Triggering events ({5} total): <ul><li>{3}: {4}</li><li>{6[1][beat][hostname]}: {6[1][metricset][module]}</li><li>{6[2][beat][hostname]}: {6[2][metricset][module]}</li></ul></p><p>For complete information, consult the <a href='{0}'>Dashboard</a>.</p>"
alert_text_args:
  - kibana_link
  - description
  - name
  - beat.hostname 
  - metricset.module
  - num_matches
  - related_events

If I may suggest, since there are a few threads like this floating around, the documentation for how to access that string be added to the attach_related docs.

[ksemaev]

@jocooler TY for explanation! Could you please provide an example of alert you get?
And what are those numbers in {6[1][beat][hostname]} ? this 1 and 2?

[jocooler]

@ksemaev - {6} is the 6th argument passed to the formatter. They're 0-indexed so, in my case 6 is related_events.
The actual example would look something like:

---

Notification for My Sample Alert

3 or more servers were above 80% resource consumption in the last 15 minutes.

Triggering events (3 total):

• blue_mustang: RAM usage
• pink_maserati: Disk Filled space
• green_ferrari: RAM usage

For complete information, consult the Dashboard.

[jocooler]

@ksemaev - here's a more built out example we're actually using in production. With a couple of minor changes to the kibana URL, you should be able to get this working with regular heartbeat-* data.

I created a gist with a complete rule: https://gist.github.com/jocooler/4312537eab4f6053c6d0bf45a1925065

And here's an actual alert it generated: https://gist.github.com/jocooler/69ee923ad025bd94844c79134878d2a1

Which looks like this:

Subject: ElastAlert - Sites Down - ArcGIS Server was down at 2018-11-08T09:21:28.210Z

---

ElastAlert Notification: Sites Down

Site pings returned down more than once in 15 minutes.

A total of 2 down sites were detected. For more information, consult the Dashboard.

Trigger events:

md5-6d527d257cc470cd243b96db9785b275

Technical details:

md5-c3ca0e263349de034c069209a90b8c94 At least 2 events occurred between 2018-11-08 04:06 Eastern Standard Time and 2018-11-08 04:21 Eastern Standard Time

[ksemaev]

@ksemaev - {6} is the 6th argument passed to the formatter. They're 0-indexed so, in my case 6 is related_events.
The actual example would look something like:

Notification for My Sample Alert

3 or more servers were above 80% resource consumption in the last 15 minutes.

Triggering events (3 total):

* blue_mustang: RAM usage

* pink_maserati: Disk Filled space

* green_ferrari: RAM usage

For complete information, consult the Dashboard.

TY very much for explaining! The part I do not get is about the second number in {6[1][beat][hostname]} - the [1].
So still, sorry for beeing slow, but even from your extended reply I can't understand how do you create a list of related events, with number of lines equal to the number of events actually happened :(

[jocooler]

@ksemaev - ah, I understand now. The second number is the index of the event. Related events is a list, so you have to get the one you want by index. So it's like {argument[event_index][property][subproperty]}

[swethashk]

please provide the solution on how to easily access related_events field without giving this format {6[1][beat][hostname]}
Still, I'm getting a missing value

[rashmichoudhary07]

@jocooler
Hello,Thanks! this one helped me to figure out how to access individual elements from related_events. I did the below to pass related_events to command alert:

command: "echo %(@timestamp)s %(log_json.transaction.conversationId)s {related_events[0][log_json][transaction][conversationId]} >> path_to_file/command_alert.log"

However, i want to extract a specific field's value from all docs inside related_events. In my case, i cant specify all indices in command alerter since my query hits in prod will keep changing, it isnt feasible. Any idea on how can i all all indices in command alerter. This will really helpful!.

[rashmichoudhary07]

@jocooler
Also, i just noticed on a few checks that my related event is only holding one document data irrespective of query hits being >3. Why would that happen? PFB my rule file for reference.

type: frequency
num_events: 2
timeframe:
minutes: 30
index: elk_test2
filter:

• term:
  log_json.response.statusCode: "404"
  attach_related: true
  alert:
  
  
  
  • command
  
  

command: "echo %(@timestamp)s %(log_json.transaction.conversationId)s {related_events[0][log_json][transaction][conversationId]} {related_events[1][log_json][transaction][conversationId]} {related_events[2][log_json][transaction][conversationId]}>> path_to_file/command_alert.log"

Can't figure out why!