Elastalert: Making use of metric aggregations for windowed comparisons

[buinauskas]

Looking at latest merges, it seems that now you have added metric aggregations, which is super nice!

However I'm quite a newbie in regards to both ElastAlert and Python and wonder whether it's possible to create such an alert:

• I got users a, b and c
• I log every SQL statement that has been captured by each of these users
• Each log has a field called duration
• Compare average of duration for each user from previous week to current one

For reference, I'd use query such as to get values for comparison:

GET /queries-*/_search
{
  "size": 0,
  "aggs": {
    "user": {
      "terms": { "field": "user_name", "size": 10000 },
      "aggs": {
        "historic": {
          "range": { "field": "@timestamp", "ranges": [ { "to": "now-1w/d" } ] },
          "aggs": { "avg_duration": { "avg": { "field": "duration" } } }
        },
        "current": {
          "range": { "field": "@timestamp", "ranges": [ { "from": "now-1w/d" } ] },
          "aggs": { "avg_duration": { "avg": { "field": "duration" } } }
        }
      }
    }
  }
}

Many thanks,
Evaldas

[mircopolo]

Not right now, currently it can only look at the one window with static thresholds. As I was writing the metrics PR I did have the idea of supporting a select few pipeline aggregations like derivatives (probably the type you need for your use case), moving averages etc. This is one of the reasons I introduced the bucket_interval sub window, which is not strictly necessary to support the current two aggregation based rules (metric and percentage match), but would be needed for potential future window comparison variants whatever they might ultimately look like.

In the end as pipeline aggregations can be quite complex, I couldn't really think of a good, general purpose use cases that would also be easy to configure (the most important part), so any suggestion on what rules like this should look configuration wise behave would be most welcome (Although I don't think I will personally be in a position to add any enhancements in the near future).

In principal it shouldn't be too hard to subclass BaseAggregationRule to accomplish this, the only real stumbling block would be how to handle the first bucket, as this will always have no value in a pipeline aggregation, so one would have to handle these somehow in the core elastalert code that queries elasticsearch (You'd need to somehow go back past the previous end time to avoid gaps).