Kibana: [Security][Detections] Create Threshold-based Rule type
This issue is for creating a new rule type based on thresholds/aggregations, and can appear as a separate card for selection within the Define Rule section of the Create Rule flow.
There are actually several kinds of aggregation-based rules that could fall in here: # hits, sum, terms, significant terms, etc.
Latest mocks:
cc @marrasherrier
spong
All 8 comments
Pinging @elastic/siem (Team:SIEM)
elasticmachine
on 5 Jun 2020
Hi @spong ,
This is really encouraging and definitely a required feature. I just had a question on how do we handle data that is coming with a delay?
I agree that this is a broad question, and not really something that should concern the log aggregator. But in reality, getting logs in realtime is not always feasible. An obvious workaround would be to increase the timeframe, but then you have to accept a delay in terms of rule triggers!
NerdSec
on 23 Jun 2020
Hey there @NerdSec -- thanks for commenting! 馃檪
To better handle delayed events we'll be adding the ability to specify a Default Timestamp setting as part of the rule (https://github.com/elastic/kibana/issues/65941) so users can specify which timestamp field the rule will use during it's execution phase. This'll allow us to specify which timestamp is closest to ingest time, so if any data is delayed in making it into the system it'll still be processed as part of the next rule run. This'll be especially helpful for endpoints or other agents that are offline for a few days and then come back online and push up all their data. Tangentially, we also have some work with regards to mitigating gaps in rule runs -- you can see more about that here: https://github.com/elastic/kibana/issues/63290.
Hope this helps!
spong
on 23 Jun 2020
One small input here. In most of the use cases we have implemented, we ended up needing the terms, date_histogram, cardinality, and bucket_selector aggs. We did not the ESQL implementation, rather built the agg queries in the console, which wasn't quite user friendly.
Some use cases:
- Multiple logon failures from single source IP
- Concurrent user logins from multiple regions/geo locations
- Single source ip targeting multiple websites
- Multiple occurrences of an unique infection from the same machine
I am not sure if it was the correct route, but we implemented a terms agg/cardinality with a bucket selector to choose which events matched our condition!
NerdSec
on 24 Jun 2020
Had discussion with @spong @MikePaquette @marrasherrier to answer questions and decrease scope.
Product is comfortable with just using default fields of Detection Alert List and Timeline for threshold rule based alerts. Where a field would have multiple values, the alert list can show "Multiple Values" on field value. User should still be able to use the right click function to get top values and filter on values.
Scope will be cut to only have a query field, a group by field, and threshold count field for user to create a threshold rule.
Backend need is adding field value of "Threshold Rule" for rule kind and "threshold count".
dontcallmesherryli
on 1 Jul 2020
marrasherrier
on 1 Jul 2020
This is completed and shipping, so closing the ticket now.
FrankHassanabad
on 30 Jul 2020
Resolved by https://github.com/elastic/kibana/pull/71371! Thanks again @patrykkopycinski
spong
on 30 Jul 2020
Related issues
bhavyarm
路
3Comments
tbragin
路
3Comments
bradvido
路
3Comments
stacey-gammon
路
3Comments
Ginja
路
3Comments
Most helpful comment
One small input here. In most of the use cases we have implemented, we ended up needing the
terms,date_histogram,cardinality, andbucket_selectoraggs. We did not the ESQL implementation, rather built the agg queries in the console, which wasn't quite user friendly.Some use cases:
I am not sure if it was the correct route, but we implemented a terms agg/cardinality with a bucket selector to choose which events matched our condition!