Kibana: Support Bucket Script Aggregation

[rcrezende]

Given the rhe new Bucket Script Aggregation, we can implement formulas from aggregated metric values (e.g. metric1 / metric2 * 100). This would allow calculation of e.g. ratio between two metric results.

https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-bucket-script-aggregation.html

Usage Scenario:
Links to: https://github.com/elastic/kibana/issues/3505

[rashidkpc]

Duplicate https://github.com/elastic/kibana/issues/4584

[Kazark]

@rashidkpc I don't see how this is a dupe of the pipelines stuff. The pipelines in in 5.4 but I don't see bucket script aggregations there. Can we re-open this?

[darrencruse]

I'm interested in this feature in order to do visualizations of values on documents that represent farmers fields where I need to weight the values greater for large fields and smaller for small fields, after the search query has selected the field documents to be included in the weighted average.

The bucket_script feature seems to be the answer (I have it working in elasticsearch) but I can't seem to reproduce such a query when using a Kibana visualization.

Are there any plans to support this soon?

Or is there any workaround?

(e.g. I was trying to see if I can directly tell Kibana the elasticsearch query I want used for a visualization instead of Kibana creating the query for me based on my UI choices - but that doesn't seem to be possible? Is that correct?)

[pant03]

+1

[smoreaud]

I would be interested in this functionality. In our use case, it would allow to work around with success/failure ratios.

[Spacefish]

+1 would like to see this as well! Should be pretty simple to implement.. Just a "Custom Metric" in the Dropdown which presents a box where you can input your aggregation JSON..

[tbragin]

Starting with Kibana 5.4, bucket script agg is now supported in Time Series Visual Builder, though it's called "Calculation" (cc @simianhacker I wonder if we should make it more explicit that this option relies on "Bucket Script"?)

Could folks on this ticket (e.g. @Kazark) give it a try and let us know if it helps in your use cases?

[alexfrancoeur]

I think @simianhacker has agreed that it makes sense to rename this aggregation. I've opened https://github.com/elastic/kibana/issues/13796 for tracking purposes.

[hetzer-fastec]

@tbragin Having the bucket script aggregation in the TSVB is a very nice feature, imo - thanks a lot for implementing it. The things which I tried worked very well.

Nevertheless it would be great if the TSVB supported additional script aggregations (e.g. scripted sum aggregation), as the bucket script aggregation can only fully leverage its potential, if all kinds of sub-aggregations are supported. But since this is not a place to discuss this in more detail, I have opened another issue.

[shaharmor]

Is there a timeline for this feature?

[ArnonMarcus]

The 'Calculate' feature can only do a bucket-script as a top-level, it can't be applied within a parent bucket-aggregation... :(

And that's only in the time-series visualization, there's no bucket-script in the standard visualizations.

Our particular use-cases require both, and it seems like none are supported - not even in 6.x(rc)
Are there any plans for this?

In our use-cases, each document has a start-time and end-time date-type fields, and we need to generate an overall-duration across buckets of those, grouped on some category-field.
This involves first bucketing on that category, then within that doing 2 metrics: min(start-time) and nax(end-time) of the documents in each bucket, then applying a pipeline-script-bucket agg that uses these 2 metric-aggs to compute the duration between them (max_end_time - min_start_time). That gives us a 'duration' metric as the output of each bucket. We then want to apply some metric-pipeline-aggs on them.

An almost identical structure has been detailed here: https://discuss.elastic.co/t/kibana-5-4-bucket-script-visualization/86475

I was able to express that in the query-DSL, and we're using it in production already - but we can't visualize these in Kibana - which REALLY sucks, as our clients have been requesting such visualizations for years, and we've always told them 'we need support from later versions, for pipeline-aggregations'...
Well, now we have those - but actually, don't...
In time-series builder, we have bucket-script pipeline aggregation that can be expressed, but it can't consume/be-applied-on a parent bucket-term-aggregation, and in the other visualization, we have bucket-term-aggregations, and some pipeline-metric-aggregations, but not the needed bucket-script pipeline aggregation needed to express that...

Is there a workaround using some "advanced JSON" thing I can put my query-DSL in there?

[nicoulaj]

Just adding my use case as requested in this question:

My index contains the following document type (which represents test executions):

• id of test (string)
• success (boolean)
• version of software

I want to build the following histogram:

• On X axis: versions
• On Y axis: number of unique tests, stacked on success/failure

The same test id can be executed several times on same version with different results: it should be counted as success if it is successful once.

I really hope this will be implemented, as this is a definitive showstopper for us.

[Spacefish]

Please just add "Bucket Script Aggregation" or "Custom Aggregation" as an option in the normal visualizations.. Should be easy, just like the other aggregation but without adding any content in the aggregation body (just the json supplied by the user)

[cjcenizal]

For anyone who's interested in an example of using this feature in TSVB, here's a relevant forum post: https://discuss.elastic.co/t/weighted-average-in-a-datatable-of-kibana/185909/2.

[ArnonMarcus]

The feature requested is NOT done(!) This was NOT about TSVB, but for the other regular visualization types(!) (bar charts and the like). We're now on 6.4 and we STILL CAN'T DO WHAT WE NEED. It's also not in 7.x ASAIK.
Again, this is about being able to choose "Bucket Script" in bar charts (it's NOT available there) and doing so at ANY nested-level of the overall aggregation.
We'd also need a "Bucket Filter" BTW, to be able to filter buckets coming out of that aggregation via a short script.

[cjcenizal]

CC @elastic/kibana-app

[sbo1695]

The 'Calculate' feature can only do a bucket-script as a top-level, it can't be applied within a parent bucket-aggregation... :(

And that's only in the time-series visualization, there's no bucket-script in the standard visualizations.

Our particular use-cases require both, and it seems like none are supported - not even in 6.x(rc)
Are there any plans for this?

In our use-cases, each document has a start-time and end-time date-type fields, and we need to generate an overall-duration across buckets of those, grouped on some category-field.
This involves first bucketing on that category, then within that doing 2 metrics: min(start-time) and nax(end-time) of the documents in each bucket, then applying a pipeline-script-bucket agg that uses these 2 metric-aggs to compute the duration between them (max_end_time - min_start_time). That gives us a 'duration' metric as the output of each bucket. We then want to apply some metric-pipeline-aggs on them.

An almost identical structure has been detailed here: https://discuss.elastic.co/t/kibana-5-4-bucket-script-visualization/86475

I was able to express that in the query-DSL, and we're using it in production already - but we can't visualize these in Kibana - which REALLY sucks, as our clients have been requesting such visualizations for years, and we've always told them 'we need support from later versions, for pipeline-aggregations'...
Well, now we have those - but actually, don't...
In time-series builder, we have bucket-script pipeline aggregation that can be expressed, but it can't consume/be-applied-on a parent bucket-term-aggregation, and in the other visualization, we have bucket-term-aggregations, and some pipeline-metric-aggregations, but not the needed bucket-script pipeline aggregation needed to express that...

Is there a workaround using some "advanced JSON" thing I can put my query-DSL in there?

@ArnonMarcus Was you able to achieve this. I have similar requirement but could not figure out how to implement in Kibana. I am currently on 6.7.1 version

[ArnonMarcus]

Nope... It's still an open issue.

[anottrott]

From my perspective as a data scientist the in ability to do basic math , e.g. ratios on aggregation queries, (this is called a Bucket Script Aggregation in Elastic speak) within Visualizations and Maps is a massive limitation of the entire Kibana platform

On the plus side you can achieve this functionality by making Custom Vega visualizations and maps. The only problem with that method is that you are limited to displaying 10000 values on your visualizations, which is pretty limiting for creating an sort of scatter plot or map with a lot of data in it

[czjxy881]

+1

[sameerpanicker]

+1

[ppf2]

One use case for this feature is to workaround Elasticsearch not having a non-negative option for derivative aggregation so that charts can end up showing negative derivative values (upon intentional server restarts): https://github.com/elastic/elasticsearch/issues/15542

[Sagesh]

+1

[fkelbert]

This is a much requested feature from users, similar to https://github.com/elastic/kibana/issues/17544.

Are there any plans to implement this?

[glmrenard]

Would be nice !

[AmandaBSobrinho]

+1

[suportecaso1]

+1

[GrahamHannington]

I'm late to this party, so my disappointment is fresh.

A few days ago, I created the Elastic discussion forum topic "Visualize a value calculated from the per-bucket sum of one field divided by the per-bucket sum of another?" (sorry about the verbose title).

Soon after, I found this issue, and belatedly realized why I couldn't figure out how to do that in "standard" Kibana visualizations ☹️.

I replied to my own forum topic, citing this issue, and showing basic examples of using bucket scripts in TSVB and—I guess this is why I'm posting this comment, in case it's helpful to anyone—Vega-Lite.

[fbaligand]

Waiting this wanted feature, an alternative for data table visualization is to use « enhanced table » community plugin, and its « computed columns » feature:
https://github.com/fbaligand/kibana-enhanced-table