Elasticsearch: Reindex API should support aggregations

[bitkowitz]

I have a common use case of summarizing event-based indices into session-based indices. I currently achieve this by fetching data from elasticsearch to a compute environment, summarizing the data by session and then ingesting the result into a summary index. When this analysis involves high-frequency events, it becomes prohibitively expensive to transfer and process large volumes of events.

It would be great if the output of an aggregation could be ingested as a new index without having to leave the cluster. I had expected that the reindex api could be used for this type of job, but was dismayed that the source query does not support aggregations.

[bitkowitz]

BTW - I’ve seen notes that suggests the xpack watcher api can be repurposed for this type of job. It also looks like the rollup api can do something similar but only supports a limited set of summary statistics.

From an api standpoint, I would have expected es to support a general purpose “reduce” api that works with arbitrary pipeline aggregations.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jpountz]

@bitkowitz We discussed this feature request in FixitFriday and would like to ask you to share more information about your use-case. In particular, what would the aggregation look like in your case, and what kind of post-processing would you need to perform?

Leaving the discuss label so that it will be discussed again when you share more information.

[tomcallahan]

No additional feedback, closing

[tylerhoo]

It's too bad this was closed as I have a use case for this feature and was surprised and disappointed to find out aggregations are not supported by _reindex or by the Logstash Elasticsearch input plugin. I can provide more details if anyone is interested.