Elasticsearch: Record number of buckets removed via bucket_selector

[polyfractal]

It could be useful/convenient to know how many buckets the bucket_selector pipeline agg removes during runtime.

I'm not sure if/how/where the response would go, however, since the bucket_selector doesn't actually "enrich" the response...just prune it. Adding to the parent agg would violate the internal immutability of element, and appending a new child bucket seems like a poor solution.

[michaelbaamonde]

The only other place that springs to mind is the aggregation metadata. That's still tricky, since it's a client-provided object. Aside from breaking the current contract ("metadata's yours and yours alone, user!"), it would also introduce the need for blacklisting a key (buckets_removed or whatever) to guard against the scenario in which user-provided metadata contains that same key already.

It does have the benefit of not messing with the semantics of the aggregation itself, however, as would be the case if we just added another child bucket.

Alternatively, we could add an altogether new metadata object for providing "internal" metadata at the aggregation level. That's probably a better, more flexible approach, as it would also allow us to add other kinds of metadata in the future without worrying about the problems I mentioned above re: user-provided metadata.

[jpountz]

This does not feel clean to me in terms of API. If this is something clients need to know, maybe they should do the pruning on client side?

[colings86]

currently the bucket_selector agg itself actually has no output (since all it does is prune a sibling agg) so we could add this information as it's own output. I am a little worried though that we will and up with users wanting to be able to turn this output off so we may need to expose it as an option on the bucket selector if we do implement it

[markharwood]

cc @elastic/es-search-aggs