Elasticsearch: Composite aggregation doesn't allow filter aggregation

[thekofimensah]

Elasticsearch version (bin/elasticsearch --version): 6.2.2

I'd like to define a filters under the composite aggregation but it seems not allowed. Why not? I could really use it!

Which aggregations are allowed under the composite agg?

GET */_search
{
  "size": 0,
  "aggs": {
    "data": {
      "composite": {
        "size": 100,
        "sources": [
          {
            "personname": {
              "terms": {
                "field": "personname.keyword"
              }
            }
          },
          {
            "test": {
              "filters": {
                "filters": {
                  "query": {
                    "query_string": {
                      "query": "all"
                    }
                  }
                }
              }
            }
          }
        ]
      }
    }
  }
}

Error:

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "invalid source type: filters",
        "line": 17,
        "col": 26
      }
    ],
    "type": "parsing_exception",
    "reason": "[composite] failed to parse field [sources]",
    "line": 17,
    "col": 26,
    "caused_by": {
      "type": "parsing_exception",
      "reason": "invalid source type: filters",
      "line": 17,
      "col": 26
    }
  },
  "status": 400
}

[colings86]

You can't add the filters aggregation as a source in the composite aggregation as it only accepts term, histogram and date_histogram sources. However you can have a filters aggregation as a sub-aggregation as below:

GET */_search
{
  "size": 0,
  "aggs": {
    "data": {
      "composite": {
        "size": 100,
        "sources": [
          {
            "personname": {
              "terms": {
                "field": "personname.keyword"
              }
            }
          }
        ]
      },
      "aggs": {
        "test": {
          "filters": {
            "filters": {
              "query": {
                "query_string": {
                  "query": "all"
                }
              }
            }
          }
        }
      }
    }
  }
}

This means that although the filters aggregation is not involved in selecting the "page" for the aggregation it will still be returned for every bucket of the composite aggregation.

Does this work for your use-case?

[elasticmachine]

Pinging @elastic/es-search-aggs

[thekofimensah]

Thanks for the recommendation but that doesn't really get my use case taken care of. There's not much difference between that and a normal aggregation set. I'm pushing data into the new Vega visuals and they require certain formats of data. Thanks for the post

[colings86]

The difference is that the composite aggregation allows you to page through all the terms in personname.keyword whereas the terms aggregation would only provide the top N terms.

What is it that you are hoping to get from having the filters aggregation in the sources section of the composite aggregation?

[thekofimensah]

I only put one filter in there but a real example there are a couple.. I'd like to do the pagation through all the terms with filters instead of categorizing by term. I can't get my aggregation from the field itself, I need to get to filtering by the value of the fields.

I know this sounds nuts, but is there a way you can think of where I can do a composite agg but then limit my results only to the top 10 result sets. I need the data in the form of {person A: value1, filtera:value1}{personA:value2,filtera:value1} etc... but only the top results. This is a workaround I'm trying to create :)

[colings86]

is there a way you can think of where I can do a composite agg but then limit my results only to the top 10 result sets.

No, the intention of the composite agg is to provide a way to systematically page through all results. Inn order to get the top 10 results you would need to use the terms aggregation with a filters sub-aggregation. If you then need to flatten the response for use in your visualisation this flattening will need to be done client side.

[jimczi]

I agree with @colings86 if you need to flatten the top N results of an aggregation the composite aggregation is not the solution. You'll have to perform the flattening on the client side.
I am going to close this issue because the composite aggregation can already have a filter aggregation as a sub-aggregation but the feature request described here is different and cannot be addressed with this aggregation.

[iukea1]

yo you guys should make it so you can exclude stuff with composite