Elasticsearch: Composite aggregation not abiding by query filter

[snikch]

Elasticsearch version (bin/elasticsearch --version):

6.2.1

Plugins installed: []

JVM version (java -version):

docker

OS version (uname -a if on a Unix-like system):

docker

Description of the problem including expected versus actual behavior:

I'm looking into implementing the composite aggregate as we effectively do this work ourselves app side, however I can't event get the simplest example to work with filtered results, it appears to always include all documents.

As an example below, I create an index with two documents. When I just search I can happily filter down to a single document by adding a Term filter on the ProductID. When I add the composite aggregate however I get two buckets, and the documents now include the unfiltered results.

Am I missing something? You can't use a filter aggregate because composite aggregates don't work as children.

Steps to reproduce:

Setup

curl -XPUT http://127.0.0.1:9206/composite_test -H "Content-Type: application/json" --data '{}'
curl -XPUT http://127.0.0.1:9206/composite_test/_mapping/test -H "Content-Type: application/json" --data '{"properties":{"ProductID":{"type":"keyword"},"Amount":{"type":"double"}}}'
curl -XPUT http://127.0.0.1:9206/composite_test/test/1 -H "Content-Type: application/json" --data '{"ProductID":"one","Amount":123}'
curl -XPUT http://127.0.0.1:9206/composite_test/test/2 -H "Content-Type: application/json" --data '{"ProductID":"two","Amount":321}'

Simple Doc Search (returns expected results)

curl -XPOST http://127.0.0.1:9206/composite_test/test/_search?pretty -H "Content-Type: application/json" --data '
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "ProductID": "one"
          }
        }
      ]
    }
  }
}
'
{
  "query": {
          "term": {
            "ProductID": "one"
          }
  }
}
'

```json
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.0,
"hits" : [
{
"_index" : "composite_test",
"_type" : "test",
"_id" : "1",
"_score" : 0.0,
"_source" : {
"ProductID" : "one",
"Amount" : 123
}
}
]
}
}

**Aggregate search** (Returns unexpected additional results in both `hits` and `aggregations`)

curl -XPOST http://127.0.0.1:9206/composite_test/test/_search?pretty -H "Content-Type: application/json" --data '
{
"aggregations": {
"flattened": {
"composite": {
"sources": [
{"product_variant": {"terms": { "field": "ProductID" }}}
]
},
"aggregations": {
"total": {
"sum": {
"field": "Amount"
}
}
}
}
}
}
},
"query": {
"term": {
"ProductID": "one"
}
}
}
'

```json
{
  "took" : 6,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "composite_test",
        "_type" : "test",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "ProductID" : "two",
          "Amount" : 321
        }
      },
      {
        "_index" : "composite_test",
        "_type" : "test",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "ProductID" : "one",
          "Amount" : 123
        }
      }
    ]
  },
  "aggregations" : {
    "flattened" : {
      "buckets" : [
        {
          "key" : {
            "product_variant" : "one"
          },
          "doc_count" : 1,
          "total" : {
            "value" : 123.0
          }
        },
        {
          "key" : {
            "product_variant" : "two"
          },
          "doc_count" : 1,
          "total" : {
            "value" : 321.0
          }
        }
      ]
    }
  }
}

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@jimczi could you take a look?

[jimczi]

Your json is invalid, you have an extra bracket at the end of the aggregations section that closes the entire body. Because of this the query part is simply omitted from the request, I don't know why we have this leniency in the parsing code of the request but I'll open a new issue for this since the problem is not about the composite aggregation.

[snikch]

Wow, how embarrassing. Thanks @jimczi. Have confirmed working :joy:

[jpountz]

I think it's even more embarrassing for us that illegal input is silently ignored. ;-)