Kibana: Underlying Elasticsearch queries do not use filter context when appropriate

[peterskim12]

Kibana version: 5.0.0 alpha5

Elasticsearch version: 5.0.0 alpha5

Description of the problem including expected versus actual behavior:
I viewed the Request for a Vertical Bar Chart visualization, expecting the queries for the selected time range in the Dashboard and "drill down" selections to be explicitly defined in the filter context of the Elasticsearch query. However, I noticed they were added to the 'must' clause of the boolean query instead of the filter clause. Unless I'm misunderstanding something, this will result in the "filters" not being cached in the ES filter cache and add latency due to influencing scoring.

e.g. instead of:

"must":[
   {
      "query_string":{
         "query":"*",
         "analyze_wildcard":true
      }
   },
   {
      "match":{
         "geoip.country_name.keyword":{
            "query":"United States",
            "type":"phrase"
         }
      }
   },
   {
      "query_string":{
         "analyze_wildcard":true,
         "query":"*"
      }
   },
   {
      "range":{
         "@timestamp":{
            "gte":1431754102370,
            "lte":1433545147679,
            "format":"epoch_millis"
         }
      }
   }
]

I think it should ideally be generated to:

"must":[
   {
      "query_string":{
         "query":"*",
         "analyze_wildcard":true
      }
   }
],
"filter":[
   {
      "match":{
         "geoip.country_name.keyword":{
            "query":"United States",
            "type":"phrase"
         }
      }
   },
   {
      "range":{
         "@timestamp":{
            "gte":1431754102370,
            "lte":1433545147679,
            "format":"epoch_millis"
         }
      }
   }
]

Steps to reproduce:

1. Create any visualization using the Terms agg. Save it.
2. Add viz to a dashboard. Click on one of the term values in viz.
3. Click up arrow in lower left corner, select Request.

[spalger]

This sounds reasonable enough, we need to be careful to not negatively impact scoring with this.

Are any of these queries calculating meaningful score values?

[peterskim12]

So my expectation is that any query parameters entered in the search box should influence scoring, while any query parameters resulting from a "drill down" (aka selection of a value or range in a visualization) should not influence scoring -- that they are simply filtering criteria.

[Bargs]

@spalger do you know if there's any reason to calculate score when we're not sorting on score? I'm thinking these could even dynamically switch between query and filter context depending on what we're sorting on.

[peterskim12]

I don't want to presume that Kibana is only used for time-series use cases but until Kibana provides the ability for users to control relevancy, my inclination is to say that scoring is not very meaningful in Kibana. Maybe there's a use case I'm not considering...

[Bargs]

Yeah sorting by score generally only happens by default when using a non-time based index pattern I believe. Otherwise it sorts by your default timestamp field

[spalger]

I know that we supported sorting on score at some point, but I'm not 100% sure that it was something you could ever do without modifying the url directly.

[Bargs]

It's definitely still possible if you add _score as a column. I'm just not sure if _score would ever be important outside of sorting?

[lukasolson]

Should we close this, seeing as how Elasticsearch no longer supports "filters"?

[Bargs]

@lukasolson I think it's still relevant. We never put any queries under the filter context, but we probably should for anything that the user wouldn't expect to contribute to scoring in order to take advantage of the filter cache.

[JacobBrandt]

This is absolutely still relevant because it's a regression introduced in version 5. Filters were put in the filter context in past Kibana versions and they need to do this again.

[JacobBrandt]

Here's the 4.6 branch using the filtered query when it existed causing filters to not be used in scoring and use the filter cache.
https://github.com/elastic/kibana/blob/4.6/src/ui/public/courier/data_source/_abstract.js#L325

Here's the 5.0 branch that didn't take Elasticsearch's advice on how to convert filtered queries to the new bool query syntax and put filters in must and must_not.
https://github.com/elastic/kibana/blob/5.0/src/ui/public/courier/data_source/_abstract.js#L345

Here's Elasticsearch's documentation on how to convert filtered queries.
https://www.elastic.co/guide/en/elasticsearch/reference/5.0/query-dsl-filtered-query.html