Elasticsearch: Performance degradation for '*' query with '_all' field disabled

[jorgelbg]

Elasticsearch version: 5.4.0 - 5.4.3

Plugins installed: []

JVM version:

java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

OS version:

Linux debian-elasticseearch-client-test 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux

After upgrading to ES 5.4.3 the following query:

{
   "query":{
      "bool":{
         "must":[
            {
               "query_string":{
                  "analyze_wildcard":true,
                  "query":"*"
               }
            },
            {
               "range":{
                  "@timestamp":{
                     "gte":1499227272692,
                     "lte":1499241672693,
                     "format":"epoch_millis"
                  }
               }
            }
         ],
         "must_not":[

         ]
      }
   }
}

becomes really slow, after a little of investigation sending this query into to the validate API:

$ curl -XGET http://10.1.12.84:9200/logs/_validate/query?rewrite=true\&pretty\=true -d '{
   "query":{
      "bool":{
         "must":[
            {
               "query_string":{
                  "analyze_wildcard":true,
                  "query":"*"
               }
            },
            {
               "range":{
                  "@timestamp":{
                     "gte":1499227272692,
                     "lte":1499241672693,
                     "format":"epoch_millis"
                  }
               }
            }
         ],
         "must_not":[

         ]
      }
   }
}'

returns the following explanation:

"explanation": "(ConstantScore(_field_names:auth) | ConstantScore(_field_names:type) | ConstantScore(_field_names:redis) | ConstantScore(_field_names:header.senderId.keyword) | ConstantScore(_field_names:jsperf_content) | ConstantScore(_field_names:path) | ConstantScore(_field_names:protocol) | ConstantScore(_field_names:hostname) | ConstantScore(_field_names:java) | ConstantScore(_field_names:unique_id.keyword) | ConstantScore(_field_names:fw_proto.keyword) | ConstantScore(_field_names:unique_ident) | ConstantScore(_field_names:ua_device) | ConstantScore(_field_names:datacenter.keyword) | ConstantScore(_field_names:memcache) | ConstantScore(_field_names:jsperf_cache) | ConstantScore(_field_names:jsperf_dom_complete) | ConstantScore(_field_names:jsperf_first_paint) | ConstantScore(_field_names:unique_id) | ConstantScore(_field_names:method) | ConstantScore(_field_names:ua_build) | ConstantScore(_field_names:query) | ConstantScore(_field_names:http_version) | ConstantScore(_field_names:jsperf_load) | ConstantScore(_field_names:geoip.isp.keyword) | ConstantScore(_field_names:forwarded) | ConstantScore(_field_names:x_forwarded_for) | ConstantScore(_field_names:tags) | ConstantScore(_field_names:local_ip) | ConstantScore(_field_names:ua_minor) | ConstantScore(_field_names:size) | ConstantScore(_field_names:ua_major) | ConstantScore(_field_names:user_agent.analyzed) | ConstantScore(_field_names:protocol.keyword) | ConstantScore(_field_names:tracking_ident) | ConstantScore(_field_names:header.senderId) | ConstantScore(_field_names:jsperf_connect) | ConstantScore(_field_names:server_name) | ConstantScore(_field_names:referer_complete) | ConstantScore(_field_names:fw_proto) | ConstantScore(_field_names:code) | ConstantScore(_field_names:target_url) | ConstantScore(_field_names:tags.keyword) | ConstantScore(_field_names:ms_ident) | ConstantScore(_field_names:jsperf_dns) | ConstantScore(_field_names:tracking_ident.keyword) | ConstantScore(_field_names:private_ip) | ConstantScore(_field_names:forwarded_for) | ConstantScore(_field_names:ua_patch) | ConstantScore(_field_names:page_id) | ConstantScore(_field_names:auth_name) | ConstantScore(_field_names:referer_host) | ConstantScore(_field_names:jsperf_dom_loaded) | ConstantScore(_field_names:geoip.country_code3) | ConstantScore(_field_names:geoip.country_code2) | ConstantScore(_field_names:solr) | ConstantScore(_field_names:jsperf_first_byte) | ConstantScore(_field_names:user_agent) | ConstantScore(_field_names:ua_os_name) | ConstantScore(_field_names:customdate) | ConstantScore(_field_names:proxy_list) | ConstantScore(_field_names:url_path) | ConstantScore(_field_names:ftp) | ConstantScore(_field_names:auth_name.keyword) | ConstantScore(_field_names:proxy_list.keyword) | ConstantScore(_field_names:forwarded_for.keyword) | ConstantScore(_field_names:geoip.asn) | ConstantScore(_field_names:ua_os_major) | ConstantScore(_field_names:geoip.number) | ConstantScore(_field_names:jsperf_redirect) | ConstantScore(_field_names:ua_os) | ConstantScore(_field_names:geoip.city_name) | ConstantScore(_field_names:datacenter) | ConstantScore(_field_names:xcache) | ConstantScore(_field_names:geoip.isp) | ConstantScore(_field_names:geoip.postal_code) | ConstantScore(_field_names:@timestamp) | ConstantScore(_field_names:ua_name) | ConstantScore(_field_names:ua_os_minor) | ConstantScore(_field_names:response_time) | ConstantScore(_field_names:jsperf_dom_inter) | ConstantScore(_field_names:db))"

Which means that is using the _field_names field causing the query to be extremely slow. By comparison the same query with the same index/mapping on ES 5.2.2-5.3.3 would output the following explanation:

"explanation": "+*:* +MatchNoDocsQuery["User requested "match_none" query."]"

We've disabled the _all field. Disabling the _field_names field doesn't help either, and we get this output on the validate API:

"explanation": "+(auth:* | type:* | redis:* | header.senderId.keyword:* | jsperf_content:* | path:* | protocol:* | hostname:* | java:* | unique_id.keyword:* | fw_proto.keyword:* | unique_ident:* | ua_device:* | datacenter.keyword:* | memcache:* | jsperf_cache:* | jsperf_dom_complete:* | jsperf_first_paint:* | unique_id:* | method:* | ua_build:* | query:* | http_version:* | jsperf_load:* | geoip.isp.keyword:* | forwarded:* | x_forwarded_for:* | tags:* | local_ip:* | ua_minor:* | size:* | ua_major:* | user_agent.analyzed:* | protocol.keyword:* | tracking_ident:* | header.senderId:* | jsperf_connect:* | server_name:* | referer_complete:* | fw_proto:* | code:* | target_url:* | tags.keyword:* | ms_ident:* | jsperf_dns:* | tracking_ident.keyword:* | private_ip:* | forwarded_for:* | ua_patch:* | page_id:* | auth_name:* | referer_host:* | jsperf_dom_loaded:* | geoip.country_code3:* | geoip.country_code2:* | solr:* | jsperf_first_byte:* | user_agent:* | ua_os_name:* | customdate:* | proxy_list:* | url_path:* | ftp:* | auth_name.keyword:* | proxy_list.keyword:* | forwarded_for.keyword:* | geoip.asn:* | ua_os_major:* | geoip.number:* | jsperf_redirect:* | ua_os:* | geoip.city_name:* | datacenter:* | xcache:* | geoip.isp:* | geoip.postal_code:* | @timestamp:* | ua_name:* | ua_os_minor:* | response_time:* | jsperf_dom_inter:* | db:*) +MatchNoDocsQuery["User requested "match_none" query."]"

This is especially problematic because the default query for discovery on Kibana is *, which is causing really slow response times in our setup. Looks like in previous versions, this query would've been rewritten to a MatchAllDocsQuery.

[dakrone]

Relates to https://github.com/elastic/kibana/issues/12097, Kibana needs to change to using *:* for a query rather than just * to mitigate this.

[jorgelbg]

It's true that kibana should default to the match_all query on discovery only limited by the time range, but on the other hand, in this case, a '*' query is not going to mean a match_all query anymore but rather "everything with some value in some field" which is definitively slow on any big setup, beyond the Kibana issue is something that could cause problems for anyone upgrading, in this case happens that is used by default on kibana.

[dakrone]

a '*' query is not going to mean a match_all query anymore but rather "everything with some value in some field" which is definitively slow on any big setup, beyond the Kibana issue is something that could cause issues for anyone upgrading

I agree, I believe that we should special case * to be a match all query. The issue with expanding to field names was introduced in https://github.com/elastic/elasticsearch/pull/23433, so @jimczi: how would you feel about special-casing * to be a MatchAllDocsQuery internally? I think the foo:* case is still important, so it would apply only to

{
  "query": {
    "query_string": {
      "query": "*"
    }
  }
}

[jimczi]

how would you feel about special-casing * to be a MatchAllDocsQuery internally

I think we should do https://github.com/elastic/elasticsearch/issues/25551 first and then we can have a special case for:

{ "query": { "query_string": { "query": "*", "fields": "*" } } }

... which seems more natural since it will rewrite this query as a real *:*.
Then if _all is deactivated and the default search field is not specified, any query_string query would default to fields:* and the optim could be applied in this context seamlessly.

[akotlar]

+1. A interface-layer version of this saves me 20s on 89 million records. Thanks for bringing up the issue, I wasn't sure if the performance regression I was seeing was limited to '*'

[jorgelbg]

👍 This also has one additional side effect on Kibana/Grafana if you have a lot of visualizations/dashboards already created they have been stored in ES with the

"query":{"query_string":{"query":"*","analyze_wildcard":true}}

which will cause all your visualizations/dashboards to become slower.