Elasticsearch: bool.max_clause_count is ignored in default distribution

[bdecarne]

Hi there !

When i'm trying to index a percolate query (query string with a lot of OR clauses), the server fails with max_clause_count error :

"caused_by": {
   "type": "too_many_clauses",
   "reason": "maxClauseCount is set to 1024"
}

I raised the indices.query.bool.max_clause_count value in configuration, restarted, checked the /_nodes entrypoint :

"indices": {
  "query": {
    "bool": {
      "max_clause_count": "10240"
    }
  }
}

But this parameter is ignored during percolate indexation...

Thx

Version : 6.6.1 (docker)

[elasticmachine]

Pinging @elastic/es-search

[andyb-elastic]

Thanks for reporting this, it looks like there's an issue with this setting in general in 6.6.1 (not particular to percolator or query string queries)

I'm able to reproduce this by

1. setting indices.query.bool.max_clause_count above 1024 in elasticsearch.yml before node startup (I set it to 5000)

2.

PUT /test/
{
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "message": {
          "type": "keyword"
        }
      }
    }
  }
}

1. Running a bool query against the field in this index with 2000 term queries in should clauses

I get

```
Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[test], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, source={"query":
org.elasticsearch.transport.RemoteTransportException: [SDLDw-H][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.index.query.QueryShardException: failed to create query:

    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:324) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:307) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:806) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:656) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:631) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:388) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService.access$100(SearchService.java:126) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:360) [elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) [elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]

Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
at org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:114) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:422) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:385) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:104) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$2(QueryShardContext.java:308) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:320) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:307) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:806) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:656) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:631) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:388) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService.access$100(SearchService.java:126) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:360) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.6.1.jar:6.6.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-02-28T12:44:02,655][DEBUG][o.e.a.s.TransportSearchAction] [SDLDw-H] All shards failed for phase: [query]

[markharwood]

This just came up in a support issue - not a percolator request but a regular search with a lot of terms.
The 6.6.0 distribution I tested failed but the 6.7 branch running in my IDE was OK

[ra-coder]

also find this bug

[markharwood]

Another case here: https://discuss.elastic.co/t/booleanquery-toomanyclauses-buyt-have-updated-config-file-to-increase-it/170858

[andyb-elastic]

I tested this on snapshot builds at the heads of master (8.0.0-SNAPSHOT), 7.x (7.1.0-SNAPSHOT), 6.7 (6.7.0-SNAPSHOT), and 6.6 (6.6.2-SNAPSHOT) and did not reproduce on any of them

[markharwood]

Reproduced on 7.0.0-beta1

config/elasticsearch.yml:

indices.query.bool.max_clause_count: 2048

Setting confirmed in http://localhost:9200/_nodes?pretty

    "indices" : {
      "query" : {
        "bool" : {
          "max_clause_count" : "2048"
        }
      }
    },

Using this python script I get an error complaining about a 1024 limit.

from elasticsearch.client import Elasticsearch
es = Elasticsearch()

indexName = "test"
es = Elasticsearch()
indexSettings = {
    "settings": {
        "number_of_shards": 1
    },
    "mappings": {
        # "_doc": {
        "properties": {
            "message": {
                "type": "keyword"
            }
        }
        # }
    }
}
if not es.indices.exists(index=indexName):
    es.indices.create(index=indexName, body=indexSettings)

# This is OK
maxIds = 1024
# But below fails with Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
# maxIds = 1025

numItems = 0
shoulds = []
while numItems < maxIds:
    numItems += 1
    shoulds.append({
        "term": {
            "message": "foo" + str(numItems)
        }
    })
q = {
    "size": 10,
    "query": {
        "bool": {
            "should": shoulds
        }
    }
}
results = es.search(index="test", body=q)

Can you share your test rig @andyb-elastic ?

[andyb-elastic]

Yeah this is what I'm doing

1. build from branch
2. Add indices.query.bool.max_clause_count: 5000 to config
3. curl -X PUT localhost:9200/test -H 'Content-Type: application/json' -d '{"settings":{"number_of_replicas": 0,"number_of_shards": 1},"mappings":{"properties": {"message": {"type": "keyword"}}}}'
4. curl --data @entire-long-bool-query.json -H 'Content-Type: application/json' localhost:9200/test/_search

The file with that query
entire-long-bool-query.json.txt

I was able to reproduce this on 7.0.0-SNAPSHOT (19fa399380f15c9d8bc6dc92601fba3ba48fa91f) from the default distribution (linux-tar) but not from the oss distribution (oss-linux-tar) so I'll go back and check the other versions' default distributions

[markharwood]

I wonder if it's a classloader thing - the code that sets and reads the BooleanQuery.maxClauseCount setting is using Lucene classes that are different to the ones executing search.

[andyb-elastic]

It looks like it's the default distribution, that's probably why it slipped through testing. I can reproduce it on the default distribution on master and other active branches

[jimczi]

I think I found the issue, the XContentObjectTransformer creates a SearchModule at https://github.com/elastic/elasticsearch/blob/3e04a90e9924f333f018c74a4c981eace7cb562b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/utils/XContentObjectTransformer.java#L44 and this code is executed after we created the original SearchModule so it overrides the max bool setting for the entire process. We should remove the SearchModule from the ml code or refactor the class so that we don't set the max boolean clause in the constructor.

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

We should remove the SearchModule from the ml code or refactor the class so that we don't set the max boolean clause in the constructor.

We will do the first of these things, as there is another problem with having a static copy in ML in that Collections.emptyList() is passed for the list of search plugins. This means that extra search functionality added by search plugins is missing. We can solve both discrepancies by using the one created in the Node constructor.

[jimczi]

Good point, thanks @droberts195

[benwtrent]

@jimczi ^^ PRs have been merged that address ML constructing a new SearchModule with empty settings. Backported to 6.7, 7.0, 7.1.

[jimczi]

Thanks a ton @benwtrent!

[AShyPig]

@benwtrent I encountered the same problem in version 6.6.2. Is version 6.6.2 not fixed?

[droberts195]

@AShyPig this bug is present in all 6.6.x versions. It is fixed in 6.7.0.