Elasticsearch: Expand testing of queries

[javanna]

Similar to what we attempted with #11608, we should find a way to expand the range of queries that we test against in our query tests. Our random query generator currently generates a query out of a very limited set of supported queries. Also those queries are always leaf queries. See https://github.com/elastic/elasticsearch/blob/master/core/src/test/java/org/elasticsearch/index/query/RandomQueryBuilder.java#L41 .

We should expand the set of random queries, at the same time without going nuts otherwise or we'll end up with too complicated queries that make test failures super tricky to debug. For instance it's important to set a reasonable maximum depth in case we introduce compound queries.

This might end up being a big-ish change to the query test infra given that at the moment our tests depend on each other (e.g. to generate a random term on any test query we do new TermQueryBuilderTests().createTestQueryBuilder()). Maybe we should extract that logic from the tests like I tried to do in #11608, not sure.

Also, we should do the same for span queries. At the moment we only create span_term queries, but span queries can be combined together (e.g. span_or that contains span_not etc.) and we never test that (besides few specific tests that have been written).

Also, while we make changes we should take into account some todos that are currently in SearchSourceBuilderTests (see https://github.com/elastic/elasticsearch/blob/master/core/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java#L214). Seems like we should find a way to test against random queries when generating a random search source too. Maybe we just need to make the query generation a little more generic.

[javanna]

@cbuescher ping I thought this might interest you as you've been working on query testing, or maybe @MaineC as well.

[clintongormley]

@cbuescher can this be closed?

[cbuescher]

@clintongormley I think this is still on the wantlist.

[mayya-sharipova]

cc @elastic/es-search-aggs

[mayya-sharipova]

@cbuescher Do you know about the status of this issue? What is the progress on it or can it be closed?

[cbuescher]

I think it is something "nice to have", although given the time it has been open and not been worked on I doubt its urgency. Maybe we need to revisit the reasons we initially opened this and decide on whether we really need this, given its quiet an involved change (I think, but I might be wrong there). @javanna additional thoughts? Should we have a short focused discussion about this?

[javanna]

I would lean towards closing this. We already do a lot of randomization in all the query tests, and I don't think we want to complicate things further. Also, in three years we haven't needed this, so probably we never will :)