Elasticsearch: New default - make search results fail rather than return partial results

[markharwood]

7.0 represents an opportunity to make a break with our more lenient past where we would _always_ try return partial hits or aggregations in the event of failures.

When failures occur hits and aggregations are returned alongside the information that says how many shards failed or if the search timed out. The suspicion is that most client applications will pay little attention to these important failure flags and unwittingly present partial results to clients with no indication that the results are incomplete.

https://github.com/elastic/elasticsearch/issues/27435 introduced a new flag which defines a cluster-level policy with the option of a per-query override. Currently the cluster default is to stick with the lenient policy of returning partial results rather than a complete failure.

The proposal is that we change the cluster default setting to a strict policy which will return an error and no search results for the scenarios that can lead to partial results e.g.

• A failure to execute within a timeout associated with the search *
• Missing shards (ie a red index)
• Failed shards - e.g. an error executing the query on index 2 of N because fielddata disabled

* _Related issue: should we set a default cluster setting for timeouts?_

[markharwood]

This is an important choice to make so we will write a blog post explaining the pros and cons and invite opinions to be expressed here.

WE WOULD LIKE TO GAUGE FEEDBACK BY COUNTING REACTIONS SO.....

• if you want to change to errors-by-default then THUMB UP this comment
• If you want to stick with partial results then THUMBS DOWN this comment.

[mattweber]

27435 is tagged for 7x only, can we get this back ported to a 6x release so we can experiment?

[tvernum]

@mattweber It looks like #27435 is incorrectly tagged.
It was backported to 6.x in https://github.com/elastic/elasticsearch/commit/6165676b369d51c9e4498c895460c89d4df741c5

@markharwood Can you confirm and update the version tags for #27435 ?

[markharwood]

My bad. Labelled the PRs but forgot the issue. This flag will debut in 6.3

[elasticmachine]

Pinging @elastic/es-search-aggs

[markharwood]

Update - we are not proposing making this breaking change in 7.0
We'd like to gather more feedback first. We have a blog lined up designed to promote discussion and gather feedback more generally.

[MosheMalin]

Can it be controlled in 6.X somehow?
Alternatively, while viewing a dashboard, where can I find an indication that the results are partial?
It happens to me from time to time, and it may lead to false alarms.

[markharwood]

Can it be controlled in 6.X somehow?

Yes, at cluster and query level. since 6.3.

while viewing a dashboard, where can I find an indication that the results are partial?

This is what I see (Kibana 6.6) when partial results are allowed and there's a partial failure:

The partial results are shown but with a warning message.

This is what I see when partial results are disabled in cluster settings and there was an issue:

Results are not re-drawn in Kibana (it keeps whatever data was visualised there before) but shows the failure from the unsuccessful search. This is probably not the place to further discuss Kibana behaviour.