Elasticsearch: DeleteByQueryConcurrentTests#testConcurrentDeleteByQueriesOnDifferentDocs fails with unexpected version conflict

[DaveCTurner]

Build scan: https://gradle-enterprise.elastic.co/s/owazjn6t7apj4/tests/:modules:reindex:test/org.elasticsearch.index.reindex.DeleteByQueryConcurrentTests/testConcurrentDeleteByQueriesOnDifferentDocs#1

Repro line: ./gradlew ':modules:reindex:test' --tests "org.elasticsearch.index.reindex.DeleteByQueryConcurrentTests.testConcurrentDeleteByQueriesOnDifferentDocs" -Dtests.seed=448D3E47C542E9A3 -Dtests.security.manager=true -Dtests.locale=es -Dtests.timezone=America/Indiana/Knox -Druntime.java=11

Reproduces locally?: Yes (but not every time)

Applicable branches: master

Failure history: https://build-stats.elastic.co/goto/2cba7892eed507aeab72b0c1fd55da0c

First failure (outside of PRs) for a long while.

Failure excerpt:

sep 18, 2020 2:42:24 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[Thread-6,5,TGRP-DeleteByQueryConcurrentTests]
java.lang.AssertionError: 
Expected: updated matches <0L> and created matches <0L> and deleted matches <43L> and versionConflicts matches <0L> and failures size matches <0> and reason cancelled matches null and slices matches an empty collection
     but: was <BulkByScrollResponse[took=1.3s,timed_out=false,sliceId=null,updated=0,created=0,deleted=43,batches=1,versionConflicts=1,noops=0,retries=0,throttledUntil=0s,bulk_failures=[{"index":"test","id":"fbwpoHQBxht_vE638ENh","cause":{"type":"version_conflict_engine_exception","reason":"[fbwpoHQBxht_vE638ENh]: version conflict, required seqNo [23], primary term [1]. but no document was found","index_uuid":"EJyLe_9CQ1-VzaCOKuL4ow","shard":"2","index":"test"},"status":409}],search_failures=[]]>
    at __randomizedtesting.SeedInfo.seed([448D3E47C542E9A3]:0)
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
    at org.junit.Assert.assertThat(Assert.java:956)
    at org.junit.Assert.assertThat(Assert.java:923)
    at org.elasticsearch.index.reindex.DeleteByQueryConcurrentTests.lambda$testConcurrentDeleteByQueriesOnDifferentDocs$0(DeleteByQueryConcurrentTests.java:59)
    at java.base/java.lang.Thread.run(Thread.java:834)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Reindex)

[costin]

Thanks for already picking this failure up.

[costin]

A somewhat similar failure on 7.x:
https://gradle-enterprise.elastic.co/s/x6lqxlhjq3h62

[dnhatn]

This test failed because the query phase returns duplicate Lucence document ids. I am not sure, but this relates to the recent Lucene upgrade in #62450. @jpountz @jimczi Can you please take a look? This test is reproducible after a few iterations on both master and 7.x.

[droberts195]

There have been loads of failures of this over the weekend, so I have muted the test:

master: cad2560d201200d0f4ae5d3c6d102ca1adcc5b1a
7.x: ecf264f61b22a51d50e1bc2448ac89a7f26365d8

[jimczi]

The issue is due to https://issues.apache.org/jira/browse/LUCENE-9449 which is present in the Lucene upgrade. @mayya-sharipova is looking.

[mayya-sharipova]

We have fixed the bug in Lucene: https://github.com/apache/lucene-solr/pull/1915.
We need to upgrade elasticsearch to a new snapshot of Lucene to enable these tests again.