Elasticsearch: [CI] Interesting refresh errors

Created on 18 Sep 2017  路  3Comments  路  Source: elastic/elasticsearch

The test in question (ParentChildClientYamlTestSuiteIT#{yaml=/20_parent_join/Test parent_id query}) seems to be unrelated to the refresh error that occurred:

 1> [2017-09-18T08:18:19,728][INFO ][o.e.j.ParentChildClientYamlTestSuiteIT] Stash dump on failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "_shards" : {
  1>         "total" : 10,
  1>         "successful" : 4,
  1>         "failed" : 2,
  1>         "failures" : [
  1>           {
  1>             "shard" : 2,
  1>             "index" : "test",
  1>             "status" : "INTERNAL_SERVER_ERROR",
  1>             "reason" : {
  1>               "type" : "refresh_failed_engine_exception",
  1>               "reason" : "Refresh failed",
  1>               "index_uuid" : "Bnhv8wlQQHK9VBRkcCjM1w",
  1>               "shard" : "2",
  1>               "index" : "test",
  1>               "caused_by" : {
  1>                 "type" : "null_pointer_exception",
  1>                 "reason" : null
  1>               }
  1>             }
  1>           }
  1>         ]
  1>       }
  1>     }
  1>   }
  1> }]
FAILURE 1.59s | ParentChildClientYamlTestSuiteIT.test {yaml=/20_parent_join/Test parent_id query} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [/20_parent_join:57]: expected [2xx] status code but api [indices.refresh] returned [500 Internal Server Error] [{"_shards":{"total":10,"successful":4,"failed":2,"failures":[{"shard":2,"index":"test","status":"INTERNAL_SERVER_ERROR","reason":{"type":"refresh_failed_engine_exception","reason":"Refresh failed","index_uuid":"Bnhv8wlQQHK9VBRkcCjM1w","shard":"2","index":"test","caused_by":{"type":"null_pointer_exception","reason":null}}}]}}]
   >    at __randomizedtesting.SeedInfo.seed([B44C6C8C3359D807:3C1853569DA5B5FF]:0)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:352)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:323)
   >    at java.lang.Thread.run(Thread.java:745)
   > Caused by: java.lang.AssertionError: expected [2xx] status code but api [indices.refresh] returned [500 Internal Server Error] [{"_shards":{"total":10,"successful":4,"failed":2,"failures":[{"shard":2,"index":"test","status":"INTERNAL_SERVER_ERROR","reason":{"type":"refresh_failed_engine_exception","reason":"Refresh failed","index_uuid":"Bnhv8wlQQHK9VBRkcCjM1w","shard":"2","index":"test","caused_by":{"type":"null_pointer_exception","reason":null}}}]}}]
   >    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:238)
   >    at org.elasti

Actual NPE on the server side:

[2017-09-18T12:18:19,590][WARN ][o.e.i.e.Engine           ] [node-0] [test][2] failed engine [refresh failed]
java.lang.NullPointerException: null
    at org.apache.lucene.codecs.lucene70.Lucene70DocValuesProducer.<init>(Lucene70DocValuesProducer.java:111) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat.fieldsProducer(Lucene70DocValuesFormat.java:132) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:292) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:375) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:175) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:94) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.0.0-snapshot-d94a5f0.jar:7.0.0-snapshot-d94a5f0 d94a5f00f08c1d0552320d3a87daeec9fab29c2f - jpountz - 2017-08-30 13:24:04]
    at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1189) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:830) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:58) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:38) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:999) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:977) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:340) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:280) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:940) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:937) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2120) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:949) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:95) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:301) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:276) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:263) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:649) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0-rc1-SNAPSHOT.jar:6.0.0-rc1-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

Build url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.0+periodic/31/

Console.txt

>test
Source
picture martijnvg

Most helpful comment

arg... I should've refreshed the browser! But good that we came to the same conclusion.

picture martijnvg on 18 Sep 2017
❤1 🎉1 👍1

All 3 comments

this looks like a JVM bug that we hit a lot of times on 8_111 I will make sure infra goes to a newer version

s1monw picture s1monw on 18 Sep 2017

This NPE occurred on a closing bracket. Very likely due to an old jvm version.

martijnvg picture martijnvg on 18 Sep 2017

arg... I should've refreshed the browser! But good that we came to the same conclusion.

martijnvg picture martijnvg on 18 Sep 2017
❤1 🎉1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

matthughes picture matthughes  路  3Comments
clintongormley picture clintongormley  路  3Comments
dawi picture dawi  路  3Comments
rbayliss picture rbayliss  路  3Comments
rpalsaxena picture rpalsaxena  路  3Comments