Elasticsearch: Es 5.1.1: unassigned (p/r) shards and corrupt indices

[itkovian]

Elasticsearch version: 5.1.1

Plugins installed: []

JVM version: java-1.8.0-openjdk-headless-1.8.0.111-1.b15.el7_2.x86_64

~~
[root@tangela1 plugins]# java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)
~~

OS version: CentOS 7.2

~~
[root@tangela1 plugins]# uname -a
Linux tangela1.ugent.be 3.10.0-327.4.5.el7.ug.1.x86_64 #1 SMP Wed Jan 27 11:28:29 CET 2016 x86_64 x86_64 x86_64 GNU/Linux
~~

Description of the problem including expected versus actual behavior:

The status goes from green to red. We see unassigned shards appear (both primary and replica) and a bunch of errors from lucene telling its index files are broken.

The files mentioned in the logs, e.g., for the EOFException or the checksum being invalid, do not seem to exists on disk.

~~
[root@tangela1 elasticsearch]# ls /var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/2/index/_9j.fdx
ls: cannot access /var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/2/index/_9j.fdx: No such file or directory
~~

The number of allowed op file descriptors is

~~
[ageorges@tangela2 ~]$ curl -XGET 'http://192.168.24.1:9200/_nodes/stats/process?filter_path=**.max_file_descriptors'
{"nodes":{"dgM6WtvtS1eq3m0PQ-NF_A":{"process":{"max_file_descriptors":131072}},"i1GWKoadQKKiRWDJgZEqNg":{"process":{"max_file_descriptors":131072}}}}
~~

Steps to reproduce:

1. restart one of the master nodes (we have two nodes in the cluster, no really an ideal setup I guess)

* Logs *

~~~~
[2016-12-15T18:20:36,087][INFO ][o.e.n.Node ] [tangela1.ugent.be] initializing ...
[2016-12-15T18:20:36,165][INFO ][o.e.e.NodeEnvironment ] [tangela1.ugent.be] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/sdb)]], net usable_space [3.1tb], net total_space [4.3tb], spins? [possibly], types [xfs]
[2016-12-15T18:20:36,165][INFO ][o.e.e.NodeEnvironment ] [tangela1.ugent.be] heap size [29.8gb], compressed ordinary object pointers [true]
[2016-12-15T18:20:37,175][INFO ][o.e.n.Node ] [tangela1.ugent.be] node name [tangela1.ugent.be], node ID [i1GWKoadQKKiRWDJgZEqNg]
[2016-12-15T18:20:37,178][INFO ][o.e.n.Node ] [tangela1.ugent.be] version[5.1.1], pid[13530], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/3.10.0-327.4.5.el7.ug.1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_111/25.111-b15]
[2016-12-15T18:20:37,973][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [aggs-matrix-stats]
[2016-12-15T18:20:37,973][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [ingest-common]
[2016-12-15T18:20:37,973][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [lang-expression]
[2016-12-15T18:20:37,973][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [lang-groovy]
[2016-12-15T18:20:37,973][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [lang-mustache]
[2016-12-15T18:20:37,974][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [lang-painless]
[2016-12-15T18:20:37,974][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [percolator]
[2016-12-15T18:20:37,974][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [reindex]
[2016-12-15T18:20:37,974][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [transport-netty3]
[2016-12-15T18:20:37,974][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] loaded module [transport-netty4]
[2016-12-15T18:20:37,974][INFO ][o.e.p.PluginsService ] [tangela1.ugent.be] no plugins loaded
[2016-12-15T18:20:41,173][INFO ][o.e.n.Node ] [tangela1.ugent.be] initialized
[2016-12-15T18:20:41,174][INFO ][o.e.n.Node ] [tangela1.ugent.be] starting ...
[2016-12-15T18:20:42,203][INFO ][o.e.t.TransportService ] [tangela1.ugent.be] publish_address {192.168.24.1:9300}, bound_addresses {192.168.24.1:9300}
[2016-12-15T18:20:42,216][INFO ][o.e.b.BootstrapCheck ] [tangela1.ugent.be] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-12-15T18:20:45,704][INFO ][o.e.c.s.ClusterService ] [tangela1.ugent.be] detected_master {tangela2.ugent.be}{dgM6WtvtS1eq3m0PQ-NF_A}{AiYImWrQSk-DqPAv1vFlRA}{192.168.24.2}{192.168.24.2:9300}, added {{tangela2.ugent.be}{dgM6WtvtS1eq3m0PQ-NF_A}{AiYImWrQSk-DqPAv1vFlRA}{192.168.24.2}{192.168.24.2:9300},}, reason: zen-disco-receive(from master [master {tangela2.ugent.be}{dgM6WtvtS1eq3m0PQ-NF_A}{AiYImWrQSk-DqPAv1vFlRA}{192.168.24.2}{192.168.24.2:9300} committed version [1]])
[2016-12-15T18:20:45,724][INFO ][o.e.h.HttpServer ] [tangela1.ugent.be] publish_address {192.168.24.1:9200}, bound_addresses {192.168.24.1:9200}
[2016-12-15T18:20:45,724][INFO ][o.e.n.Node ] [tangela1.ugent.be] started
[2016-12-15T18:20:47,784][INFO ][o.e.c.s.ClusterSettings ] [tangela1.ugent.be] updating [cluster.routing.allocation.enable] from [ALL] to [all]
[2016-12-15T18:40:15,499][WARN ][o.e.a.b.TransportShardBulkAction] [tangela1.ugent.be] unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest to [logstash-2016.12.15] containing [1] requests]
org.elasticsearch.index.IndexNotFoundException: no such index
at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:139) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.primary(TransportReplicationAction.java:686) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:621) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:771) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:338) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:186) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:748) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:920) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2016-12-15T18:40:15,510][WARN ][o.e.a.b.TransportShardBulkAction] [tangela1.ugent.be] unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest to [logstash-2016.12.15] containing [3] requests]
org.elasticsearch.index.IndexNotFoundException: no such index
at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:139) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.primary(TransportReplicationAction.java:686) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:621) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:771) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:338) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:186) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:748) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:920) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

[2016-12-15T18:42:08,238][WARN ][o.e.a.b.TransportShardBulkAction] [tangela1.ugent.be] [[logstash-2016.12.15][2]] failed to perform indices:data/write/bulk[s] on replica [logstash-2016.12.15][2], node[dgM6WtvtS1eq3m0PQ-NF_A], [R], s[START
ED], a[id=HgZchgrHQ2mq6FVIp0iU2w]
org.elasticsearch.transport.RemoteTransportException: [tangela2.ugent.be][192.168.24.2:9300][indices:data/write/bulk[s][r]]
Caused by: org.elasticsearch.index.engine.IndexFailedEngineException: Index failed for [%{@type}#AVkDk54hPIA-E0j0GfwZ]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:418) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnReplica(TransportIndexAction.java:166) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:454) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:73) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:81) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:48) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:465) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:438) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationLock(IndexShard.java:1671) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:538) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:422) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:412) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/2/index/_r.dii")))
at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:282) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:96) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4893) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:516) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:480) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:384) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:463) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:520) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:409) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnReplica(TransportIndexAction.java:166) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:454) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onReplicaShard(TransportShardBulkAction.java:73) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:81) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnReplica(TransportWriteAction.java:48) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:465) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:438) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationLock(IndexShard.java:1671) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:538) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:422) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:412) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
[2016-12-15T18:58:28,076][WARN ][o.e.i.e.Engine ] [tangela1.ugent.be] [logstash-2016.12.15][4] failed engine [refresh failed]
java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/4/index/_80.cfe")
at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:108) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:69) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:95) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:74) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:689) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:831) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:742) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Suppressed: org.apache.lucene.index.CorruptIndexException: checksum status indeterminate: remaining=0, please run checkindex for more details (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/4/index/_80.cfe")))
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:451) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:126) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:69) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:95) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:74) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:689) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:831) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:742) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2016-12-15T18:58:28,082][WARN ][o.e.i.IndexService ] [tangela1.ugent.be] [logstash-2016.12.15] failed to run task refresh - suppressing re-occurring exceptions unless the exception changes
org.elasticsearch.index.engine.RefreshFailedEngineException: Refresh failed
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:658) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:689) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:831) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:742) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/4/index/_80.cfe")
at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:108) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:69) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:95) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:74) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) ~[elasticsearch-5.1.1.jar:5.1.1]
... 9 more
Suppressed: org.apache.lucene.index.CorruptIndexException: checksum status indeterminate: remaining=0, please run checkindex for more details (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/4/index/_80.cfe")))
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:451) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:126) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:69) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:95) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:74) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:689) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:831) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:742) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2016-12-15T18:58:28,083][WARN ][o.e.i.c.IndicesClusterStateService] [tangela1.ugent.be] [[logstash-2016.12.15][4]] marking and sending shard failed due to [shard failure, reason [refresh failed]]
java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/4/index/_80.cfe")
at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:108) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:69) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:95) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:74) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:689) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:831) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:742) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

[2016-12-15T19:03:28,253][WARN ][o.e.i.e.Engine ] [tangela1.ugent.be] [logstash-2016.12.15][4] failed engine [refresh failed]
org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/4/index/_24_Lucene54_0.dvm")))
at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:282) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:96) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4893) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:516) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:480) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:438) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:689) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:831) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:742) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2016-12-15T19:03:28,261][WARN ][o.e.i.c.IndicesClusterStateService] [tangela1.ugent.be] [[logstash-2016.12.15][4]] marking and sending shard failed due to [shard failure, reason [refresh failed]]
org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/elasticsearch/tangela/nodes/0/indices/zXNSWURiTkmOVEGCCmXTvQ/4/index/_24_Lucene54_0.dvm")))
at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:282) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:96) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4893) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:516) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:480) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:438) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:646) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:629) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:689) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService.access$400(IndexService.java:93) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:831) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:742) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
~~~~

[clintongormley]

Hi @itkovian

From the logs it looks like you have corrupt indices, and I'd be taking a careful look at your disks to make sure you're not going to lose more data. From your comments you seem to think this is a bug instead of corruption. Why is that?

[itkovian]

I'm not seeing anything /var/log/messages that's indicative of failing disks. smartctl also says everything is fine. We've also not seen this occur until we moved to CentOS 7.2 (from SL6). As long as both nodes are up, things seem to stay green and there's nothing in the elasticsearch logs. Once a node goes down (as was the case during yesterday's update), we see the above or similar.

One thing that came to mind is the following. Could this be caused by a split-brain event, as we only have two nodes and (fingers crossed) the event seems to occur when one of them has been down.

Any pointers to chase down the cause are much appreciated.

[itkovian]

After using the pointers from http://www.jillesvangurp.com/2015/02/18/elasticsearch-failed-shard-recovery/, we seem to be green again. I'm going to await the state after tomorrow's logstash index is created before closing this.

[itkovian]

It seems that when a new lucene block was started (?), the corruption was occurring again.

[clintongormley]

@itkovian everything is pointing at your hardware.

[roffe]

Strange, getting the same problems but only when i store the data on a glusterfs backed storage.
If i use node local storage i don't get these errors.

[colings86]

No further feedback. @itkovian if you are still seeing this issue please reopen this issue and provide any information you can.

[drekle]

I am also seeing this using the helm charts elasticsearch and fluentd-elasticsearch in a kubernetes cluster where the default storage class is gluster.

[pblaas]

Same as Drekle. I currently experienced this also with fluentd to es which is stored onto GlusterFS.
I also noticed this with other then fluentd workloads to es on gluster volumes.
At a random time its corrupts the indices.

[DaveCTurner]

The first few Google hits for elasticsearch glusterfs lead to some serious-looking open GlusterFS issues:

https://bugzilla.redhat.com/show_bug.cgi?id=1390050
https://bugzilla.redhat.com/show_bug.cgi?id=1379568
https://bugzilla.redhat.com/show_bug.cgi?id=1552257

There may be more. It seems likely that corruption on GlusterFS-backed clusters is due to issues outside of Elasticsearch.