Elasticsearch: [bug report]LockObtainFailedException throws under presure

[makeyang]

Elasticsearch version:
2.1
Plugins installed: []
delete-by-query
elasticsearch-analysis-ik
repository-hdfs
JVM version:
8u60
OS version:
CentOS release 6.6 (Final)
Description of the problem including expected versus actual behavior:
one of the data node keep throw below exception:
[2016-10-12 11:34:04,769][WARN ][cluster.action.shard ] [XXXX] [indexName][2] received shard failed for [indexName][2], node[rckOYj-DT42QNoH9CCEBJQ], relocating [v2zayugFQnuMiGu-hS1vXg], [R], v[7091], s[INI
TIALIZING], a[id=bkpcEq2qTXaPEKHl9tOunQ, rId=xeJJijQCRyaJPcSgQa7eGg], expected_shard_size[22462872851], indexUUID [sOKz0tW9Sw-u137Swoevsw], message [failed to create shard], failure [ElasticsearchException[failed to create shard]; nested: LockObtainF
ailedException[Can't lock shard [indexName][2], timed out after 5000ms]; ]
[indexName][[indexName][2]] ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [indexName][2], timed out after 5000ms];
at org.elasticsearch.index.IndexService.createShard(IndexService.java:389)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:650)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:550)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:179)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:494)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.LockObtainFailedException: Can't lock shard [indexName][2], timed out after 5000ms
at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:565)
at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:493)
at org.elasticsearch.index.IndexService.createShard(IndexService.java:307)
... 9 more
Steps to reproduce:(not very presious, I haven't reproduced it yet)

1. give cluster a lot presure and one node out of cluster
2. then remove presure and after a while, the node come back and try to recover some shard, it keeps throw below exception

[bleskes]

this happens when same background process is still ongoing (or just failed to finish properly) and still holds the shard lock. An example of this is a recovery process which needs access to the shard folder to copy files.

Do you have index.shard.check_on_startup set by any chance?

[makeyang]

@bleskes no, we don't set that. should it better to set it?

[bleskes]

@bleskes no, we don't set that. should it better to set it?

Oh no - this is one of those things that I know that can take long in the background. Since 2.1 is quite old - do you have the chance to upgrade to 2.4 and try to reproduce?

[makeyang]

I'll try to reproduce in 2.4 in test env.

[setaou]

We are experiencing the same problem with ES 2.4.1, java 8u101, Ubuntu 14.04.

It has happened two times, and each time was triggered by starting a full snapshot while one index was under heavy indexing load (3-4k docs/s). About 10 minutes after the beginning of the snapshot, some shards of this index begin to throw lots of LockObtainFailedException, and the situation finally gets back to normal about one hour later. Meanwhile, about 2-3k LockObtainFailedException have been thrown.

I hope a solution will be found because currently our only option is to disable our daily snapshot while we are doing heavy indexations.

[abeyad]

The likely reason why the LockObtainFailedException keeps occurring is if we have a scenario where the node holding the primary is under heavy load so is slow to respond and leaves the cluster, while a snapshot is taking place. The snapshot holds a lock on the primary shard on the over-loaded node. When the master node realizes that over-loaded node is not responding, it removes it from the cluster, promotes a replica copy of the shard to primary, and cancels the snapshot. When the over-loaded node rejoins the cluster, the master node assigns it to hold a replica copy of the shard. When the node attempts to initialize the shard and recover from the primary, it encounters a LockObtainFailedException because the canceled snapshot process still holds a lock on the shard. The shard lock isn't released until the snapshot actually completes. We are looking into an appropriate fix for this.

[makeyang]

@abeyad thanks for u guys' hard work