Elasticsearch: [DOCS] Provide an explanation of what dangling indices are and how they can occur

[geekpete]

The elasticsearch docs make reference to dangling indices:

• https://www.elastic.co/search?q=dangling&section=Learn%2FDocs%2FElasticsearch%2FReference%2F5.5

but there's no explanation of what they are or how they come about.
It would be good to have a glossary of some type to refer to for various terminology like this that could be expanded over time.

It would be good to link/reference Index Tombstones when documenting dangling indices too.

[DongwooGeum]

You can get information about "Dangling indicies" by typing "Dangling indicies" in the search line, then you can find the link under the search line. I reference the link

https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-gateway-local.html#_dangling_indices

[geekpete]

Looks like we had it documented in 1.7 docs but need to ensure it's also in the latest docs after we've reorganised them. I'll see if I can fix this later...

Thanks!

[zakkg3]

Dangling indices
When a node joins the cluster, any shards stored in its local data directory directory which do not already exist in the cluster will be imported into the cluster. This functionality is intended as a best effort to help users who lose all master nodes. If a new master node is started which is unaware of the other indices in the cluster, adding the old nodes will cause the old indices to be imported, instead of being deleted.
https://www.elastic.co/guide/en/elasticsearch/reference/master/_dangling_indices.html

[jbarwick]

I have a situation where deleted indexes are not deleted from all nodes. As such, when ES is restarted, this Dangling Shard is imported as a Dangling Index and is thus UNASSIGNED and causes the cluster to enter "red" status and alarm bells start going off and things stop working.

I run filebeat/metricbeat and curator. Curator deletes all "metricbeat" indexes older than 30 days. It runs every night. My cluster is green for 'days' (not months..and this is a problem).

Then being already July, ES may restart for some reason (update of config or some other cycle) and then I get metricbeat indexes UNASSIGNED from january.

I then run curator again, deletes these indexes, and the cluster goes green.

How can I run a program (which I am happy to do on all nodes) that would delete all index shards from the disk that don't have an index in ES? Any idea how to go about this?

[geekpete]

@jbarwick Your question is better posted in the Elastic forums: https://discuss.elastic.co/ but do mention the link in this ticket once you have posted so others might follow it and find any further info/answers for similar questions.