Beats: ILM Created Indices remaining unmanaged

[andrewvc]

Tested on my mac with master ES/Beats/Kibana. Seen in the wild on these discuss posts https://discuss.elastic.co/t/graphql-errors-uptime-heartbeat/178473 and https://discuss.elastic.co/t/uptime-version-7-0-graphql-error/176661/6

Querying GET /heartbeat-8.0.0/_ilm/explain after setting up beats/kibana/heartbeat on a clean stack shows:

{
  "indices" : {
    "heartbeat-8.0.0-2019.04.25-000001" : {
      "index" : "heartbeat-8.0.0-2019.04.25-000001",
      "managed" : false
    }
  }
}

The heartbeat index does not have any of its mappings applied.

When functioning managed should be true. I suspect this is a race because it seems to work for some people and not others. It worked once for me, but not after that.

[elasticmachine]

Pinging @elastic/uptime

[andrewvc]

@urso I think the issue here is caused by users manually deleting the heartbeat index backing the heartbeat-8.0.0 alias while a heartbeat process is running. The next write to heartbeat-8.0.0 will auto-create that index which will not have mappings or any ILM policies applied.

This is not strictly a bug, things are working as intended, but the user experience is poor.

I've discussed this with @jakelandis of the ES team. One option we discussed was adding a _require_alias option to index requests, where the request would fail if the target was not an alias, allowing the software to try and recover by recreating the proper index + alias.

[andrewvc]

The workaround for this state, by the way, is to stop all heartbeats, delete the heartbeat-8.0.0 index, then restart the heartbeats.

[andrewvc]

Closing in favor of the better ES issue here: https://github.com/elastic/elasticsearch/issues/41699