Logstash: Behaviour of ILM in particular when manage_template =>false

[OrangeDog]

I'm managing templates externally and building index names dynamically.
I'd really like logstash to manage ILM for me, but currently (6.7.0) this isn't possible.

First issue is that ilm_policy and ilm_rollover_alias are not dynamic. This can be worked-around with conditionals but that will not scale very well beyond a couple of field permutations.

Second issue is that logstash expects them to be set in a template. This would then require a separate template for every permutation, which again won't scale, and for this to be synchronised with the ilm_rollover_alias or logstash will not create a correctly-named index.

Instead (especially if manage_template => false) the index.lifecycle properties should be set on the index itself.

[OrangeDog]

I realise the elasticsearch documentation says

Its [sic] recommended not to use the create index API with a policy that defines a rollover action.

Perhaps it might be feasible to have logstash manage templates that only specify index.lifecycle.*, magically creating as required for new computed values of ilm_rollover_alias...

[juusom]

Similar issue here. We have dynamically named indices and templates are managed externally.

In our case, Logstash should not need to not know anything about the policy but just create the initial index and alias dynamically for new indices, while the default template (not managed by logstash) should define the policy.

Having to change logstash configs or do any other manual actions for each new index is not really an option.

[OrangeDog]

I'll have to go check, but I think that instead of using rollover you can have logstash dynamically create index names with the date in them, have the template apply a policy, and the warm/cold/delete phases should all take effect?

[juusom]

That's true, but I'd like to use rollover too.