Pfelk: ILM compatible index name

[kingtat]

Is your feature request related to a problem? Please describe.
A possible enhancement.

Describe the solution you'd like
Index naming and index templates that support using ILM in elastic/kibana.

Additional context
It would be good if pfELK provided alternate (or guide on how to implement) index names and index templates (and other requirements I am not sure of) that lend themselves to using Elastic's ILM.

The current setup creates new daily indexes which don't work well with ILM. I used curator to manage indexes in the past, but it would be good if pfELK generated indices and used templates conducive to ILM.
Just an idea. Thanks.

[a3ilson]

@kingtat - thanks for the inquire.

Setting up an ILM is fairly straight forward: (https://www.elastic.co/guide/en/kibana/current/creating-index-lifecycle-policies.html)

You'll also need x-pack configured and ILM created (https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html)

These would be dependent upon ones setup and I would simply defer to Elastic documentation for anyone looking to configure.

[kingtat]

Thanks. I did take a look prior to posting the suggestion.
I think I follow the ILM setup.
AIUI in order to have the pfelk config work with Elastic's ILM I need to:

1. Modify logstash to output a static index name. e.g. (so NO new index is created each day).
2. Potentially modify any index templates that reference pattern 'pf-*', in the event that pattern no longer applies.

Any thing else?
Comment edited

[TehMuffinMoo]

As long as you have bootstrapped/created your inital index with your alias and applied the index.lifecycle.rollover_alias setting then the only thing you should need is your first point.

For your second point, as long as your initial index is created/bootstrapped as pf-00x (scaling the qty of digits to your requirements) then this would match the existing index pattern and be compatible with ILM rollover.

[kingtat]

@TehMuffinMoo
Thanks for the clarification.

[revere521]

This appears to be resolved, no further question sin 28 days

[kingtat]

It turns out it is possible to set up ILM almost entirely within the logstash Elasticsearch output plugin.

After creating a custom lifecycle policy e.g. 'pf-ilm-policy' (in order to leave default policies untouched), modify /etc/logstash/conf.d/50-outputs.conf:

• No index name.
• Specify ILM rollover alias and policy.
• Enable template overwrite (or delete any existing pf-geoip template before restarting logstash).

output {
  elasticsearch {
    id => "pf"
    hosts => ["http://localhost:9200"]
#    index => "pf-%{+YYYY.MM.dd}"
    manage_template => true
    template => "/etc/logstash/conf.d/templates/pf-geoip-template.json"
    template_name => "pf-geoip"
    template_overwrite => true
    ilm_rollover_alias => "pf"
    ilm_policy => "pf-ilm-policy"
  }
}

It may be necessary to delete existing pf-* indices in order for an initial rollover index to be successfully created.
Restart logstash.