Fluent-bit: [DOC] Elasticsearch index alias

[BentCoder]

Hi,

Not a bug!

Is there a way of using index aliases so that we can benefit from ES rollover API? I couldn't see anything in the doc unless I missed. I use ES (7.1), Fluent-bit (1.0.4) and Kibana (7.1).

Thanks

[SERVICE]
    Flush     5
    Daemon    Off
    Log_Level debug

[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

[OUTPUT]
    Name  es
    Match hello_*
    Host  elasticsearch
    Port  9200
    Index hello  # I like this to be alias instead
    Type  logs
    Include_Tag_Key On
    Tag_Key tag

[epcim]

I considered to use in the past, but then I realised that alias can point to single index (that do roll over). Basically I use fluentd to send to ES, it creates index per application (ie recognized from tag). The conclusion is that index alias mgmt was bit overdue. It can certainly be handy if you write to one index.

Yes, if fluentbit/d would support creation of alias with the roll over setup, per app, that would be superb. As well as more features related to index template.

On the other hand there are other means for index rotation as well as currator jobs can do rest. The biggest benefit of roll over api I see in evenly spread size (document count,..) of indexes.

[BentCoder]

If it was an application that depends on an index then an alias would be a "must have" feature for the sake of "zero down-time" ability. However, since logging/monitoring is mainly for debugging purposes I wouldn't mind a bit of down-time at all so hence reason not utilising the aliases is not a blocker to me. It would be nice if there was a mention in the doc saying that ES index aliases are not supported - at least for the given fluent-bit version etc.

[andye2004]

Hi @BentCoder, I'm wondering why you closed this issue?

I was searching issues for any mention of this as it is something that we would find particularly valuable and seems like a perfectly valid feature request to me.

The main benefits I see are the ability to use the in-built index lifecycle management (ILM) policies to ensure indices are rolled over at pre-defined intervals and are then deleted, again after pre-defined intervals. These are features that are especially useful when using ES as a centralised logging store.

[BentCoder]

I went back to ELK stack because the DOC is really poor and resources/examples are minimal when you need something. Feel free to open up a new question please because I went back to ELK.

[JanKowalik]

So what is the answer? Is it possible to use index aliases and elasticsearch Index Lifecycle Management (ILM) with Fluentbit?

Will it work if I provide an alias in the fluentbit index configuration field? The alias would point to a pre-created index with ILM linked to it. Or would it cause issues, as the precreated index would not have mappings defined.

[andye2004]

@JanKowalik you might want to have a look @ #1670