Fluent-bit: Index aliasing on ES output

[andye2004]

Firstly, just want say a huge thanks 👏 for all the effort you guys put into fbit and for producing what we consider an essential component of our tech stack!

Is your feature request related to a problem? Please describe.

We use ES primarily for centralised logging of on-prem k8s clusters and we are experiencing frustration, rather than problems trying to manage the over-all size of our ES DB and how to keep it manageable.

Given our primary usage, we don't really need to retain log events longer than a certain period of time and they need to remain 'hot', to use ES phraseology, only for a fairly short period of time. Given this we set Logstash_Formatto on and this effectively gives us daily rolling of indices.

However, this leaves us having to manage the ever growing list of daily created indices which isn't ideal. We are actually doing this manually, yes, we could use curator and may end up having to do this but a 'better' solution is to use the in-built ILM (see below).

Describe the solution you'd like

In newer versions of ES (6.7+) there is the index lifecycle management (ILM), which provides a way to create a policy that will allow us to rotate log indices using more granular timeframes than daily, something that is not essential but still very useful. Moore importantly, it allows us to specify a maximum retention time for indices meaning they are automatically deleted after a specified period of time. This is now the recommended way to manage index lifecycles.

The use of the ILM feature requires that indices are created with an alias such that the underlying index can be rolled over and anything writing to the alias can carry on as normal. It is for this reason that I'd like to request the addition of an aliasing feature to the ES output plugin that can be used in conjunction with the Index attribute.

Thanks, Andy.

P.S. Thought it worth adding that this feature is available when using filebeats.

[andye2004]

Hi @edsiper, I was just wondering if this was likely to make it into the 1.3 release, whenever that may be?

[JanKowalik]

This feature is something that would be very useful for us too.

Also, as mentioned in #1381 I was wondering if it could work without any changes. Would it work if I provide an alias name in the fluentbit index configuration field? The alias would point to a pre-created index with ILM linked to it. Or would it cause issues, as the precreated index would not have mappings defined?
In the above case I would not use LogStash format so fluentbit does not create index every day.

Thanks!

[JanKowalik]

Would it work if I provide an alias name in the fluentbit index configuration field? The alias would point to a pre-created index with ILM linked to it. Or would it cause issues, as the precreated index would not have mappings defined?

This does not work.
I disabled Logstash_Format so it does not create an index per day. This way it only uses fluent-bit index name.
Next, before I started Fluentbit, I created an index with an ILM policy and a fluent-bit alias. I was hoping that I can trick it to use the alias instead of creating its own index. I was not successful. Moreover Fluentbit did not send any data to Elasticsearch at all. I looked briefly into kubernetes deamonset pod logs but did not see any errors poping up there.

[JanKowalik]

Hi again :)
I have said the above to early. It does work. It collects data and rollsover, ages indices following the ILM I specified.

I still have a couple of concerns though.

1. I am not sure if it creates correct mappings if I provide it with an empty index (via an alias).
2. I still want to wait for a day at least to see if Fluentbit won't do something funny and rollover to a new index name on a day boundary (as it does when LogStash_Format is enabled)

[JanKowalik]

I am not sure if it creates correct mappings if I provide it with an empty index (via an alias).

Mappings are being deduced by elasticsearch from a document that is indexed as first for each new index. This might cause issues, if the first one produces mapping with a datetime field mapping and other subsequent ones from other sources do not match it and require plain text field mapping.

I still want to wait for a day at least to see if Fluentbit won't do something funny and rollover to a new index name on a day boundary (as it does when LogStash_Format is enabled)

All works fine here and indices rollover without issues.

[naseemkullah]

Extra points if fluent-bit es output could setup the ILM policy, how filebeat does today.

[mapdegree]

Setting a write index and alias on an index should not be rocket science. One of the reasons we are switching back to filebeat is exactly the lack of support for ILM through index aliases.

[Giaco9]

@edsiper is there any news about this issue? We have the same need. We would like to apply an ILM policy to an index created by fluent bit.

Thank you