Beats: [Filebeat] Kubernetes metadata fields not expanded in rollover_alias and policy_name

[TomaszKlosinski]

Describe the enhancement:

As I mentioned in this comment, it's possible to use index per kubernetes namespace with old (daily) index format:

output.elasticsearch.index: "%{[kubernetes.namespace]:filebeat}-%{+yyyy.MM.dd}"

But it's not possible with the new format supported by ILM. Therefore, I can use either one feature or second. I can't use both logs index per kubernetes namespace and ILM. It would be a great enhancement for us, if we could use both.

Currently, my workaround is to use daily indices, but implement separately ILM with API calls - this however has a limit, as rollover action in ILM does not support indices with daily format. I can use ILM only to delete old indices. Alternatively, I could drop ILM and use curator, but I'd prefer to use the newer solution.

Describe a specific use case for the enhancement or feature:

I'd like to be able to use an index and ILM per kubernetes namespace, like this:

    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: "%{[kubernetes.namespace]:filebeat}" 
    setup.ilm.pattern: "{now/d}-000001"
    setup.ilm.overwrite: true
    setup.ilm.policy_name: "%{[kubernetes.namespace]:filebeat}" 

[TomaszKlosinski]

One update: I've realised that it's not about the index name format I can use e.g.:

output.elasticsearch.index: "%{[kubernetes.namespace]:filebeat}-000001"

There are two problems here. One is that the fileds are not expanded, but the other is that ILM configuration is done only once during initialization of filebeat, so even with the field expansition fix, I still can't have a more dynamic log forwarding based on the kubernetes fields (for each log message coming from different pods/containers). I think the solution would be to first check whether a given ILM configuration exists for a given event (log message) and then send it there, otherwise initialize new ILM configuration and then send it.

[zorgzerg]

+1

[andresrc]

Unfortunately, we do not plan to support this directly in the near future, but to solve it in a somewhat different way. Closing for now. Thanks!