Beats: Change index config in elasticsearch output

[ruflin]

In beats the elasticsearch index is currently defined as following:

output.elasticsearch.index: "metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

This combines the index name with the pattern of the daily indices. The problem that this now brings is that index template pattern and the index name with its pattern should correlate. This means for metricbeat-%{[beat.version]}-%{+yyyy.MM.dd} the template pattern metricbeat-%{[beat.version]}-* is needed. By default this works as expected but breaks if someone changes the pattern logic.

As the index name part in the config contains also the pattern part, it cannot be used for the index template logic. So far we just assume someone didn't change the config. So if someone removed for example the version part from the index name and replaced it with something else, the template will not be loaded.

To solve this problem I suggest to split up the index into two parts:

index: metricbeat-%{[beat.version]}
index_pattern: -%{+yyyy.MM.dd}

Like this the index.name part can be used in the template for setup.template.name and for setup.template.pattern also the index name + * can be used. See https://github.com/elastic/beats/pull/4769

The above change can be made in 2 ways:

Option 1: Non breaking but template magic?

We could introduce index_pattern instead of index.name and index.pattern. If not set, the index.pattern would be empty and everything would work as before. Like this we would not have a breaking change. But if someone uses a new config it would ship with the following:

index.name: metricbeat-%{[beat.version]}
index.pattern: -%{+yyyy.MM.dd}

The problem with this is the old index settigns are used, the auto generation of the template could be wrong as it will include the daily patterns. We could add some magic that detects the daily pattern and removes it but sounds like too much magic to me.

Option 2: Breaking change

We can't have index and index.name as the two will conflict (values vs object). If we introduce the namespaced option above this is a breaking change. If we do a breaking change, we could offer a script that automatically splits up pattern and index name if defaults are used. The good part about the breaking change is that users become aware that this now also has some relation to the template.

[andrewkroh]

Relates to #4724.

[monicasarbu]

I think ideally would be to configure the template and the index in one place under the setup.template and setup.index, considering that one output is enabled at a time.
For example:

setup.index.name=filebeat=%{[beat.version]}

# setup.index.pattern = setup.index.name +yyyy.MM.dd
setup.index.pattern=-%{+yyyy.MM.dd}

# equal to setup.index.name
setup.template.name=

# the template pattern equals setup.template.name + setup.template.pattern
setup.template.pattern=-*

In order to be able to remove the index configuration from the outputs, we need to do some adjustments. For example, for Logstash, the index=filbeat by default. We need to send as metadata not only filebeat but also the Beat version and date ({yyyy.MM.dd}) (what is configured in the setup.index.pattern) and concatenated it in Logstash.

[ruflin]

It's an interesting idea to remove the index pattern from elasticsearch output and move it to setup. This brings related config options closer together. Index is something specific to the ES output but at the same time it's a core to how the beats work and handle data in ES.

It would be great if having all this as meta information would also make it work through Kafka for example.

[kazizi-dev]

@monicasarbu and @ruflin , I cannot change my default Metricbeat index name for Elasticsearch. Did you guys find a fix? I appreciate if you can share it with me.

[ph]

@ruflin Is this solved by the new indexing strategy in EPM? If so can we close this issue?

[ruflin]

@ph The new indexing strategy is not going to fix this for Filebeat and Metricbeat but for the agent by not loading the template anymore from the agent side.

I'm good with closing this more with "wont fix" for MB and FB.

[ph]

Going to close this as "wont fix"