Elasticsearch: ILM support for the rollover date math to be based on custom date

[andreidan]

Currently, a rollover action configured to roll based on the index age uses the index creation date in order to calculate the index age. The creation age cannot be updated.

With https://github.com/elastic/elasticsearch/issues/42449 phase timings date math based on a custom date (the origination_date) was added.

This proposes adding a way to control the max_age date math of the rollover action (maybe the same origination_date? ) such that users who, for eg, create an index at noon under a policy with a rollover action as shown below, can rely on the fact that the rollover will occur on or around midnight

"rollover" : {
   "max_age" : "1d"
 }

[kostasb]

In the meantime, until this enhancement is implemented, it would be good to clarify in the docs of Rollover's max_age that it takes into account the creation_date, no matter if parse_origination_date is enabled on the index.

[meowtochondria]

Oh man! I can't say how important is to add kostasb's comment to documentation. I've been struggling with this for a couple of months now, started poking around code couple of days back, and finally landed here.

Considering andreidan has implemented this feature, I hope you can answer the following question (other avenues have not yielded an answer): when parse_origination_date is enabled and hot phase has max_age as one of the conditions, will the rollover action create an index with date of rollover? Specifically for this issue, if new date is not written into index name, wouldn't it cause issues when rollover actually uses origination_date for calculating age on rolled over indices?

I am losing some data right now and i think it is because of not having updated date in the index name. If I am misunderstanding something, I'd love some clarity on it as this is a show-stopper for v7 deployment. I can provide more details, however, I do not want to spam this issue if turns out to be not relevant.

[BBQigniter]

so max_age is totally ignored :|

@meowtochondria it seems the rollovered index doesn't have a new date. struggling currently with the same issue and it's totally unlogic that it is not working as expected. so ILM stuff seems not to work with manual setup like I did with logstash described in this discussion: https://discuss.elastic.co/t/logstash-and-ilm-rollover-not-working/253530/3

[andreidan]

@meowtochondria Apologies for the late reply. I've opened a PR to clarify the documentation on max_age.

when parse_origination_date is enabled and hot phase has max_age as one of the conditions, will the rollover action create an index with date of rollover? Specifically for this issue, if new date is not written into index name

If you use date math in the managed index name, the rollover index (the new index) will have in the name the date according to the date math pattern
For eg.

// the below is the equivalent of /<numbers-{now/d}-1>
PUT /%3Cnumbers-%7Bnow%2Fd%7D-1%3E

Creating a managed index like the above will today (the 30th of October) create index numbers-2020.10.30-1 which will have the origination date parsed and set to 1604016000000 (assuming the parse_origination_date configuration for new indices is in a template that matches this index). If on the 31st of October this index is bound to be rolled over (by any matching conditions) the rollover index created will be numbers-2020.10.31-000002 and this index will have the origination date parsed and set to 1604102400000.
The parse origination date configuration does not configure the rolling of the date in the index name to a new date when the index is rolled over, but only to configure the origination date setting for the managed index (it's the date math configuration in the index name that instructs elasticsearch to update the name of the rollover index based on the date math expression)

wouldn't it cause issues when rollover actually uses origination_date for calculating age on rolled over indices

The origination_date is only used for the phase transition date math.

@BBQigniter ILM validates the age of the indices when it runs. You can validate if ILM is running or not using _ilm/status.
ILM checks the indices every configured poll interval. It defaults to 10 minutes and you can check and change it using the indices.lifecycle.poll_interval setting

[BBQigniter]

@andreidan max_age is used - just a typo from my side. i meant age as seen if you look at "_ilm/explain"

yes, ilm is running and shows me "RUNNING" and as explained in the forum post I made the rollover now kind of works, but the new date is not set. I today got a newly rolled over index with name "compliance_logstash_syslog-2020.10.28-000003"

my index template is configured with pattern compliance_logstash_syslog* is this maybe the problem? should it be
compliance_logstash_syslog-*-* instead?

[andreidan]

@BBQigniter I replied with more details on your Elasticsearch forum post. This allows us to use GitHub for verified bug reports, feature requests, and pull requests.
Thank you very much for your interest in Elasticsearch.