Elasticsearch: ILM - Index name picked future date in index creation

[itskannanraj]

Elasticsearch Version : 7.9.0

Index name took the future date while creating index and ILM rollover stopped working. we verified system date, system date was correct during index creation.

we could confirm by seeing the Elasticsearch log that system date is 2020-12-27 when index is created but index name created with future date apache-prod-ouse-es2-1-2021.12.27-000006

elasticsearch.log
[2020-12-27T12:38:39,895][INFO ][o.e.c.m.MetadataCreateIndexService] [prod-ouse-es2-1-kpr3] [apache-prod-ouse-es2-1-2021.12.27-000006] creating index, cause [rollover_index], templates [apache_template], shards [10]/[1]

rollover is set to "max_size" : "400gb" or "max_age" : "1d" but rollover stopped working after 2021.12.27, as you could see index size is reached to 4.7TB

https://hostname:9200/_cat/indices?v| grep apache
green open apache-prod-ouse-es2-1-2021.12.27-000006 10 1 1966117106 0 4.7tb 2.3tb
green open apache-prod-ouse-es2-1-2020.12.25-000004 10 1 83445554 0 196.2gb 98.1gb
green open apache-prod-ouse-es2-1-2020.12.26-000005 10 1 135030671 0 318.1gb 159.1gb
green open apache-prod-ouse-es2-1-2020.12.22-000001 10 1 68143215 0 158.3gb 79.1gb
green open apache-prod-ouse-es2-1-2020.12.23-000002 10 1 98142259 0 230.9gb 115.4gb
green open apache-prod-ouse-es2-1-2020.12.24-000003 10 1 88462871 0 205.4gb 102.7gb

we use parse_origination_date index settings, to have yyyy-mm-dd in index name.

Index settings:

{
"apache-prod-ouse-es2-1-2021.12.27-000006" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "ilm_apache",
"parse_origination_date" : "true",
"rollover_alias" : "ilm_apache",
"origination_date" : "1640563200000"
},
"routing" : {
"allocation" : {
"require" : {
"data" : "hot"
}
}
},
"mapping" : {
"total_fields" : {
"limit" : "5000"
}
},
"refresh_interval" : "5s",
"number_of_shards" : "10",
"provided_name" : "",
"creation_date" : "1609072719887",
"unassigned" : {
"node_left" : {
"delayed_timeout" : "5m"
}
},
"number_of_replicas" : "1",
"uuid" : "qQSYSCuiTzKI0_wF3L-Uow",
"version" : {
"created" : "7090099"
}
}
}
}
}

ILM Policy:

{
"ilm_apache" : {
"version" : 3,
"modified_date" : "2020-12-22T16:58:30.812Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "400gb",
"max_age" : "1d"
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : {
"min_age" : "30d",
"actions" : { }
},
"warm" : {
"min_age" : "15d",
"actions" : {
"allocate" : {
"include" : { },
"exclude" : { },
"require" : {
"data" : "warm"
}
},
"set_priority" : {
"priority" : 50
}
}
}
}
}
}
}

Template:

{
"apache_template" : {
"order" : 1,
"version" : 60002,
"index_patterns" : [
"apache-prod-*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "ilm_apache",
"parse_origination_date" : "true",
"rollover_alias" : "ilm_apache"
},
"routing" : {
"allocation" : {
"require" : {
"data" : "hot"
}
}
},
"mapping" : {
"total_fields" : {
"limit" : "5000"
}
},
"refresh_interval" : "5s",
"number_of_shards" : "10",
"unassigned" : {
"node_left" : {
"delayed_timeout" : "5m"
}
},
"number_of_replicas" : "1"
}
},
"mappings" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"mapping" : {
"norms" : false,
"type" : "text"
},
"match_mapping_type" : "string"
}
},
{
"string_fields" : {
"match_pattern" : "regex",
"unmatch" : "^(ot|cs|acs)$",
"mapping" : {
"norms" : false,
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"match_mapping_type" : "string"
}
},
{
"longs_to_integer" : {
"match_pattern" : "regex",
"mapping" : {
"type" : "integer"
},
"match" : "^(ot|cs|acs)$"
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
},
"@version" : {
"type" : "keyword"
}
}
},
"aliases" : { }
}
}

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[danhermann]

It looks like the likely cause of that is because your date math expression uses upper case YYYY instead of lowercase yyyy for the year.

[itskannanraj]

@danhermann ILM was working fine and it causes issue on 2020-12-27.

[adinarayanagudla]

@danhermann

Me and @itskannanraj works in same team, Could you please shed some light on why YYYY ( capital ) causes future date?

[danhermann]

@adinarayanagudla, the uppercase YYYY is "year-of-week" in Java rather than just year. You can see more info on the distinction between the two in the Java docs (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) and this article (https://www.alexcorrigan.co.uk/blog/20191231-0644-javas-week-based-year-datetimeformatter/).

[adinarayanagudla]

Thank you @danhermann for the explanation. we have updated our year string to yyyy.

[danhermann]

Glad to help. I'm going to close this as not an issue. We've opened a separate issue (https://github.com/elastic/elasticsearch/issues/67014) around disallowing or at least warning about the YYYY date pattern since it's almost year rather than "year of week" that people want.