Logstash: Use ISO 8601/RFC 3339 international date format

[ypid]

To be compliant with the international date format as specified by ISO 8601 and RFC 3339 I would propose to change the default index suffix from %{+YYYY.MM.dd} to %{+YYYY-MM-dd} in a coming major version release. Am I missing something? Why was YYYY.MM.dd chosen over YYYY-MM-dd?

https://xkcd.com/1179/

RFC 3339

Steps to Reproduce

Use the elasticsearch output with default configuration.

Expected

Clear international date format being used by default for daily indexes: logstash-2017-06-20.

Actual

Index: logstash-2017.06.20.

[untergeek]

Be that as it may, there's a tremendous user base which has been using the dotted notation for as long as 6 or 7 years—some possibly even longer. Changing the default behavior now would break search, index management, and many other things.

Since the setting is configurable, feel free to make your particular setup use the ISO default nomenclature. We simply cannot break usability for thousands and thousands of users by changing a default that has been there for so long. Beats also ships indices with dotted nomenclature, come to think of it. It's not going away.

With that said, I will leave this issue open so as to allow further comment, especially regarding the "why was dotted chosen over dashed?" question, which I cannot personally address.

[jordansissel]

regarding the "why was dotted chosen over dashed?" question

I think it was a typo. I introduced it here 3f56b3d8481f5e8c9d5357bddf77a81fc5e64046 (6 years ago) which changed the
index from "logstash" to "logstash-YYYY.MM.dd". Why dots vs dashes? Again, probably typo.

For history, the main point of this decision was to enable dated partitions of your data. Before 3f56b3d8481f5e8c9d5357bddf77a81fc5e64046, the index was just "logstash" which didn't scale particularly well over time. Daily index is nice because you can make index setting changes day to day and scale out a bit more -- and also easily delete whole indices.

Given the focus was to choose daily index names, the "how" to achieve this (what date format to use) seems to have not negatively impacted many users in day to day usage. If it has, I am not recalling much complaint, though my memory is flawed and may be inaccurate.

In hindsight, I agree that my choice of dots instead of dashes may be an odd one ;)

Fast-forward to today. Would we change this now? In addition to what @untergeek mentioned, Elasticsearch has a rollover api which lets you roll indexes in some pretty nice ways (by age, by size, etc). This rollover system is, in my opinion, more flexible than the current Logstash default of "one index per day".

If we had to revisit this, I'd personally rather move us towards index rollover, which would mean the default index would go back to "logstash" and we'd use the rollover API to rotate indices. As a side effect, switching to index rollover would also resolve the concern for the chosen index name date format.

[ypid]

Thanks very much for your detailed answers :) Very much appreciated. I agree that changing it now is unlikely as it would break backwards compatibility. Also nice that the problem might solve itself with other changes in the future like index rollover. Looking forward to that :) Feel free to close.

[jordansissel]

@ypid indeed! This is a good question to have asked. Thank you :)

[untergeek]

@ypid The truth is, you can use Rollover right now. While we are planning to put some form of rollover into Logstash, you don't have to wait for that to happen.

1. Create index logstash-000001 and add alias logstash to that:

PUT /logstash-000001 
{
  "aliases": {
    "logstash": {}
  }
}

1. Edit your Logstash configuration to point to this alias instead of using the default index:

output {
  elasticsearch {
    # ... config stuff here
    index => "logstash"
    # ... more config stuff here
  }
}

1. Use Curator to rollover the indices at regular intervals (e.g. cron):

actions:
  1:
    action: rollover
    description: Rollover the index associated with alias logstash
    options:
      name: logstash
      conditions:
        max_docs: 1000000

If you want to rotate based on age, use max_age: instead of max_docs:. You can even use both, but both conditions would have to be true for the rollover to happen.

One caveat: In ES versions 5.0 - 5.4.x, max_docs counts docs from all primary _and_ replica shards, effectively meaning you had to add the number of max_docs times the number of replicas: (max_docs = primary shard doc count + (replica count * primary shard doc count)). This was confusing, and users had a right to expect it to only count the primary shard's docs. Beginning in version 5.5, only the documents in the primary shard will be added.

[ypid]

Thanks @untergeek. Very nice, I will give index rollover a try when I get to it :)