Elasticsearch: date_time format change in elasticsearch from version 6.5 to 7.2

[Tomassino]

Elasticsearch version: 6.5.4 and 7.2.0

Plugins installed: []

JVM version (java -version): openjdk version "1.8.0_212"

OS version (uname -a if on a Unix-like system): Linux tommy 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

I noticed that some dates that were handled by elasticsearch version 6.5 are rejected by elasticsearch 7.2, see below for an example (original forum post: https://discuss.elastic.co/t/date-time-format-change-in-elasticsearch-from-version-6-5-to-7-2/188546 )

Steps to reproduce:

Elasticseach 7.2.0

PUT localhost:9200/my_index
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "date_time"
      }
    }
  }
}

This document is successfully indexed:

POST  localhost:9200/my_index/_doc
{
  "date": "99999-12-31T23:59:59.999+0000"
}

While this one is rejected:

POST  localhost:9200/my_index/_doc
{
  "date": "100000-01-01T00:00:00.000+0000"
}

The error is:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [date] of type [date] in document with id 'vp5Ms2sBNm1nVrvtzmYN'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [date] of type [date] in document with id 'vp5Ms2sBNm1nVrvtzmYN'",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "failed to parse date field [100000-01-01T00:00:00.000+0000] with format [date_time]",
      "caused_by" : {
        "type" : "date_time_parse_exception",
        "reason" : "Failed to parse with all enclosed parsers"
      }
    }
  },
  "status" : 400
}

Elasticseach 6.5.4

PUT localhost:9200/my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "date": {
          "type":   "date",
          "format": "date_time"
        }
      }
    }
  }
}

Both documents are successfully indexed:

POST  localhost:9200/my_index/_doc
{
  "date": "99999-12-31T23:59:59.999+0000"
}

POST  localhost:9200/my_index/_doc
{
  "date": "100000-01-01T00:00:00.000+0000"
}

[elasticmachine]

Pinging @elastic/es-core-infra

[arthurvb]

Same problems here, which actually lead to a Datetime from the Nest-client not being indexed:

"error":{"type":"mapper_parsing_exception",
"reason":"failed to parse field [files.lastAccessed] of type [date] in document with id '12345'",
"caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2018-11-14T15:35:52.18] with format [strict_date_optional_time||epoch_millis]",
"caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Failed to parse with all enclosed parsers"}},

I suspect it is because the millis part is not padded with zero's.

[pgomulka]

@Tomassino you are right, this is because since ESv7 we limited year for date_time to max 5digits. This should probably be changed to 10. What is your view on this @spinscale ?
previous implementation using joda year was 4-9digits

[pgomulka]

@arthurvb I am surprised to see your use case failing. strict_date_optional_time can accept from 1-9 digits on fraction of a second part.
Can you provide more details in a new bug?

[arthurvb]

@pgomulka Tnx for looking in to this, I will first check on the latest version, and if this is still the case file the report (current version is 7.0.0)

[arthurvb]

Confirming that this is fixed in version 7.2

[pgomulka]

@Tomassino we discussed this, and for the time being you should best use a custom date format for dates which are having more than 4 digits for a year.
as per documentation

date_time or strict_date_time
A formatter that combines a full date and time, separated by a T: yyyy-MM-dd'T'HH:mm:ss.SSSZZ.

https://www.elastic.co/guide/en/elasticsearch/reference/6.5/mapping-date-format.html

I will keep this issue on my list, and might revisit this in the future.

[Tomassino]

Ok, thanks for your help :-)