Elasticsearch: DateFormat incompatibility between 6.x and 7.x

[ywelsch]

QA testing of ML functionality in an upgrade from 6.7.1 to 7.0.0-rc2 revealed an issue where moving shards from 6.7.1 to 7.0.0-rc2 nodes were getting stuck in the translog phase. The reason for this was that the 7.0.0-rc2 would not be able to parse the documents from the 6.7.1 node, internally throwing a MapperParsingException of the form failed to parse field [@timestamp] of type [date] in document with id 'AVr3KgpQqy1yzebk-FVk'. This would in turn make the peer recovery hang, as the node with the MapperParsingException would indefinitely wait for a cluster state update with an updated mapping. This issue does not only present a problem for rolling upgrades, but also does not allow any peer recovery of that index in a pure 7.x cluster unless the translog with the corresponding documents is completely purged.

The problematic dateformat field in the mappings of the index looked as follows:

  "@timestamp" : {
    "type" : "date",
    "format" : "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
  }, ...

The documents were of the form

{"@timestamp":"2013-01-25T20:16:55.000Z", ...}

The problem seems to be an incompatibility between 6.x and 7.x when it comes to date format parsing, possibly related to https://github.com/elastic/elasticsearch/issues/27330

The following test illustrates the problem on 7.x:

    // put this in DateFieldMapperTests
    public void testFormat() throws Exception {
        String mapping = Strings.toString(XContentFactory.jsonBuilder().startObject().startObject("type")
            .startObject("properties").startObject("@timestamp").field("type", "date")
            .field("format", "yyyy-MM-dd'T'HH:mm:ss.SSSZ").endObject().endObject()
            .endObject().endObject());
        DocumentMapper mapper = parser.parse("type", new CompressedXContent(mapping));
        ParsedDocument doc = mapper.parse(new SourceToParse("test", "type", "1", BytesReference
            .bytes(XContentFactory.jsonBuilder()
                .startObject()
                .field("@timestamp", "2013-01-25T20:16:55.000Z")
                .endObject()),
            XContentType.JSON));
        assertNotNull(doc);
    }

and the corresponding test on 6.x passes:

    public void testFormat() throws Exception {
        String mapping = Strings.toString(XContentFactory.jsonBuilder().startObject().startObject("type")
            .startObject("properties").startObject("@timestamp").field("type", "date")
            .field("format", "yyyy-MM-dd'T'HH:mm:ss.SSSZ").endObject().endObject()
            .endObject().endObject());
        DocumentMapper mapper = parser.parse("type", new CompressedXContent(mapping));
        ParsedDocument doc = mapper.parse(SourceToParse.source("test", "type", "1", BytesReference
                .bytes(XContentFactory.jsonBuilder()
                    .startObject()
                    .field("@timestamp", "2013-01-25T20:16:55.000Z")
                    .endObject()),
            XContentType.JSON));
        assertNotNull(doc);
    }

Involved in this investigation: @pheyos and @dolaru (ML QA), @joegallo (Cloud, where rolling upgrades got stuck), @dnhatn (ES debugging + tests)

[elasticmachine]

Pinging @elastic/es-core-infra

[pgomulka]

Should the pattern before the upgrade from 6.x to 7.x be prefixed with 8 and in java-time format?
if the pattern in a test in 6.x is prefixed with 8 it will fail the same (as it would use JavaDateFormatter)

how about patterns like these?
8yyyy-MM-dd'T'HH:mm:ss.SSSXX in 6.x and yyyy-MM-dd'T'HH:mm:ss.SSSXX in 7.x?

[elasticmachine]

Pinging @elastic/es-search

[jaredmcqueen]

I have discovered this too.

in 6.x this worked:
2019-04-09T00:34:10+00:00
yyyy-MM-dd'T'HH:mm:ssZZ

in 7.x it fails - but the work around I found is changing the ZZ to zz

So in 7.x this works:
2019-04-09T00:34:10+00:00
yyyy-MM-dd'T'HH:mm:sszz

not sure why the two lowercase z's fix it.

[pgomulka]

@jaredmcqueen on what scenario was this failing for you? Was this the ML upgrade too?

you can use 'zz' pattern for this example, this will use java's OffsetIdPrinterParser with (+HH:MM:ss and Z as default.
The 'z' is intended for zone names (like GMT, PST, UTC etc. ) but can fall back to parsing with offset as well (like in your example)

You can use 'XXX' pattern if you want to allow ONLY +HH:MM:ss and Z

[jaredmcqueen]

The scenario it was failing was general index mappings. I was trying to index ISO 8601 datetimes.

[ITzhangqiang]

I have discovered this too.

in 6.x this worked:
2019-04-09T00:34:10+00:00
yyyy-MM-dd'T'HH:mm:ssZZ

in 7.x it fails - but the work around I found is changing the ZZ to zz

So in 7.x this works:
2019-04-09T00:34:10+00:00
yyyy-MM-dd'T'HH:mm:sszz

not sure why the two lowercase z's fix it.

NB，bro

[pgomulka]

@ywelsch are you ok if I close this issue? This problem occurred because the incorrect pattern was used - missing 8 prefix and Z instead of X
as per my comment here https://github.com/elastic/elasticsearch/issues/40913#issuecomment-480487107
and it seems like it is being linked to other time related issues, different then the problem described in the first comment.
I would personally prefer if there is a new date related problem, a new issue to be opened.

[ywelsch]

Yes, you can close the issue if you think this is resolved.