Beats: Timestamp processor fails to parse date correctly

[fkelbert]

The Filebeat timestamp processor in version 7.5.0 fails to parse dates correctly. Only the third of the three dates is parsed correctly (though even for this one, milliseconds are wrong).

Input file:

13.06.19 15:04:05:001
03.12.19 17:47:38:402
03.04.19 15:04:05:999

filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.txt
processors:
  - timestamp:
      field: message
      layouts:
        - '02.01.06 15:04:05:999'
output.console.pretty: true

Output:

{
  "@timestamp": "2019-12-09T15:28:55.973Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "7cbeaeaf-4378-4703-8509-e43399081c16",
    "hostname": "fk",
    "id": "6525664c-da2b-4b49-a676-87e1e71d5591",
    "version": "7.5.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/tmp/test.txt"
    }
  },
  "message": "13.06.19 15:04:05:001",
  "input": {
    "type": "log"
  }
}
{
  "@timestamp": "2019-12-09T15:28:58.915Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "agent": {
    "version": "7.5.0",
    "type": "filebeat",
    "ephemeral_id": "7cbeaeaf-4378-4703-8509-e43399081c16",
    "hostname": "fk",
    "id": "6525664c-da2b-4b49-a676-87e1e71d5591"
  },
  "log": {
    "offset": 22,
    "file": {
      "path": "/tmp/test.txt"
    }
  },
  "message": "03.12.19 17:47:38:402"
}
{
  "@timestamp": "2019-04-03T15:04:05.000Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "log": {
    "file": {
      "path": "/tmp/test.txt"
    },
    "offset": 44
  },
  "message": "03.04.19 15:04:05:999",
  "input": {
    "type": "log"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "7cbeaeaf-4378-4703-8509-e43399081c16",
    "hostname": "fk",
    "id": "6525664c-da2b-4b49-a676-87e1e71d5591",
    "version": "7.5.0"
  },
  "ecs": {
    "version": "1.1.0"
  }
}

[sayden]

TLDR: Go doesn't accept anything apart of a dot . to parse milliseconds in date/time.

I have been doing some research and, unfortunately, this is a known issue in the format parser of Go language. https://github.com/golang/go/issues/6189 In this issue they talk about commas but the situation is the same regarding colon. It's very inconvenient for this use case but all in all 17:47:38:402 (triple colon) is not any kind of known timestamp.

I couldn't find any easy workaround. Maybe some processor before this one to convert the last colon into a dot . (with the appropiate layout change, of course)

[fkelbert]

Thank you for doing that research @sayden.

As a user of this functionality, I would have assumed that the separators do not really matter and that I can essentially use any separator as long as they match up in my timestamps and within the layout description. I was thinking of the layout as just a "stencil" for the timestamp.

At the very least, such restrictions should be described in the documentation. Ideally, we would even provide a list of supported formats (if this list is of a reasonable lenvth). Users shouldn't have to go through https://godoc.org/time#pkg-constants

[KamilJedrzejuk]

This still not working cannot parse?
timestamp: field: '@timestamp' layouts: - '2020-05-14T07:15:16.729Z' test: - '2020-05-14T07:15:16.729Z' - drop_fields: fields: ['@timestamp']
@fkelbert

[sonalkr132]

TLDR: Go doesn't accept anything apart of a dot . to parse milliseconds in date/time.

Only true if you haven't displeased _the timestamp format gods_ with a "non-standard" format. '2020-10-28 00:54:11.558000' is an invalid timestamp. :ng_man:

 Exiting: error initializing processors: failed to parse test timestamp: failed parsing time field custom_time='2020-10-28 00:54:11.558000'

[matsgoran]

This rfc3339 timestamp doesn't seem to work either: '2020-12-15T08:44:39.263105Z'

Error while initializing input: failed to parse test timestamp: failed parsing time field vault.@timestamp='2020-12-15T08:44:39.263105Z'