Beats: Proposal: change default time key from @timestamp to @ingestionTime

[naseemkullah]

To distinguish between event time, processing time and ingestion time, where the former two can be added by other parts of the logging pipeline (logger may add event time, log aggregator may add processing time).

[faec]

Thanks! I'm not sure what is meant by changing the "default" time key, as the intention of @timestamp is to specify the time of the original event (as closely as possible given the data source), which is usually what is wanted for an event's "time," but you're right that ingestion time is also important.

There has been work on this issue recently: https://github.com/elastic/ecs/pull/582 added event.ingested to the Elastic Common Schema, and https://github.com/elastic/beats/pull/14001 (which should be merged soon) adds beats support for it. Would those fit the use cases you have in mind?

[naseemkullah]

Thanks for clearing up the intention of @timestamp as I was under the impression that it was the time that the document reaches Elasticsearch.

I think my confusion arose by @timestamp often being ever so slightly after the time emitted in the actually log that gets stored.

On that topic, is it possible to merge the log's value emitted by the application with @timestamp somehow?

If I understand correctly the intention of @timestamp, (which gets added by beats, is that correct?) is to be the event time.

[faec]

If @timestamp doesn't match the time in the original log entry, then (aside from time zone issues and such) it probably means that the time in the log entry isn't being recognized or parsed, in which case it falls back on the time that the entry was read. This could be because the input isn't configured for the type of logs it's reading. Modules will typically handle this automatically, but if you're using a raw log input or an unsupported format then it might need configuration adjustments... you could ask in the beats discuss forums if you have a particular setup you're trying to fix :-)

[naseemkullah]

Thanks, that's really insightful!
I am using json logs that are emitted in UNIX format (float type). Would they have to be in Date type? In any case I'll go see what they say in the forum. Thanks. This issue can be closed.