Ecs: Clarify @timestamp vs event.created

[mbrancato]

To quote the current readme from event.created:

event.created contains the date when the event was created.
This timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp.

If say we take a fake login pseudo-event:
Jan 20 10:17:24 SERVER auth: username logged in

The user actually logged in at 10:17:24 for all we know. Based on this comment by @ruflin https://github.com/elastic/ecs/issues/8#issuecomment-393418791 it would seem that event.created is the time that whatever tool (logstash, beats, etc) reads the file or received the syslog event. I do not think Filebeats works this way, and I know Winlogbeats does not. It will set @timestamp to the time the event occurred or when it landed in the local logging system, which I view as correct.

Can what "processed" means here be clarified? To me, when the event was created is when it occurred, and processed is when some tool (which could be many, Beats, LS, ES, etc) received the event. My real-world use case allows for events collected by a management console to make the time an event happened and the time when it was processed to be several minutes apart.

[webmat]

Hey @mbrancato, yeah I agree the doc isn't super clear here. And I think there's an issue with the field name event.created as well.

Here's the behaviour you'll see by default in Beats 7.0 (7.0.0 beta 1 just out, btw):

• @timestamp is the time restored from the event
• event.created is the time an agent (e.g. Filebeat or Winlogbeat) has first seen the event

So the simple answer is you should likely try to follow this ^

Here's some more thoughts about this, though:

The name event.created:

1. makes sense if you look at it from the POV of "created in the event pipeline"
2. when interpreted in the event domain's terms (e.g. my Apache log), event.created sounds like the ultimate creation of the event, aka the timestamp parsed out of the event. Aka @timestamp.

Out of these two, I think no 2 is what most people will assume event.created means.

So I see two things we can do here:

• Modify the definition of event.created to make it clearer that no 1 is the correct interpretation. That's a minor doc change, and pretty easy to fix
• Find a less ambiguous name than event.created. That's the cleanest answer for the long term, IMO. However since it's a breaking change vs 1.0, we'll have to perform a deprecation, then cutover to the new field name for 2.0.

We'll definitely do the first, we'll see if we need to do the second.

[mbrancato]

I think this sounds like a good plan. My goal was not to call for a renaming of event.created, just a clarification.

There are some data sources I see where an event happens on an endpoint, the management console for that tool collects the event a few minutes later, then passes it to Logstash. IMO the event happened when it happened, and that is the @timestamp. This lets me correlate that event with say network flows, or other data. However, the event may have been received a few minutes later, and I think that is a good candidate for event.created.

To use the Winlogbeats example. If the Winlogbeats service is stopped for a few minutes, and then restarted, it should not change the @timestamp of events in its backlog to process, but it might affect the event.created field.

[praseodym]

There is one disadvantage to this. Many log sources (especially syslog) do not use sub-second precision timestamps, which means that using a parsed value as @timestamp will cause Kibana to display events in the wrong order (the ordering within a second will be essentially random). In these cases, it might be better to use event.created as the time field in Kibana.

[webmat]

Having both fields is specifically to help work around or understand situations as you both describe: pipeline delays and imprecise source timestamps 👍

[mbrancato]

I understand, and I think there is value here with multiple items. In fact, I track timestamps ate multiple steps in an event pipeline. In general, I understand the precision issue, but to be honest, most of the events I get do not have sub-second precision.

So I think to propose and initial solution is just to come up with new wording for @timestamp versus event.created.

[webmat]

Yes, it's one of the fields we'll be clarifying in time for the 1.0.0 release, with the release of 7.0 :-)

Thanks for your input!