Ecs: proposal for Event fields

[jcallaha]

I would like to propose two new _event_ fields that I think would be useful

• event.level (keyword)
• event.sequence (long)

_event.level_ is the human readable companion field to _event.severity_, essentially identical in use to _log.level_ (which could be deprecated/removed in favor of _event.level_).

_event.sequence_ is another integrity field (like _event.hash_) for event sources that emit events in sequence, to detect missing events and/or order events that may have time stamps based on receipt time rather than emit time (syslog over UDP comes to mind).

While I'm on the topic of the Event group, is _event.id_ meant to be a unique id for for an instance of an event (such that two identical events would have different and unique event.id) or is it meant to handle the case where events have numeric or coded identifiers, e.g.:
%ASA-3-610001: NTP daemon interface interface_name : Packet denied from IP_address
might look something like the following in ECS 1.0

{
  ...
  "message": "000045: %ASA-3-610001: NTP daemon interface interface_name : Packet denied from IP_address",
  "event": {
    "action": "denied",
    "category": "firewall",
    "id": "610001",
    "severity": 3,
    "level": "ERROR",
    "sequence": 45,
    "version": "1.0.0"
  }
  ...
}

[ruflin]

For event.level: Do you see some other level types except the common log ones like ERROR, WARN, etc.?

[jcallaha]

@ruflin off the top of my head I can think of the following commonly used "levels" (human readable severity levels):

• syslog/RFC3164/RFC5424: Emergency, Alert, Critical, Error, Warning, Notice, Informational, Debug
• log4j: FATAL, ERROR, WARN, INFO, DEBUG, TRACE
• Windows Event Log: Critical, Error, Warning, Informational, Verbose
• CVSS Scoring: Critical, High, Medium, Low

[ruflin]

Seems like the first 3 are all log related. Last one is more about scoring. I wonder if level would still be the right field here?

One interesting point that shows the above: The naming is very similar but lower / upper case is different which mill make correlation tricky. I think epecially between log4j, syslog there should be correlation. Should we standardise the naming?

[willemdh]

Imho there should be an event.level too. This should be the 'normalised' / 'standardized' version of log.level (which should imho only hold the original level). This could allow us to for example reduce the original level of certain events to a more applicable level to prevent 'noise'.

[webmat]

There's been no follow-up on sequence, but I like the idea.

As for level, I can see the use of having one for the raw value and one that's normalized, as @ruflin points out.

Should this be log.level & event.level? I could go with that. I think it would be better than the approach we took with risk score. For risk score, the raw value is risk_score and the normalized value is risk_score_norm. We went this direction because we were unsure of how useful/clear the normalization would be, vs a given system's documented risk scores. In the case of .level, I think the normalized value will be more useful and the one used by default, so I wouldn't put it in .level_norm, I think I do prefer event.level.

[jcallaha]

Great discussion! I love the idea of normalization from a correlation standpoint but if one of the design goals is to encourage implementations to use as many ECS fields as possible I would be careful about being overly prescriptive.

I found some Symantec and Palo Alto documents that used: MAJOR and MINOR in addition to the terms mentioned earlier. If we ask developers to normalize to a specific list, what do they do when the source system's terms aren't included in our normative list?

As a reasonable compromise - what about we ask/suggest implementations to at least use ALL CAPS for this field?

[webmat]

AH YEAH I LOVE ALL CAPS. THEY REALLY MAKE THE INFORMATION POP OUT! Haha I'm joking.

I do see value in being a bit more loose with how this gets normalized. But we'd have to have two prescriptions

• Either all caps or all lowercase. I have a slight preference for the latter, but I'm ok with both.
• Normalize to full words if this is a contraction. E.g. INFO => INFORMATIONAL

On the other hands this is one of the rare cases where I think being prescriptive could potentially work. Just out of your comment from yesterday, I think we can already build a pretty comprehensive list:

• emergency
• fatal
• alert
• critical
• error
• warning
• notice
• informational
• debug
• trace
• verbose
• unknown (from Sensu/Nagios)

I could see a few more being reported and ending up with a pretty complete list of 15-ish tops, no?

Note that I didn't include the CVSS scoring ones for now, as it's more risk related than event criticality level. I feel this belongs more in a vulnerability part of ECS. This top level object is still to be added, but has been proposed already in #113, and the team agrees we need them, so it's part of the plan (#115)

[jcallaha]

Heh - all lower case does look better - agreed.

If we have ECS normalize event.level do you also want to normalize numeric values for event.severity, say something like below? (I also looked at a few more uses to see how well they map)

Syslog | Log4J | Windows | MongoDB | Postgresql | event.level | event.severity
-- | -- | -- | -- | -- | -- | --
Emergency (0) |   |   |   | PANIC | emergency | 100
  | FATAL |   | Fatal (F) | FATAL | fatal | 90
Alert (1) |   |   |   |   | alert | 80
Critical (2) |   | Critical |   |   | critical | 70
Error (3) | ERROR | Error | Error (E) | ERROR | error | 60
Warning (4) | WARN | Warning | Warning (W) | WARNING | warning | 50
Notice (5) |   |   |   | NOTICE | notice | 40
Informational (6) | INFO | Information | Informational (I) | INFO | informational | 30
Debug (7) | DEBUG |   | Debug (D) | DEBUG[1-5] | debug | 30
  | TRACE |   |   |   | trace | 20
  |   | Verbose* |   |   | verbose | 10
  |   |   |   |   | unknown |  

*I am not sure if anything uses Verbose level in Windows, but it is defined in this which I was using as a reference: https://docs.microsoft.com/en-us/windows/desktop/wes/defining-severity-levels
I also know there are Audit Success and Audit Failure events in Windows - perhaps how those are mapped could be left as a configuration option in Winlogbeat

[webmat]

For now I would start with only guidance on how to normalize event.level.

I would also be very careful to not map words from one to another. I would rather add PG's "panic" than map it to "emergency".

I would hold off for now on assigning a numeric value. Here are two reasons that come to mind:

• Some products may rank the same words in a different order (product X: alert > critical vs product Y: critical > alert)
• If we add levels we had missed, our severity mapping will have to change
  
  
  
  • or we insert a severity in between existing numeric values: once at 45, then at 42, like in the good old days of numbered code lines :-)
  
  

I'm thinking this mapping to numeric could very well be useful, but for now I would leave it out of the spec. Organizations are free to attempt this, see they derive value out of this normalization, and report back their findings.

Most up to date list, now including "panic":

• panic
• emergency
• fatal
• alert
• critical
• error
• warning
• notice
• informational
• debug
• trace
• verbose
• unknown

[jcallaha]

Sorry I probably should have made clear my table was meant to be an example of how several implementations /might/ map their internal severity/priority to our proposed normalization in event.level and a hypothetical normalization of the event.serverity.

I assumed PG's use of "panic" was based on the deprecated "panic" in syslog.h given PG's lineage, but it would appear in actuality when PG emits events via syslog it maps PANIC to syslog's Critical

What about event.severity_norm defined as a float from 0 to 100 just as a place holder? Same scale as risk_score_norm and essentially same idea?

[dcode]

I think we should leave event.severity as a distinct value that is orthogonal to the use of the proposed event.level. I'm coming at this from a security perspective, where the severity may be ranked on a scale of 1 to 3 (in the case of Suricata/Snort alerts where 1 is the most severe) or 1 to 10 as a float with regards to something like CVSS vulnerability scores (where 10 is the most severe).

I think severity in particular is likely a very use-case specific thing.

[vbohata]

Why to reinvent the wheel? Syslog defines some commonly used severities codes and names. This could be also the normalized value for other use cases. So why not to do it like this:

• event.severity.name
• event.severity.code
• log.level

While event.severity.* are normalized values, name is critical, error, ... code is a number.

This will not only add normalized value of event but also ECS useful fields for syslog usage.

[webmat]

439 adds event.sequence. It doesn't address the normalization of log.level yet, however. So we'll keep this issue open.