Beats: [meta] Update to ECS 1.2 to 1.4

[andrewkroh]

ECS 1.2 has been released with these changes. Beats should be updated where possible.

ECS Changes (copied from changelog)

• Added threat.* fields to apply a taxonomy to events and alerts.
• Added fields in log.* to allow for full Syslog mapping.
• Added package.* to installed software packages.
• Added registered_domain to url, source, destination, client, and server.
• Added top_level_domain field to url, dns.question, source, destination, client, and server.
• Added group.domain field.
• Added url.extension.
• Added observer.name and observer.product.
• Added dns.question.subdomain field.
• Added error.stack_trace field.
• Added log.origin.file.name, log.origin.function and log.origin.file.line fields.
• Added service.node.name to allow distinction between different nodes of the same service running on the same host.
• Added error.type field.

Changes to Beats

• [x] Import latest ECS fields.yml to libbeat/_meta. elastic/beats#14052
  
  
  
  • Removing any duplicate field definitions.
  
  
  • Test beat export template for each Beat.
  
  
• [ ] Update Auditbeat system/package metricset with new package fields.
• [ ] Add dns.question.top_level_domain.
  
  
  
  • [ ] Packetbeat
  
  
  • [x] Filebeat Suricata
  
  
  • [ ] Filebeat Zeek
  
  
  • [ ] Winlogbeat Sysmon DNS
  
  
  • [ ] Filebeat CoreDNS
  
  
• [ ] Syslog fields
  
  
  
  • Filebeat syslog input
  
  

• [x] TLS fields to Filebeat filesets elastic/beats#15757

  • [x] zeek/ssl (maybe other zeek filesets?)
  • [x] aws/elb
  • [x] aws/s3access

• [x] Upgrade activemq module to ECS 1.4 elastic/beats#16151

• [x] Update apache module to support ECS 1.4 elastic/beats#16032
• [x] Upgrade auditd module to ECS 1.4 elastic/beats#16153
• [x] Upgrade aws module to ECS 1.4 elastic/beats#16154
• [x] Upgrade azure module to ECS 1.4 elastic/beats#16155
• [x] Upgrade cef module to ECS 1.4 elastic/beats#16157
• [x] Update cisco module to ECS 1.4 elastic/beats#16028
• [x] Update elasticsearch module to ECS 1.4 elastic/beats#16160
• [x] Update envoyproxy module to ECS 1.4 elastic/beats#16161
• [x] Update googlecloud module to ECS 1.4 elastic/beats#16030
• [x] Update haproxy module to ECS 1.4 elastic/beats#16162
• [x] Update ibmmq module to ECS 1.4 elastic/beats#16163
• [x] Update icinga module to ECS 1.4 elastic/beats#16164
• [x] Update iis module to ECS 1.4 elastic/beats#16165
• [x] Update iptables module to ECS 1.4 elastic/beats#16166
• [x] Update kafka module to ECS 1.4 elastic/beats#16167
• [x] Update kibana module to ECS 1.4 elastic/beats#16168
• [x] Update logstash module to ECS 1.4 elastic/beats#16169
• [x] Update misp module to ECS 1.4 elastic/beats#16026
• [x] Update mongodb module to ECS 1.4 elastic/beats#16170
• [x] Update mssql module to ECS 1.4 elastic/beats#16171
• [x] Update mysql module to ECS 1.4 elastic/beats#16172
• [x] Update nats module to ECS 1.4 elastic/beats#16173
• [x] Update netflow module to ECS 1.4 elastic/beats#16135
• [x] Update nginx module to ECS 1.4 elastic/beats#16174
• [x] Update osquery module to ECS 1.4 elastic/beats#16176
• [x] Update panw module to ECS 1.4 elastic/beats#16025
• [x] Update postgresql module to ECS 1.4 elastic/beats#16177
• [x] Update rabbitmq module to ECS 1.4 elastic/beats#16178
• [x] Update redis module to ECS 1.4 elastic/beats#16179
• [x] Update santa module to ECS 1.4 elastic/beats#16180
• [x] Update suricata module to ECS 1.4 elastic/beats#16181
• [x] Update system module to ECS 1.4 elastic/beats#16031
• [x] Update traefik module to ECS 1.4 elastic/beats#16183

• [x] Update zeek module to ECS 1.4 elastic/beats#16029

• _Please add addition things that need updated._

• [ ] Make sure modules populate source.address and destination.address any time the source/destination.ip field is used to allow for default field searches on IPs to work.

• [ ] Update Auditbeat network.direction values. https://github.com/elastic/beats/issues/12445
• [ ] Add file.extension to Auditbeat FIM. https://github.com/elastic/beats/issues/7138

ECS 1.3 Changes

• Added vulnerability.* fields to represent vulnerability information. elastic/beats#581
• Added event.ingested as the ingest timestamp. elastic/beats#582
• Added package.reference. elastic/beats#585
• Added package.build_version. elastic/beats#586
• Added package.type. elastic/beats#587
• Added host.domain field. elastic/beats#591
• Added process.command_line. elastic/beats#599
• Added process.exit_code. elastic/beats#600
• Added fields in tls.* to support analysis of TLS protocol events. elastic/beats#606
• Added process.parent.*. elastic/beats#612
• Added process.args_count. elastic/beats#615

ECS 1.4 Changes

• Added default text analyzer as a multi-field to user_agent.original. elastic/beats#575
• Added file.attributes. elastic/beats#611
• Added file.drive_letter. elastic/beats#620
• Added rule fields. elastic/beats#665
• Added default text analyzer as a multi-field to around 25 more fields. elastic/beats#680
• Added registry.* fieldset for the Windows registry. elastic/beats#673
• Publish initial list of allowed values for the categorization fields (previously reserved) event.kind, event.category, event.type and event.outcome. #684, #691, elastic/beats#692
• Added related.user elastic/beats#694

[webmat]

Is this on track for 7.5 FF?

[cwurm]

We also need to govendor for ecs.version.

[andrewkroh]

@cwurm Thanks for the reminder. I pushed an update to #14052 with the vendor changes.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[nemhods]

The new Winlogbeat 7.6.0 publishes events that claim to be ECS 1.4.0.
However, the security processor (https://github.com/elastic/beats/blob/master/x-pack/winlogbeat/module/security/config/winlogbeat-security.js) assigns illegal values to the field event.type:

• "authentication_success"/""authentication_failure"" should be expressed as event.type:access (?) and event.outcome:success/failure
• "process_start" and "process_end" should be "start" and "end"

I thought I'd report that here, or should i create a post on discuss.elastic.co?

[tsg]

Thanks for the report @nemhods. Reporting here is enough, and the adoption of ECS 1.4 among Beats is still in-progress targetting 7.7, so I hope we'll solve this by then.

[rw-access]

Checking in on this issue.
Will process_start/process_end be updated to start/end what about other event types? Should we (cc @elastic/security-intelligence-analytics) work with beats and ECS if necessary to help get these through? @randomuserid knows a few of these well

The more we follow ECS, the simpler it is for our rules to work across multiple data sources. Otherwise, we have to OR together the invalid values of fields event.type or rely on other fields like event.action which are implementation defined.

Update: I just checked and saw that _both_ values are currently populated, which works fine. If @randomuserid finds more inconsistencies, I'll have him add them here.

[webmat]

Yes, the fields are arrays primarily to support events that fall into multiple categories.

But the array fields also let us keep bwc with the earlier values, prior to releasing the official list. 👍