Ecs: Parity with Elastic Endpoint

[rw-access]

As part of the EQL integration within Elasticsearch (https://github.com/elastic/elasticsearch/issues/49581), it's important that we have maximum interoperability so that queries can run on the endpoint or on Elasticsearch, and this requires the schemas to be shared between them. There is still a significant delta, so we've been using the endgame.* namespace in the interim for anything unmapped to ECS. Hopefully, this can serve as a meta issue and I'll try to track endpoint-specific mappings from here:

Related Issues

• Case insensitivity (#623) which EQL expects in general and seems fairly standard
• Digital signatures (#681)
• DNS (#10)
• DNS Answers (#596)
• DLL/Driver Fields (#675)
• PE header metadata (#676)
• Process parent field set (#597), exit code (#600), command line (#599)
• Unique Pid (#672)
• Windows Registry (#673)

Endgame mappings

Within our EQL analytics library, we have several fields that are mapped. I bolded the ones that are used in analytics within the repo, and checked off when mapped to ECS.

• [ ] authentication id / logon id -- for Windows, this is an identifier to reference the active token
• [x] command_line-> process.command_line and process.args
• [x] destination_address -> destination.address
• [x] destination_port -> destination.port
• [x] file_name -> file.name
• [x] file_path -> file.path
• [x] hostname -> host.hostname
• [x] image_name (#675)
• [x] image_path (#675)
• [x] original_file_name (#676)
• [x] parent_process_name -> process.parent.name (as of ECS 1.3.0)
• [x] parent_process_path -> process.parent.executable (as of ECS 1.3.0)
• [x] pid -> process.pid
• [x] ppid -> process.ppid
• [x] process_name -> process.name
• [x] process_path -> process.executable
• [x] protocol -> network.transport
• [x] registry_data (#673)
• [x] registry_key (#673)
• [x] registry_path (#673)
• [x] registry_value (#673)
• [x] signature_signer (#681)
• [x] signature_status (#681)
• [x] source_address -> source.address
• [x] source_port -> source.port
• [x] total_in_bytes + total_out_bytes -> network.bytes
• [x] unique_pid (#672)
• [x] unique_ppid (#672)
• [x] ~user~ this is just a more "qualified" domain + name combo (e.g. NT AUTHORITY\SYSTEM)
• [x] user_domain -> user.domain
• [x] user_name -> user.name
• [x] user_sid -> user.id

Categorization

We also have a concept of enums for subtypes, which is really just a list of accepted values for a given field. This will add a layer of standardization when we standardize values for ECS, which is going to be very necessary if users expect to share across data sources.

Some of the values we've enumerated, which often map to Endgame's event_subtype_full or opcode fields. These all require solving categorization and enumerated values for event.category, event.type and event.action:

• network subtypes
  
  
  
  • [ ] incoming
  
  
  • [ ] disconnect
  
  
  • [ ] outgoing
  
  
• file subtypes
  
  
  
  • [ ] modify
  
  
  • [ ] create
  
  
  • [ ] delete
  
  
• process subtypes
  
  
  
  • [x] create
  
  
  • [x] terminate
  
  

[webmat]

Thanks for opening this, Ross!

With yesterday's 1.3 release, parent process details can be captured on the event, at process.parent.*. I'm editing the issue body to that effect.

ECS issues to open

• Can you open an issue or PR for adding registry support? There's no doubt in my mind that ECS needs to support registry events. The details of how we structure that can be hammered out over there, without crowding out this issue.
• Could you or @andrewstucki open an issue about unique PIDs? I don't recall the exact details of how Endpoint computes it. I think this would be a useful concept to spread via ECS, if the algorithm is simple enough to explain via schema documentation.

Questions

Here's a few points or questions wrt to the rest of what I see:

• 3 out of 4 fields are mapped for user details. Why isn't user mapped? What kind of value is typically in this field?
• total_in_bytes and total_out_bytes should probably be mapped to source.bytes and destination.bytes.
• What's a logon_id vs a user id?

[andrewstucki]

@webmat , @rw-access here's the unique pid discussion issue: https://github.com/elastic/ecs/issues/672

[webmat]

And Ross opened the registry discussion at https://github.com/elastic/ecs/issues/671

[webmat]

I'm looking at the body of the issue and I'm wondering why DNS is there. Could you open an issue stating specifically what's missing from your POV, from the current DNS field set?

[rw-access]

Oh sorry, I think DNS is good. I was just being verbose and linked to existing issues.