Ecs: [meta] Add support for email in ECS

[webmat]

We'd like to add support for email in ECS. Email is a big topic, so this issue isn't meant to be exhaustive 😂

This could cover:

• Email activity streams from email server logs or events published by email SaaS solutions
  
  
  
  • delivery events, engagement events
  
  
• DNS records around email like SPF, DMARC

Past discussions

411, #534, #593, #815

Beats: elastic/beats#13466

Related: #749 (mime types)

[mbudge]

I've been putting a lot of email data from logs/phishing automation into elastic recently.

Here's an summary of the structure we use in elastic

email.subject: keyword
email.from.address: string array - some log sources like Mimecast generate multiple from addresses. These include the original from address and the postmaster address. This happens when the email solution has rules configured to forward emails to other internal accounts. Logs with the same unique ID (like Mimecast aCode) are generated for the original email and the forwarded email triggered by a rule. Some users might use painless scripting to merge those logs into 1 document so they are easier to use.
email.mailfrom.address: The mailfrom address in the email header.
email.to.address: string array
email.date: timestamp for email logs and keyword for phishing. If some users extract the Date: string from the envelope Date field in the email body, this might not be a valid timestamp format accepted by Elasticsearch. Examples - 22 July 2019 11:13, Thursday, 17 October 2019 7:15 PM, 2020-08-03T15:26:21, 10/8/19 9:16 AM (GMT-04:00) or a malformed string as this field can be set to anything.
email.direction: keyword: Inbound, Outbound, Internal
email.action:
destination.ip
source.ip

The email.to, email.from and email.mailfrom also have the following fields

• domain
• registered_domain
• subdomain
• top_level_domain

email.message_id: Message ID from the email header
email.process.action: Accept, Reject, On Hold
email.process.file.extension: string array of file extensions
email.process.file.name: string array of file names
email.process.timestamp: Timestamp
email.receipt.action: Accept, Reject, On Hold
email.receipt.timestamp: timestamp
email.receipt.tls.cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
email.receipt.tls.version: TLSv1.2
mimecast.acode: Mimecast ID but the field name could be more generic.

The receipt/process fields in our case make merging mimecast logs clearer to the end user.
https://www.mimecast.com/tech-connect/documentation/tutorials/understanding-siem-logs/

[webmat]

Thanks for dropping all of these details here! This will be very useful ❤️

[nicpenning]

@mbudge How are you handling the header fields?

We too ingest emails but more specifically reported emails into Elastic.

We use:
reportphish.attachment_names
reportphish.email.body_plain_text
reportphish.email.body_raw
reportphish.email.from
reportphish.email.header.Authentication-Results
reportphish.email.header.Content-Type
reportphish.email.header.Date
reportphish.email.header.From
reportphish.email.header.MIME-Version
reportphish.email.header.Message-ID
reportphish.email.header.Received
reportphish.email.header.Received-SPF
reportphish.email.header.Reply-To
reportphish.email.header.Return-Path
reportphish.email.header.Subject
reportphish.email.header.To
reportphish.email.header.X-CrossPremisesHeadersFiltered
reportphish.email.header.X-CrossPremisesHeadersPromoted
reportphish.email.header.X-EOPAttributedMessage
reportphish.email.header.X-Forefront-Antispam-Report
reportphish.email.header.X-MS-Exchange-CrossTenant-AuthAs
reportphish.email.header.X-MS-Exchange-CrossTenant-AuthSource
reportphish.email.header.X-MS-Exchange-CrossTenant-FromEntityHeader
reportphish.email.header.X-MS-Exchange-CrossTenant-Id
reportphish.email.header.X-MS-Exchange-CrossTenant-Network-Message-Id
reportphish.email.header.X-MS-Exchange-CrossTenant-OriginalArrivalTime
reportphish.email.header.X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp
reportphish.email.header.X-MS-Exchange-Organization-AuthAs
reportphish.email.header.X-MS-Exchange-Organization-AuthSource
reportphish.email.header.X-MS-Exchange-Organization-ExpirationInterval
reportphish.email.header.X-MS-Exchange-Organization-ExpirationIntervalReason
reportphish.email.header.X-MS-Exchange-Organization-ExpirationStartTime
reportphish.email.header.X-MS-Exchange-Organization-ExpirationStartTimeReason
reportphish.email.header.X-MS-Exchange-Organization-MessageDirectionality
reportphish.email.header.X-MS-Exchange-Organization-Network-Message-Id
reportphish.email.header.X-MS-Exchange-Organization-SCL
reportphish.email.header.X-MS-Exchange-Processed-By-BccFoldering
reportphish.email.header.X-MS-Exchange-Transport-CrossTenantHeadersStamped
reportphish.email.header.X-MS-Exchange-Transport-EndToEndLatency
reportphish.email.header.X-MS-Office365-Filtering-Correlation-Id
reportphish.email.header.X-MS-Oob-TLC-OOBClassifiers
reportphish.email.header.X-MS-PublicTrafficType
reportphish.email.header.X-MS-TrafficTypeDiagnostic
reportphish.email.header.X-Microsoft-Antispam
reportphish.email.header.X-Microsoft-Antispam-Mailbox-Delivery
reportphish.email.header.X-Microsoft-Antispam-Message-Info
reportphish.email.header.X-OrganizationHeadersPreserved
reportphish.email.header.X-OriginalArrivalTime
reportphish.email.header.X-OriginatorOrg
reportphish.email.header.X-SEG-SpamProfiler-Analysis
reportphish.email.header.X-SEG-SpamProfiler-Score
reportphish.email.received
reportphish.email.reported
reportphish.email.sender
reportphish.email.subject
reportphish.email.to

The header is dynamically ingested into a field via a PowerShell script so header data may vary. The sample above is just the filed names from 1 email.

[AydinChavez]

I'd rather recommend to put the email fields as subfields to the already existing root level fields like source and/or destination like it is already proposed in the documentation for the user fields.
Also do not rely on the email field names (like from or sender, recipient or to, ..) but using abstract names like address:

Example:

source.email.address
destination.email.address

[mbudge]

A combination of the following would work well.

source.email.address
destination.email.address
email.from.address
email.to.address
email.cc.address
email.bcc.address
email.reply_to.address

These fields would be more generic so email addresses are easier to search.
source.email.address
destination.email.address

These fields would keep a record of how each address was extracted from the email.
email.from.address
email.to.address
email.mailfrom.address
email.cc.address
email.bcc.address
email.reply_to.address

[BenB196]

Just a thought, but would it be better to not do:

source.email.address
destination.email.address

And instead use:

source.user.email
destination.user.email

[jamiehynds]

I've submitted an RFC for an email fieldset in ECS. Would love to get some thoughts on the fields and any additional fields you'd like to see included.

https://github.com/elastic/ecs/pull/999

[the-pixel-hunter]

Is there any chance for this to be added would be very beneficial for a lot use cases

[djptek]

Hi @the-pixel-hunter there is an RFC open for an email fieldset - you can view that here: https://github.com/elastic/ecs/blob/master/rfcs/text/0010-email.md

[jamiehynds]

Thanks @djptek! @the-pixel-hunter as the email RFC advances, we're seeking as much feedback as possible to ensure the proposed fields meet the use cases our users have in mind. If there's some e-mail fields you feel that we're missing in the RFC, feel free to chime in :)

[the-pixel-hunter]

I would love to see some more around security actions, such as fields for when emails are held. I personally think there should be a header field to clearly define the email headers. also, a lot of email solutions provide a unique identifier that's different from the internet message_id is there a way to add this in?

But absolutely love this, any idea how long it will take to be moved into to ECS?

[jamiehynds]

Thanks for the feedback @the-pixel-hunter! When emails are held, I think the 'event.action' and 'event.reason' fields may be a better fit than an email field. e.g. event.action = quarantined and event.reason = malicious. What do you think about that approach?

Are there any header fields you're particularly interested? Looking at the list of header values, including all header fields may be overkill for common use cases.

Great point on a unique identified applied by email solutions - will come up with an appropriate field.

[the-pixel-hunter]

O defiantly all header fields would be overkill, some of the useful I think could be reply-to, from would probably be the most useful.

would you also be against having a email.sender.display_name field etc f?

[djptek]

Hi @the-pixel-hunter I think you'll find reply-to is already in the RFC, see email.reply_to.address, so we are working on that, however, display_name isn't part of the standard so you'd probably want to add that as an extended field if/when the RFC is able to move forward

[smokris]

Some questions on https://github.com/elastic/ecs/blob/master/rfcs/text/0010-email.md:

email.message_id | keyword | Internet message ID of the message

Does this mean the RFC5322 Message-ID: header ("a [globally] unique message identifier that refers to a particular version of a particular message")? If so, maybe clarify that, to disambiguate it from other IDs such as local queue ID.

email.direction | keyword | Direction of the message based on the sending and receving domains

Should it specify which values to use? (Prior discussion: https://github.com/elastic/ecs/pull/999/files#r527885823)

email.domains | keyword[] | domains related to the email

Could you clarify what "related" means here?

---

[@jamiehynds] If there's some e-mail fields you feel that we're missing in the RFC, feel free to chime in :)

Some other email attributes that we include in our Elasticsearch schemas, which might be useful to standardize:

• Local queue ID (parsed from the final hop's RFC822 Received: header)
• SpamAssassin-style floating-point spamminess rating (X-Spam-Status:'s score= key), where higher values indicate a message is more spammy
• List of individual spam tests that the message triggered, as X-Spam-Status:'s tests= key
• Time delay due to greylisting (X-Greylist:)
• RFC4021 Importance: / X-Priority:
• RFC4021 Content-Language:
• RFC4021 List-ID:
• RFC2076 X-Mailer:
• RFC3834 Auto-Submitted: / RFC3834 Precedence: