Ecs: Address possibility of multiple code_signatures per object

[marshallmain]

Currently the code_signature fieldset is present in a few places, including file.code_signature. We've encountered the possibility of a single file having multiple signers. With the current official ECS fields the way to represent this would be to make the code_signature field an array of objects. However, with this approach we lose the ability to query the signatures by multiple fields at once, e.g. searching for signers that are both trusted and match a specific signer name, since the array of objects is flattened under the hood.

There are a couple options I can think of that would potentially solve this issue:
Option 1
Add a new fieldset code_signatures that has the same fields as code_signature but is a nested type instead. This avoids any breaking changes while allowing arbitrary queries with multiple code_signatures. The downside is the inclusion of a nested type which is potentially dangerous to performance, as well as the potential confusion from having both code_signature and code_signatures.

Option 2
Change the type of code_signature from object to nested. This is the simplest and would be backwards compatible despite the type change. The downside is that it would be nested even for the common case where there is only one signature which would have some performance impact (unless elasticsearch does an optimization to "inline" nested fields that only have one value? that would be interesting).

Option 3
Use trusted.code_signature and untrusted.code_signature to separate trusted and untrusted code signatures. This was the initial proposed solution for our custom use case where we wanted to look specifically for trusted code signatures with certain subject names. It comes with major downsides though: it would either be a breaking change or would duplicate information if we left in the code_signature fieldset, and it does not solve the general case of querying for a code signature that matches multiple fields.

[andrewstucki]

@marshallmain So, just to clarify--when you say, "multiple signers" are you talking about an actual signature or describing a certificate chain in a signature?

While technically feasible due to Microsoft's support of nested signatures inside of Authenticode I don't believe that anything we ship/parse actually uses multiple signatures. I'm not entirely sure the state of non-executable signatures (like RPMs or something).

That said, both Mach-O and PE have embedded _cert chain_ information in their signature format which is based off of pkcs7. So, in that case, what you'd want to be able to have is multiple x509 field sets to potentially describe an entire cert chain (as opposed to code_signature field sets). See the PR I opened recently https://github.com/elastic/ecs/pull/852.

Pinging @rw-access on this since he was the one who added the code_signature field sets and he might have a bit more info.

[rw-access]

Do you have an example @marshallmain? I haven't seen that and I think it'll be easier to have a discussion and eventually a decision when we have an example to work with.

[marshallmain]

@andrewstucki I'm talking about actual signatures.
https://github.com/elastic/endpoint-app-team/pull/142#discussion_r379563151
The screenshot in that comment is showing a file that we shipped that was signed by both Endgame and Microsoft.
cc @rw-access @gabriellandau

[andrewstucki]

So--based off of that screenshot, I'm pretty sure those are coming from nested Authenticode signatures--correct me if I'm wrong @gabriellandau but I don't believe we actually ship multiple signatures at the moment and only send the primary signature that's in the security directory entry in the PE.

In which case, I do think that we should support this eventually--as is the case with modeling multiple users, multiple x509 certs, multiple processes, etc., but I don't think we actually have anything that's sending this info as of today.

[rw-access]

In which case, I do think that we should support this eventually--as is the case with modeling multiple users, multiple x509 certs, multiple processes, etc., but I don't think we actually have anything that's sending this info as of today.

My thoughts exactly. Can we leave this issue open as low priority until we're actually faced with that problem directly and not just conceptually? Until then, the current approach seems to be working pretty well with the data we have, both from Endpoint and from Sysmon/Winlogbeat

[ferullo]

One driving concern for this issue is allowlisting right now. I don't think we can punt on it. We want to make sure that a user can't create an allowlist exception for a signer and then allowlist malware that is _illegitimately_ signed (unless they really want to). We want to make sure a user can't allowlist on untrusted signers by accident.

From Endpoint's perspective I think we can do 3 things.

Append a string like (untrusted) to signers that aren't trusted.
This has a major downside that if we support wildcarding then Microsoft Windows* would match Microsoft Windows (untrusted).

Use different field names
See trusted.code_signature and untrusted.code_signature above. This works well for allowlisting (both for Endpoint an SIEM use cases) but comes at a data formatting oddity.

"Secretly" only allowlist for trusted signers
There is a boolean field in code_signature that indicates if the signature is trusted. Endpoint can always check this field is true if it checks any other code_signature fields. Theoretically this is lousy since its not what the user specified, but in practice it is good. The security app can require this when users create exceptions as well.

cc @gabriellandau

[marshallmain]

Summing up yesterday's slack conversation:

Short term
We'll add copies of code_signature as custom nested fields (nested in the sense of ES field type, to be clear) in the endpoint namespace of the schema. The full paths would be endpoint.file.code_signature, endpoint.dll.code_signature, endpoint.process.code_signature, and endpoint.process.parent.code_signature. The UI will have a guard that forces exceptions for specific code_signature.subject_name values to also check that code_signature.trusted == true.

With this approach the exception format will need to be updated with an option similar to the nested query for ES. This will let us define boolean logic that needs to execute on each element in a nested field rather than across the entire document.
cc @peluja1012 @yctercero

Long term
The issue with multiple code_signatures exists with other field sets in ECS, e.g. user. Fixing this would be a major breaking change, and as such whatever solution is chosen would likely not be available until ECS 2.0. We should continue exploring options for how to best implement multi-value fieldsets. A solution for one fieldset would ideally be applicable to other multi-value fieldsets as well.

[webmat]

If I understand correctly, we don't need to move forward with this yet?

[marshallmain]

@webmat Yeah we don't need to move forward with making changes to ECS immediately. If nobody objects we can close this issue and fold further discussion of the code_signature fieldset into more general discussions about how to handle all the potentially multi valued fieldsets. We can continue the exception format discussion in the appropriate private github issue.

[webmat]

Ok, closing now. Please reopen or open a new issue if/when this comes up again :-)