Beats: [Elastic Agent] Should add event.dataset to the logs and metric collected by the agent

[ph]

Should we add the event.dataset in the logs collected by the elastic agent, this might be an issue with all the integrations.
We already have this information in the dataset.name so we could just copy it over.

This seems to affect the siem app and the ml app.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[EricDavisX]

@bradenlpreston @crowens is this something you were following up on too?

[crowens]

@james-elastic and/or @jonathan-buttner are going to look at this I think.

[james-elastic]

Thanks we'll look into this

[EricDavisX]

are there other data fields we'd need to add? Just mentioning, I expect its not just the one. And thanks PH for logging it. :)

[jonathan-buttner]

@james-elastic @crowens @EricDavisX is the issue here that the event.dataset field is not present in the documents in elastic search that are sent by the endpoint?

[ph]

@webmat Is there any other field than event.dataset that solutions UI depends?

[webmat]

@ph This is a question that should be asked of the solutions.

The Security app's field requirements are detailed at length here https://www.elastic.co/guide/en/siem/guide/current/siem-field-reference.html

Log UI's field requirements are detailed here https://www.elastic.co/guide/en/logs/guide/current/logs-fields-reference.html

If you're thinking of another solution I suggest reaching out to them, to find their own list of fields they care about :-)

[EricDavisX]

@webmat do you think that public doc is updated and current with the additive needs that the 'Endpoint' half of the Security app will need? I'm not aware of that doc, and I'm in much of the Security side discussion so it seems a fair question as to who was involved and when it was updated, etc. Thanks for the convo - we can move it out of this closed issue if better?

[webmat]

Happy to follow the discussion wherever is most appropriate, if you feel the need to move it. I was just answering PH, so I'm good :-)

This was developed by the SIEM team, prior to the merge with Endpoint indeed. I know this doc page was very frequently requested, and its creation was celebrated. So I assume it's pretty up to date wrt what the Security app in Kibana consumes.

But whether this takes into account other related areas like Endpoint and the detection engine is a great question to ask the overall security team. I'm not sure what their plans are, whether there's another doc page, or if this one should be updated.

[ph]

Thanks @webmat I didn't know about theses docs.