Beats: [Filebeat] Field aliases yield confusing Kibana Discovery results

[andrewkroh]

When browsing the Kibana Discover tab I noticed that suricata.eve.timestamp and suricata.eve.flow.start were present for events that were NOT from Suricata. It's somewhat misleading to have the suricata.* fields rendered when it's not a suricata event.

These fields are not in the _source. They are aliases pointing to @timestamp and event.start, respectively, which did happen to exist in the events that I was browsing.

• https://github.com/elastic/beats/blob/e4686d03bef1ad0c88686ef07f316804508b3c06/x-pack/filebeat/module/suricata/eve/_meta/fields.yml#L180-L182
• https://github.com/elastic/beats/blob/e4686d03bef1ad0c88686ef07f316804508b3c06/x-pack/filebeat/module/suricata/eve/_meta/fields.yml#L679-L681

History

These aliases were added (https://github.com/elastic/beats/pull/10377) in order to avoid duplicating data storage in an event while still maintaining the original suricata EVE JSON structure.

Possible Solutions

• Remove the aliases and duplicate the data within _source.
• Remove the aliases and remove the suricata.* fields that renamed to an ECS field. Users familiar with the original EVE format might be a little lost until they learn ECS.
• Ideas?

---

Version: Filebeat 7.0.0 / Kibana 6.6.0

[elasticmachine]

Pinging @elastic/secops

[webmat]

But this kind of usage of aliases is to help migration from old schemas to the new ECS-centric schema. Is this a problem across the board, for example with fields such as source.address?

[ruflin]

I tried this out with the apache Filebeat module and -E migration.enabled which means all aliases for nginx, apache etc. are loaded. Looking at the event, I don't see also the alias fields. It seems the above only applies to date values? I'm not sure what the logic here is on the Kibana side.

Here a screenshot of an event:

[ruflin]

@andrewkroh Just noticed one difference in my tests: I tested against KB 7.0.0 Snapshot. Thought worth mentioning in case it makes a difference.

[sophiec20]

Found in 7.0.0-SNAPSHOT { "build" : { "hash" : "a90fcf0", "date" : "2019-02-20T18:17:35.110454Z" },

Loaded vanilla nginx logs into a clean instance and see the following fields in Discover which I don't think should be there.

Possibly not a kibana problem as the following return hits despite there not being any suricata data.

GET filebeat-*/_search
{
    "query": {
        "exists" : { "field" : "suricata.eve.timestamp" }
    }
}

[willemdh]

I'm seeing this suricata.eve.timestamp field in several datasets that have nothing to do with suricata.. (also we are not using suricata anywhere) Glad I found this issue, is there any plan to get rid of it?

[Jimmy-Singh]

We are using filebeat 7.3.0 on our Kubernetes cluster and noticed the "suricata.eve.timestamp" field popping up on every event. Did a bit of digging up and found that the field is set on the filebeat template; under suricata properties; as an alias to field @timestamp (line 3899).

"timestamp": {
"path": "@timestamp",
"type": "alias"
}

Rectified the problem by removing those particular lines of code in the template and rolled over the index to a new one.

[J7mbo]

@Jimmy-Singh How exactly did you do this? You had to export the template or something first? What steps / commands can I run to do the same that you did?

[Jimmy-Singh]

You will have to modify the filebeat template; you can do so by exporting -> modifying -> import via CURL or by using the GET/PUT template API via the kibana dev tool ( https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html ).

Make sure to rollover the index after that ( https://www.elastic.co/guide/en/elasticsearch/reference/7.6/indices-rollover-index.html )

[willemdh]

Why is this still in the latest beats templates?

It's very confusing for some users to see this in every filebeat source..

[willemdh]

@andrewkroh Even in 7.8.1 this is still in the template? Is there any technical reason why removal of "suricata.eve.timestamp" takes so much time? The number of times I had to explain this...

[andrewkroh]

It hasn't been changed because it would be a breaking change to remove the field, but I think the bugfix justifies the breaking change and we should remove it.