Beats: [ECS] Meta-issue to track last minute ECS-related fixes for 7.0

[webmat]

With more eyes on 7.0, we're finding a bunch of small things that still need adjusting. This is to keep track of them all.

Please add your items right in this list, and ping via a comment to notify of any additions

• [x] #11512, #11531 Fix field alias for nginx.access.remote_ip
• [x] #11538, #11544 Revert the event.type changes. Must not be used in 7.0 (Mat)
  
  
  
  • [x] don't forget the second instance (elasticsearch.audit.event_type => event.type)
  
  
• [x] ML jobs (Mat to loop in ML team)
  
  
  
  • ML jobs not updated (may have been modified by customers). discussion, es upgrade docs mention ML jobs
  
  
• [x] #11527 forward port of Ethan's fixes to master (Nic)
• [x] Do a second manual pass over all dashboards for missing renames (Mat)
  
  
  
  • [x] #11545,#11547 Add missing ecs-mig.yml entries for system module
  
  
  • [x] Migrate Redis slowlog's duration to ECS
  
  
  • Not fixing: this fileset is not based on a tailed file, but on poking Redis directly. The fileset has no test, so adding the facility to test changes required here is out of scope of last minute fixes.
  
  
  • [ ] [optional] Have the LS dashboard use event.duration instead of the millis field. Marked optional because event.duration and the millis field are present in the logs, this is purely a cosmetic dashboard adjustment, no breaking change.
  
  
• [x] Double-check dashboards with visualizations on event.duration, to see if scale differences are going to cause problems

[webmat]

ping @ruflin @EthanStrider @fearful-symmetry

[fearful-symmetry]

Currently doing some scripting to find suspect dashboard fields. Still need to look them over, then I'll post what I find here.

[fearful-symmetry]

Okay, so I modified @ruflin's script to look for all pre-ecs fields, not just alias ones. It's a tad janky, but I found a few things we should at least look at.

https://github.com/elastic/beats/blob/3c59cc891b5de81e4ba56f3250d9ad5104515300/filebeat/module/icinga/_meta/kibana/7/dashboard/Filebeat-icinga-startup-errors.json#L83

Looks like the ecs-migration script wants to to this into "log.log.level:critical"

Ditto with the mongo dashboard here

https://github.com/elastic/beats/blob/3c59cc891b5de81e4ba56f3250d9ad5104515300/filebeat/module/mongodb/_meta/kibana/7/dashboard/Filebeat-Mongodb-overview.json#L75

and here

https://github.com/elastic/beats/blob/3c59cc891b5de81e4ba56f3250d9ad5104515300/filebeat/module/mongodb/_meta/kibana/7/dashboard/Filebeat-Mongodb-overview.json#L109

The packetbeat dashboards also need to be looked at:

https://github.com/elastic/beats/blob/7.0/packetbeat/_meta/kibana/7/dashboard/Packetbeat-mongodb.json

https://github.com/elastic/beats/blob/7.0/packetbeat/_meta/kibana/7/dashboard/Packetbeat-mongodb.json

https://github.com/elastic/beats/blob/7.0/packetbeat/_meta/kibana/7/dashboard/Packetbeat-mongodb.json

https://github.com/elastic/beats/blob/7.0/packetbeat/_meta/kibana/7/dashboard/Packetbeat-pgsql.json

https://github.com/elastic/beats/blob/7.0/packetbeat/_meta/kibana/7/dashboard/Packetbeat-thrift.json

All show fields that the python script wants to change, mostly of the form "method" -> "http.request.method"

The script also seems fairly blunt, for a lot of the sql dashboards it wants to change
"query": "method: SELECT" -> "query": "http.request.method: SELECT" Which doesn't seem right.

[webmat]

@fearful-symmetry Ah the Packetbeat ones must not be migrated. It parses many kinds of protocols, so while the value of method is being copied over to http.request.method, Packetbeat still uses method across the board for all protocols.

So Pb doesn't need to be adjusted for this

[fearful-symmetry]

Yah, thought those seemed suspect.

[webmat]

Added a task:

Double-check dashboards with visualizations on event.duration, to see if scale differences are causing problems

[webmat]

Ok, with #11527 merged, things aren't as bad for the Filebeat modules. Still a few left, though.

Here's the relevant entries from searching with ag '"field":' filebeat/module/*/_meta/kibana:

filebeat/module/logstash/_meta/kibana/7/dashboard/Filebeat-logstash-slowlog.json
120:                                "field": "log.level",
175:                                "field": "@timestamp",
186:                                "field": "log.level",
303:                                "field": "logstash.slowlog.took_in_millis"
313:                                "field": "logstash.slowlog.took_in_millis"
323:                                "field": "logstash.slowlog.plugin_name",
336:                                "field": "logstash.slowlog.took_in_millis"
346:                                "field": "logstash.slowlog.plugin_type",

filebeat/module/redis/_meta/kibana/7/dashboard/Filebeat-redis.json
34:                                "field": "redis.log.role",
47:                                "field": "log.level",
111:                                "field": "@timestamp",
122:                                "field": "log.level",
321:                                "field": "redis.slowlog.duration.us"
331:                                "field": "redis.slowlog.cmd",

filebeat/module/system/_meta/kibana/7/dashboard/Filebeat-auth-sudo-commands.json
30:                                "field": "@timestamp",
41:                                "field": "system.auth.user",
102:                                "field": "@timestamp",
113:                                "field": "system.auth.sudo.error",
176:                                "field": "system.auth.sudo.command",
188:                                "field": "system.auth.user",

filebeat/module/system/_meta/kibana/7/dashboard/Filebeat-new-users-and-groups.json
38:                                "field": "host.hostname",
51:                                "field": "system.auth.useradd.name",
64:                                "field": "user.id",
77:                                "field": "system.auth.useradd.gid",
90:                                "field": "system.auth.useradd.home",
103:                                "field": "system.auth.useradd.shell",
159:                                "field": "@timestamp",
170:                                "field": "system.auth.useradd.name",
233:                                "field": "system.auth.useradd.shell",
245:                                "field": "system.auth.useradd.name",
304:                                "field": "system.auth.useradd.home",
316:                                "field": "system.auth.useradd.name",
374:                                "field": "system.auth.groupadd.name",
386:                                "field": "group.id",
442:                                "field": "@timestamp",
453:                                "field": "system.auth.groupadd.name",

filebeat/module/system/_meta/kibana/7/dashboard/Filebeat-ssh-login-attempts.json
45:                                "field": "@timestamp",
56:                                "field": "system.auth.ssh.method",
121:                                "field": "@timestamp",
132:                                "field": "event.action",
191:                                "field": "system.auth.user",
252:                                "field": "source.geo.location",

Some of these fields are correct but some don't look right. I'll open up a PR

[webmat]

Here's the breakdown on these fields.

• The last 3 chunks are all for the system module, and were missing from ecs-migration.yml. So the script didn't change them in the dashboards, and they're not in the documentation file
• Redis' redis.slowlog.duration.us was simply not migrated at all. I'll open a separate PR for this one, in case someone disagrees about such a change so late.
• Logstash' slowlog has 2 fields for duration. One in nanos, one in millis. The nanos one was migrated to event.duration, and the dashboard uses the millis (which is still present in the events). No breaking changes required for this fix, so this could be fixed later, if there's no time to get to this.

[fearful-symmetry]

Yah, I was gonna say, based on your data it looks like a few things were left out of the ecs-migration file.

[webmat]

Hand curation FTW

[webmat]

Closing this. I've double-checked the last remaining thing that worried me in what we looked at last week.

Seems like the only kibana object using event.duration in a way that needed to be adjusted has been adjusted in https://github.com/elastic/beats/pull/10604

[ruflin]

@webmat @fearful-symmetry Thanks for making this happen 🎉