Beats: Refactors for 7.0 (Breaking changes I want to make)

[andrewkroh]

This is a running list of things to consider changing for 7.0. Feel free to add your own with attribution.

• [ ] Refactor the logging configuration (remove all the to_files, to_syslog settings etc.).
• [ ] Make Windows event log the default log output for Windows.
• [x] Set escape_html to false: elastic/beats#7445 elastic/beats#9914 @urso
• [x] Remove notion of prospector in the filebeat config. (@ph) PR https://github.com/elastic/beats/pull/8909
• [ ] Cleanup the methods used by go-plugins
• [ ] Default Template Mappings for IP fields are keyword type elastic/beats#7847 elastic/beats#9303 @ruflin
• [x] Remove ports from LS and Redis https://github.com/elastic/beats/issues/7881 elastic/beats#9934 @urso
• [x] Dissect * and named skip field change https://github.com/elastic/beats/issues/8054, PR At https://github.com/elastic/beats/pull/8750 (@ph)
• [x] Make beats log to journald by default on systems with systemd (add -e by default to systemd unit) (@jsoriano) elastic/beats#8942
• [x] Dedot docker and k8s labels by default, ~or use new queryable object type elastic/beats#9286~ (@kaiyan-sheng)
• [x] Remove calls to cfgwarn.CheckRemoved5xSetting(s) (@ph) https://github.com/elastic/beats/pull/9002
• [ ] Remove 5.x dashboard directories, add 7 directories? elastic/beats#8927
• [x] Assert any 'deprecated' comment in beats. (@ph)
• [ ] Change types to double in metricbeat modules for big gauges (@jsoriano, see #8991, #9271)
• [x] Rename custom.* to journald.custom.* fields (@kvch, already on master, needs changelog entry)
• [x] New location of registry of Journalbeat (@kvch, already on master, needs BC and changelog)
• [x] Deprecate fields/tags and introduce add_fields/add_tags processors https://github.com/elastic/beats/issues/4518 elastic/beats#9973 (@urso)
• [x] Remove ES 1.x and 5.x support in template/dashboard loading code
• [x] Unify module names for the same services elastic/beats#9395 (@jsoriano)
• [ ] Fixed any issues with the _xpack deprecation elastic/beats#9424

Would be nice to have

[ph]

Added https://github.com/elastic/beats/issues/7881

[jsoriano]

I have added a point to make beats log to journald by default when running on systemd, open to discussion, I'll open a PR.

@exekias I have added also a point about dedoting labels so we don't forget about this.

[ruflin]

I added a point about removing the 5.x dashboard directories to the above list. We should probably add a directory for 7 as the data structure for Kibana will change. @elastic/apm-server This might also effect you.

I put it on the service meeting agenda today.

[ph]

@jsoriano

I have added a point to make beats log to journald by default when running on systemd, open to discussion, I'll open a PR.

Can you create an issue for that, I don't know how big of a task it is.

[ph]

@ruflin I believe your ECS migration will take care of: Default Template Mappings for IP fields are keyword type #7847

[ph]

@urso @ruflin @exekias I have added your name to the list of items you have added to this list, feel free to either remove the items or delegate the work.

[ruflin]

The IP stuff is not directly related to ECS but parts of it might be resolved through ECS.

[jsoriano]

@ph for the systemd unit logging to journald I mean something like #8942

[ph]

FYI I've added the following methods to fails on removed options, if you want to provide better feedback use them and it make it easy for us to remove them in 8.0.

https://github.com/elastic/beats/blob/74779e4b31880eb098fae97d5d29472a4f018e17/libbeat/common/cfgwarn/removed.go#L77-L85

[urso]

recent changes:

• [x] Deprecate fields/tags and introduce add_fields/add_tags processors https://github.com/elastic/beats/issues/4518 (@urso)
• [x] Remove ES 1.x and 5.x support in template/dashboard loading code

[elasticmachine]

Pinging @elastic/infrastructure

[ph]

Added a note about _xpack's rest endpoint deprecation #9424

[urso]

I think we can close this one.

@andrewkroh @ruflin @ph @kaiyan-sheng @kvch @jsoriano Any of the items mentioned still open?

[jsoriano]

I didn't check if we should change some potentially big gauges from long to float, but I think this wouldn't be so breaking.

[kaiyan-sheng]

Nothing open on my side 👍For dedot we do have https://github.com/elastic/beats/issues/10773 to track some cleanup work.

[urso]

Not sure if all check boxes can be ticked off. But 7.0 is out for quite a while. Closing this.