Beats: Metricbeat Technical Debt meta issue

[ruflin]

This issue to collect a list of technical debt issue / tasks which should be handled after 7.0.0-beta1 is shipped:

• [ ] Clean up make update target and move it to mage update by using Go commands for all the commands
• [ ] Remove metricbeat hacks in update command inside libbeat/scripts/Makefile
• [X] Update all modules / metricsets to use v2 reporter method instead of Fetch. The goal is to make using module / global fields easier. https://github.com/elastic/beats/issues/10774
• [ ] Clean up code around dashboard / kibana to remove support for older 5.x versions.
• [ ] Simplify system tests by abstracting common components (see ES module as example)
• [ ] Simplify go integration tests by reusing common components (see ES module as example)
• [x] Move docs check to golang from python. This should allow most of the system tests to not require ES anymore but only try to run the module (without service running). https://github.com/elastic/beats/pull/11127
• [x] Implement a http test helper that can be used to fetch data, map it and compare output. This will allow to test various inputs to outputs without having to run the service. Similar to the .expected files in Filebeat modules.
• [x] Move to kubernetes client-go https://github.com/elastic/beats/issues/10337
• [x] Add missing dashboards: https://github.com/elastic/beats/issues/10594

[andrewkroh]

First bullet (and maybe second) are covered by #9842. Hopefully I can return to that PR soon.

[fearful-symmetry]

It looks like we're seeing the beginnings of 8 with #11113 and #11112 ? Considering that this will only work for modules that depend on an http interface, do we want to make a separate meta-issue of tracking the migration of that?

Also, 6 and 8 might be related. The new http test helper allows us to get rid of the TestData() pattern, which simplifies a lot. So, all we have left is the TestFetch() pattern.

There's also the new features introduced by #10727 and #11106 which we might also want to track.

[ruflin]

@fearful-symmetry Good points, feel free to update the issue directly.

+1 on a new meta issue to track the migration to the new test helper.

[fearful-symmetry]

I think I'm gonna wait until we have the new test helper in a more steady-state for me to open a separate meta-issue on it, as #11131 is still going on. So, here's the new interfaces and refinements we currently have in progress:

• ReporterV2 migration, individual metricsets tracked in #10774
• The new Reporting interface with error support, introduced in #10727. Do we want to migrate all metricsets to this? Now, or should we wait for additional features?
• The per-module logging introduced in #11106. After #11126 is merged do we want to migrate?

[ruflin]

• For the interface changes: I don't expect more changes to come in the near term. We should migrate the metricsets where it makes sense. I would expect both interfaces to stay around for a while.
• New Logging: We should migrate, also to clean up all deprecated logp.Info( etc. usages. I would assume this migration can happen together with the interface migration.

[ruflin]

@exekias Assigned this one to you, hope that is ok.