Beats: Doc: Clarify <stack_product> vs. <stack_product>-xpack modules

[ppf2]

In short, while we have documentation showing that when using metricbeat for monitoring stack components, we will want to enable the -xpack module (e.g., logstash-xpack), it doesn't discuss the difference between the logstash vs. logstash-xpack modules.

This can be confusing to some users because they are effectively the same module if they look in the yml files (logstash.yml vs. logstash-xpack.yml)

- module: logstash

There are some subtle differences. For example, the logstash-xpack module will create some additional metadata (e.g., the index field) where as logstash will not.

         "@metadata" => {
           "version" => "7.5.0",
             "index" => ".monitoring-logstash-7-mb",
              "beat" => "metricbeat",
        "ip_address" => "127.0.0.1",
              "type" => "_doc"
    }

More importantly, logstash-xpack must be used instead of logstash. Otherwise, the Stack Monitoring UI will not look at any of the collected monitoring data.

While we do have logstash-xpack in the steps here, it can be helpful to also clarify this in the modules documentation itself:

Within the actual yml files in the modules.d directory, in both logstash-xpack.yml and logstash.yml, we simply point users to the doc link: https://www.elastic.co/guide/en/beats/metricbeat/<version>/metricbeat-modules.html. However, this link (example: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-modules.html), simply mentions 1 module (the "logstash" module). Clicking into this module, there is no mention of the difference between logstash and logstash-xpack (in fact, no mention of the -xpack form at all)

Btw, while we are on the page above (for 7.6), it also shows The logstash module is tested with logstash 6.3. - which looks like a bug?

Then in the Elasticsearch module doc page:

It does have a mention of both modules. But they both mention that they are used for monitoring with the xpack one capturing more metricsets. I think it can be helpful to make this statement stronger to indicate that Stack Monitoring UI users must use the xpack module. It also has the strange statement on the modules only being tested for 6.3/6.x ....

Similar feedback for Kibana module:

To summarize, it will be helpful to explain better the difference between the v.s., -xpack modules on the Beats' "modules" documentation page and the importance of picking the right module (esp. when we used to only have a single .yml in the past). Also, fixing the compatibility statement on these module page will be great. Thx!

[ycombinator]

The compatibility statements are being fixed. See https://github.com/elastic/beats/pull/16400 and related backport PRs.

[ycombinator]

I agree about improving the module docs to be clearer about the differences between enabling the <stack product> vs. the <stack product>-xpack modules.

/cc @dedemorton

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)