Beats: Make use of prometheus filter settings on IBM-MQ Metricbeat module

[ChrsMark]

After https://github.com/elastic/beats/issues/15493 is closed/implemented we can refactor ibmmq light module introduced with https://github.com/elastic/beats/pull/15301.

We could change manifest file, tuning `defaults to look like:

defaults:
  metrics_path: /metrics
  metrics_filters:
    include: ["ibmmq_.*"]

cc: @mtojek

[ma-hei]

@mtojek could you give some basic instructions on how to include the ibmmq module (or point to documentation that I can read to figure this out as a person that's new to metricbeat and this codebase)? When including the following to metricbeat.yml (as suggested here https://www.elastic.co/guide/en/beats/metricbeat/7.x/metricbeat-module-ibmmq.html)

metricbeat.modules:
- module: ibmmq
  metricsets: ['qmgr']
  period: 10s
  hosts: ['localhost:9157']

  # This module uses the Prometheus collector metricset, all
  # the options for this metricset are also available here.
  metrics_path: /metrics

I'm getting "Exiting: 1 error: metricset libmmq/qmgr not found" when starting up metricbeat.

[ChrsMark]

Hey @ma-hei , did you enable the module by running metricbeat modules enable ibmmq? See https://www.elastic.co/guide/en/beats/metricbeat/7.x/configuration-metricbeat.html#configuration-metricbeat.

[ma-hei]

@ChrsMark I'm wondering if I missed something when setting up my dev environment. I followed the steps given here: https://www.elastic.co/guide/en/beats/devguide/7.6/beats-contributing.html
I'm on branch master with the latest commit being ad2672df280618fe57c131519325b5893d84a8b9
I can run "make".
I'm getting Module ibmmq doesn't exist! when running metricbeat modules enable ibmmq.
Observation:
I found that the modules that are listed when running metricbeat modules list is a set of all the modules in the metricbeat/modules.d directory. However, the ibmmq module is not in this directory but in beats/x-pack/metricbeat/module. When setting the metricbeat.config.modules.path to the latter path (which does contain the ibmmq.yml file), I can enable the ibmmq module but I'm getting Exiting: 1 error: metricset 'ibmmq/qmgr' not found when starting metricbeat. I probably missed something essential here.

[ChrsMark]

@ma-hei as you noticed you should do all the operations in the beats/x-pack/metricbeat directory since this module is under x-pack. You can run make, metricbeat modules list and everything under beats/x-pack/metricbeat exactly like you do in beats/metricbeat. Let me know if you meet any other issue 🙂 !

[ma-hei]

@ChrsMark That works! Now another question. I see that the commit that implements the IBM MQ module comes with a docker-compose file. I can run this docker container and see a IBM MQ webserver starting up. Is this docker container supposed to be used for development purposes?

[ChrsMark]

@ma-hei this kind of containers are used by the automated tests. However you can use it for testing manually your changes, I do this from time to time ;).

[ma-hei]

@ChrsMark Oh, I see. I have a general question regarding the IBM MQ metricbeat module. What is the relation between the Prometheus and the IBM MQ module? My current understanding is, that Prometheus is a metric format and the Metricbeats/Prometheus-module is capable of scrapping metrics from any kind of exporter that exports metrics in the Prometheus format. Isn't the /metrics endpoint of IBM MQ a Prometheus-metrics exporter? I guess what confuses me is: why is there a Prometheus module and a IBM MQ module? Shouldn't the Prometheus module already be capable of what the IBM MQ module does?

[ChrsMark]

Hey @ma-hei, this is a light module. Light modules can be seen as predefined configurations for generic metricsets as it is mentioned at this blogpost.

You are right that you could achieve the same result only by using Prometheus module, but this would not be predefined so as many users can reuse any dashboards, field mappings etc.

[ma-hei]

@ChrsMark That blogpost is super helpful, thanks! I think then it's really just the change that's already described in the initial comment above. I create a PR for this: https://github.com/elastic/beats/pull/16971
Would be happy about any feedback.

[ma-hei]

@ChrsMark Is there some documentation on how to run specific tests? In my PR, I'm seeing a build failure where it looks like the test in beats/x-pack/metricbeat/module/ibmmq/qmgr is failing. However, when running "go test" locally, inside that directory I'm getting
PASS ok github.com/elastic/beats/v7/x-pack/metricbeat/module/ibmmq/qmgr 0.431s
so I can't reproduce the failure locally. I assume that I'm not running the test correctly.
I'm currently running make clean check testsuite from /beats/x-pack/metricbeat/ (I see that this is what the Jenkins build is doing) but it's taking a pretty long time. I was hoping there was some way of running the specific ibmmq/qmgr test in order to debug the failure I'm seeing in my PR.
Note: I just noticed that running mage update build unitTest is actually fairly quick and that I can reproduce the failure that way. I got something to work with so no reply needed. Thanks anyways.

[ChrsMark]

Closed by #16971.