Beats: Support for wildcard in jolokia module in metricbeat

[diavel]

Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

For security vulnerabilities please only send reports to [email protected].
See https://www.elastic.co/community/security for more information.

Describe the feature:
Support wildcards for Metricbeat Jolokia module

As pointed out in this discussion, there are many places where the wildcard support in the jolokia module would come in handy. I am trying to collect jmx metrics from flink through the jolokia module and lack of wildcard support makes it unusable. Jolokia does support wildcard in it's queries.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Setup Jolokia metric beat module
2. Use jmx mapping similar to what is shown below.

jmx.mappings:
       mbean: 'Catalina:name=*,type=ThreadPool'
       attributes:
                  attr: maxConnections
                  field: max_connections 3. 

Provide logs (if relevant):
Error observed when using wildcard

errors: metric key '' not found in response; metric key '' not found in response

[ruflin]

Thanks for the feature request. I agree it would be valuable to have this feature in our module.

[daniilyar]

We would very benefit from wildcards support also. Our Jolokia configs for Kafka, Spark and Hadoop have thousands of lines that makes them really hard to maintain.

To feel our pain, just try to gather Kafka JMX metrics for e.g. 50 topics with metricbeat jolokia )
In Kafka JMX for each topic there are identical requestsPerSec, bytesPerSec and etc metrics (totally about 20) that without wildcards support one should just hardcode in metricbeat config for each topic. 20+ metrics multiplied by 50 topics = 1000 metrics to hardcode, or 2000 lines in metricbeat config file. And that would be only topic metrics, imagine that there are also ~10 identical metrics metrics for each of 15 Kafka request types, and many others.

We have written some generators of metricbeat configs and we run them each time we upgrade Kafka and metric names change or when we need to add or remove some fields from monitoring. But ideally we would like to avoid any metricbeat config generation or at least simplify it and wildcards support would be the best way do make this happen.

[ruflin]

@daniilyar Thanks for sharing your use case. Perhaps you can also chime in on https://github.com/elastic/beats/pull/6462 to what your preferred way would be related to event generation (see discussion).

[jsoriano]

We already merged #6462, I think this covers the cases mentioned here, closing this, thanks for your comments!