Beats: Add Dashboards for Metricbeat

[monicasarbu]

Missing dashboards

The following Metricbeat modules are missing a dashboard:

• [ ] aerospike
• [ ] ceph
• [ ] couchbase
• [ ] dropwizard
• [ ] elasticsearch
• [ ] etcd
• [ ] graphite?
• [x] haproxy #6934
• [ ] http?
• [ ] jolokia?
• [x] kafka #4665 #8457
• [ ] kibana
• [ ] memcached
• [x] nginx https://github.com/elastic/beats/pull/5991
• [ ] php_fpm
• [ ] postgresql
• [ ] prometheus?
• [ ] vsphere
• [ ] windows/perfmon
• [ ] windows/service - needs converted 6x to 5x
• [ ] zookeeper

The following Metricbeat modules have dashboards, but need to be transformed to use TSVB visualizations:

• [ ] golang
• [ ] mysql

Note that some modules have ? as they are too generic, and I am not sure if we can build a useful dashboard for them.

Naming convention

There is a naming convention for the dashboards, visualizations, searches across the dashboards provided by Logstash and Beats starting with 6.0 release.

Dashboards are named like this: "[Filebeat Nginx] Access logs"
Visualizations and searches are named like this: "Top processes [Filebeat Nginx]"

Documentation
https://www.elastic.co/guide/en/beats/devguide/master/new-dashboards.html

FAQ
Q: What Kibana should we use for creating the dashboards?
A: I would say the latest release.

Q: The saved dashboards objects are different for Kibana 5.x than for 6.x (see the 5.x and 6.x directories under _meta/kibana). Should we save the dashboards only for the Kibana 6.x releases or 5.x too?
A: I would say to start with 6.x first and leave 5.x for later.

Q: How to subscribe that you are building the dashboards for a module?
A: Just add your name near the module to make sure we are not duplicating effort.

Q: Are we accepting community contributions?
A: Yes, we are happy to get community contributions.

[ruflin]

@monicasarbu For the naming convention: I'm curious why the one used for Dashboards is different from the for Visualisations? First one starting with [ ... ] and the other one ending with [ ... ] ?

[monicasarbu]

@acchen97 came with this proposal. I guess it's because for the dashboards is important first to see the module from where it belongs. In Kibana, only the first x characters are displayed in the title of each visualization, so it's important for the user to see the description of the visualization and not the module from where it belongs.

[loekvangool]

@monicasarbu there seems to be a dashboard shipped with Kibana now.

[ruflin]

To get a better picture on the current state of which modules have a dashboard, we auto generate it now for our docs: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-modules.html

[loekvangool]

Forgot something important, it seems that there is now an nginx dashboard so this can be closed?

[ruflin]

@monicasarbu I'm thinking to close this meta issue as this issue had not really lead to the community contributions, we are now tracking in the docs automatically which modules have dashboards and in general adding dashboards is part of making the modules GA.

For the details on the naming we should add it to the dev guide.

[jsoriano]

Haproxy dashboards added in #6934

[monicasarbu]

@ruflin Feel free to close this issue.

[anthonyserious]

I would be open to contributing a kafka dashboard, since I probably need to brew one up anyway. I scanned over open issues and don't think anyone's created one. Just don't want to duplicate effort if there's already work under way.

[ruflin]

@anthonyserious A dashboard would definitively be welcome.

[tollgen]

Any progress on a dashboard for postgresql or anyone have a working one?

[anthonyserious]

@ruflin 👍 No promises on a timeline, but if it happens it'll be in the next few weeks.

[ruflin]

@tollgen No progress on the postgres dashboard so far.

@anthonyserious Of course, let us know if we can help in any way.

[gingerwizard]

@anthonyserious i'm looking at adding a dashboard for kafka. Unfortunately, due to the current fields its going to be difficult to produce something compelling. More specifically:

1. The stats are split across two metricsets - consumergroups and partitions. These do not have consistent fields e.g. topic, partition on which to drill down. @jsoriano is aware of this and this should be quick fix.
2. Most users are probably interested in consumer lag. The consumergroup docs have no indication of the max offset of the partition from which they are consuming - any calculation will need the partition docs therefore. This is going to produce very unreliable metrics OR result in complex kibana queries which might not perform on high partition counts. I considered something like sum(consumergroup)-sum(parition offsets) by topic. This is subject to bucketing issues however and can result in poor counts. Maybe we could give an overall indication of lag instead of a precise count.

I will host any initial dashboard on staging.demo.elastic.co for comment.

[anthonyserious]

@gingerwizard sounds good to me. I haven't had the opportunity to start anything yet, and I haven't even looked at what fields are available. Thanks for doing this!

[gingerwizard]

https://staging.demo.elastic.co/app/kibana#/dashboard/ea488d90-8e63-11e8-8fa2-3d5f811fbd0f

@anthonyserious

[gingerwizard]

This needs fields that currently aren't available in the module. @jsoriano is looking at extending the module to add these fields.

[jsoriano]

PR with the changes on fields https://github.com/elastic/beats/pull/7767

[jsoriano]

PR with the Kafka dashboard merged into master #8457

[sayden]

Re-issued here: https://github.com/elastic/beats/issues/10594

[samirsss]

Thanks for the awesome work. Metricbeats has been really helpful to us for monitoring the resources. I'd really like to see some updates for the Postgresql Dashboard.

[jsoriano]

@samirsss are you activelly using the Postgresql dashboard? It'd be great if you could open a new issue with whatever you think we can improve there. As a user of it your feedback will be really valuable! Thanks!