Rabbitmq-server: Expose metric for a network partition

[Artimi]

Hi,
I'm happy that RabbitMQ got its own Prometheus exporter. :+1: also for prepared Grafana dashboard, it looks great.
I was using deadtrickster prometheus rabbitmq exporter before. It allowed me to monitor network partition with its rabbitmq_node_up metric https://github.com/deadtrickster/prometheus_rabbitmq_exporter#nodes because it shows whether one node is connected to another. I have three nodes in my cluster for now and thus I can write an alert

sum(rabbitmq_node_up{app="rabbitmq"}) != 9

To check that each RabbitMQ node is connected to the other two. When it is not I'm alerted. Would it be possible to add something similar to this plugin?

[michaelklishin]

There is a separate Erlang distribution exporter and Grafana dashboard. It provides a lot more information than a boolean or an integer gauge.

[michaelklishin]

Whether a node is "connected" is not very clear cut. Network links can slow down, be permanntly saturated and so on. These new dashboards are significantly more descriptive and advanced:

@gerhard can explain how to provision them but I believe the above panels demonstrate that this is a much better approach.

[gerhard]

While showing network partitions from RabbitMQ's perspective would be helpful, it is not as helpful as understanding what happens at the Erlang Distribution level. This is the layer within the Erlang VM (RabbitMQ's runtime), on which all RabbitMQ communication depends. For example, links between RabbitMQ nodes are bi-directional, may have TLS enabled, and may be switching between different states. The Erlang-Distribution dashboard captures all this information. Without understanding what is happening in this context, a RabbitMQ partition is worth knowing about, for sure, but it doesn't contain sufficient detail to understand why it happened.

I can appreciate that the Erlang-Distribution dashboard contains more information than you would expect, and I can see how being able to see network partitions on the RabbitMQ-Overview dashboard would be helpful. I will re-open this issue since I would like to add this on RabbitMQ-Overview, and most likely link to Erlang-Distribution from RabbitMQ-Overview for those that want to dig deeper.

While Erlang-Distribution dashboard is not uploaded to RabbitMQ Grafana org yet, you can get an import-friendly version by running the following make target in the root of this repo, i.e.:

cd github.com/rabbitmq/rabbitmq-prometheus
make Erlang-Distribution.json 
{
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "6.0.0"
    },
...
  "timezone": "",
  "title": "Erlang-Distribution",
  "uid": "d-SFCCmZz",
  "version": 1,
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "prometheus",
      "description": "",
      "type": "datasource",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ]
}

There will be a recording of Observe & Understand RabbitMQ from RabbitMQ Summit on YouTube later on this month which will have more detail on this dashboard, as well as others. There will also be a RabbitMQ webinar on the 12th of December on the same topic. If you want to join, follow our Twitter where we will post more info.

[Artimi]

Thank you @michaelklishin for pointing me to the Erlang - Distribution dashboard. And thank you @gerhard for taking care of this issue. I was working only with RabbitMQ - Overview dashboard and wasn't aware of Erlang - Distribution. I took the dashboard from https://github.com/rabbitmq/rabbitmq-prometheus/tree/master/docker/grafana/dashboards and it seems really informant. I will prepare my alerts according to the queries on the dashboard.

[michaelklishin]

@gerhard @Artimi when should we consider this issue to be resolved?

[gerhard]

We can close this issue when we have exposed a network partition metric from RabbitMQ's perspective, and when this metric is displayed on RabbitMQ-Overview dashboard. RabbitMQ can still consider itself partitioned while everything looks healthy from the Erlang Distribution perspective. To account for this scenario, operators will need to take into account both perspectives - meaning both metrics - to be certain that it's a genuine network partition, and not something that can be resolved by restarting the partitioned RabbitMQ node so that it can re-join the cluster and resume service.

[sventschui]

Disclaimer: I‘m totally new to erlang and rabbitmq, therefore sorry for the noob questions right away.

I might be able to invest some hours into a PR to implement this. From browsing the code I suspect one approach would be to call rabbit_mnesia:partitions/0 and expose the length of the partition list as rabbitmq_partitions_count. Does that sound reasonable?

[michaelklishin]

Thank you for considering a contribution. Yes, that's the source of
information we should use. Whether we expose just a count or more details
is up for debate. Having a metric that describes what nodes the emitting
one is partitioned from can be useful in my opinion.

On Fri, 10 Apr 2020 at 11:32, Sven [e-mail] wrote:

Disclaimer: I‘m totally new to erlang and rabbitmq, therefore sorry for
the noob questions right away.

I might be able to invest some hours into a PR to implement this. From
browsing the code I suspect one approach would be to call
rabbit_mnesia:partitions/0 and expose the length of the partition list as
rabbitmq_partitions_count. Does that sound reasonable?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rabbitmq/rabbitmq-prometheus/issues/15#issuecomment-611938041,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAAIQRIW7CFBR4G5NCZETLRL3KSRANCNFSM4JH2B43Q
.

>

Staff Software Engineer, Pivotal/RabbitMQ