Rabbitmq-server: Queue masters should rebalance when necessary

[lukebakken]

Reported via a StackOverflow question.

To reproduce:

• Set up cluster with at least 3 nodes
• Create a queues policy like this: {"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic","queue-master-locator":"min-masters"}
• Create at least six durable queues and note that their master node is balanced among cluster members
• Start up PerfTest or some other application to exercise the queues
• Stop a node. The queue masters will migrate, but may not balance themselves among remaining nodes

During the process of moving queue masters, the ha-mode and min-masters policies should be taken into account.

A post-3.8.0 idea is to use a consensus-based rebalancing strategy becomes a possibility. It can be hash-ring based or even be pluggable. This is TBD.

[152319258]

[Ayanda-D]

hi @lukebakken - we've been working on a plugin[1] that addresses this issue. We define an ideal state in which cluster nodes possess an acceptable level of balance (which we term Queue Equilibrium ), which the plugin (FSM based) works towards achieving. We also take into account the risks involved in moving queues which aren't mirrored but yet hold messages, and leave onus on users to setup an HA-policy of choice (since variation of deployments beyond our control), prior plugin usage.

Vet it out, and see if it fits as a possible/complimenting solution for this issue. Some of our customers are already making use of this 👍

cc, @michaelklishin @hairyhum @Gsantomaggio

[1] https://github.com/Ayanda-D/rabbitmq-queue-master-balancer

[lukebakken]

@Ayanda-D hey that is really slick, I will give it a try. Thanks!

[sathishbabu96]

Hi @lukebakken right now, with the queuemaster balancer plugin, we can rebalance the queues only when we trigger it manually. Is there any possibility for enhancement or upgrade in which the plugin automatically identifies the downing of a node and rebalances the queue masters without human intervention.

[michaelklishin]

@sathishbabu96 this is a question for the maintainers of said plugin.

[michaelklishin]

Updated the issue with some more ideas we have (for post-3.8).