Rabbitmq-server: Provide a "merging" partition handling strategy

[gsogol]

Per @tsaleh request:

If there's a network partition between a cluster of 3 nodes (2 vs 1). If one publisher sees the 2 of the nodes and the other publisher sees the other and both of the publishers publish a message then you have a scenario where the 2 partitions have messages that the other partition does not have. So we need an alert that it happened. Once alerted, we could use a UI interface to move messages from one partition to another.

[michaelklishin]

There's more than one partition handling strategy. 2 of them involve dropping client connections. In autoheal's case, the minority nodes will re-sync from the majority after they re-connect.

Is that not what you're asking for?

[gsogol]

So the main ask is to union or merge the 2 clusters when autoheal happens. You can't just discard data. These message could have million $ orders. I'd hate to lose them :)

[michaelklishin]

Ok, this is something we definitely plan on doing.

On 20/4/2015, at 18:31, gsogol [email protected] wrote:

So the main ask is to union or merge the 2 clusters when autoheal happens. You can't just discard data. These message could have million $ orders. I'd hate to lose them :)

—
Reply to this email directly or view it on GitHub.

[jemc]

I've seen lots of discussion about this, and just wanted to step in and +1 this feature request.

Specifically, I'd think it would be most useful to focus on merging _messages_ rather than conceptual server objects (like queues and exchanges), which in most use cases seem to be fairly steady state. This may simplify the problem space enough to make it feasible.

I'll also add in some relevant links I've seen of discussion on this topic:
http://www.refactorium.com/distributed_systems/messaging/rabbit/#conclusion
https://aphyr.com/posts/315-call-me-maybe-rabbitmq#comment-1930
http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/33032

[michaelklishin]

This is being addressed in two ways:

• Quorum queues (there's a more recent talk, this has been merged in master for 3.8)
• Experimental new schema data store (mentioned in this update from RabbitMQ Summit 2018)

[jemc]

Is there anywhere you can point me to read more about the new schema data store?

[michaelklishin]

This RabbitMQ Summit talk.

[jemc]

Yeah, I saw that link in your earlier post, thanks.
Are there any mailing list discussions, issue tickets, or code commits you can point me to that I can read about it in more detail?

[michaelklishin]

There are no mailing list discussions but both Ra and Mnevis are developed in the open.

[jemc]

Thanks!