Emqx: Automated partition recovery mode options.

[jemc]

The docs currently state that in the event of a network partition, manual intervention is required to stop the nodes and clear the mnesia databases.

RabbitMQ (which also uses distributed mnesia) allows you to select from one of several automated partition recovery strategies. I don't necessarily think emqtt should have the same recovery strategies as RabbitMQ, but I do think some form of automated recovery would be nice.

In fact, the lack of such a feature is the primary reason I can't justify using emqtt in production, where it's not uncommon for us to have network issues in cloud infrastructure.

Merging strategies are inherently application-specific, which is why mnesia leaves it up to the application, but I think it should be possible to find one or more merging strategies that work well with MQTT semantics.

If a nice merging strategy can't be defined, it should at least be possible to automate the clearing of the mnesia tables (and disconnection of current clients so that they will re-issue their subscriptions) that currently is a manual step.

[emqplus]

@jemc, I agree with you that the cluster design should be enhanced to handle netsplit.

1. Provide tools which help administrator recover the cluster from net partition quickly
2. Define some strategies to recover the cluster from network partitions automatically

[mjason3]

@emqplus What will it happen right now to the running cluster when NetSplit occurs? Shutting down the whole cluster or just some nodes stop serving traffic?

[emqplus]

Hi @mjason3, have to reboot the nodes in the minority.

[je-al]

@emqplus so, how can we identify the minority? would we restart any node in which the inconsistent_database event pops-up, or do we have to actually run cluster status against all of the nodes and parse the output to determine how many / which partitions there are?

[emqplus]

EMQ R2.3 integrated with Ekka library to support Autocluster and Autoheal features.