Moleculer: Rabbitmq integration as work queues

[cristin]

Hi all,

Has anyone managed to integrate Rabbitmq into moleculer? Or do you have any insights or best practices on how to better integrate this as work queues?

Thank you!

[lucduong]

@cristin Sure, https://github.com/hoanganh25991 @hoanganh25991, Explain him pls.

[Nathan-Schwartz]

Hi @cristin

There is an AMQP transporter could probably support your use case. It is used in the same way that other transporters are (there is no special API), but it does have some nuances. I wrote the first iteration, but @dkuida and @urossmolnik have done most of the development since then so they might have more up-to-date information than I do.

Here are some things that I believe are specific to the AMQP transporter, hopefully you find them useful:

1. AMQP Queues and Moleculer actions have a 1:1 relationship (this could be changed in a PR)
2. Currently queue and channel configuration is fairly limited (this could be changed in a PR)
3. You can do load balancing with AMQP's prefetch instead of Moleculer's built-in balancing strategies, but you must specify this at the node level. The option is called disableBalancer.
4. If you configure durable queues and persistent messages, node outages will not result in message loss (assuming action handlers don't resolve until all work is done).
   a. If a worker goes down while processing, the message(s) would be redelivered to an available worker as soon as one is available and the caller would get a response once the message has been processed.
   b. If the caller goes down before the message is handled, the message will still be redelivered and the response would go back to the caller's response queue. However since the caller went down and response queues are specific to each node, the response wouldn't be utilized so the caller might have to make the request again. This makes idempotency fairly important since messages may be processed twice.

The transporter works, and I have been using it in production for a year, but as you can see it has some nuances.

Another option would be to utilize a Moleculer module for worker queues. Currently there isn't one for AMQP, but here is a list: https://moleculer.services/modules.html#tasks

[cristin]

Hi @Nathan-Schwartz ,

Thank you for all these details, they are very helpful. If I understand correctly from your answer, there is no specific need to use RabbitMq, right? I can only use the AMQP transporter that's built-in Molecular, right?

[Nathan-Schwartz]

@cristin I've always used RabbitMQ with the transporter, but the transporter should work with other implementations of AMQP as well. Does that answer your question?

[omerts]

(assuming action handlers don't resolve until all work is done).

Is this true for events as well? If we return a promise, would the load balancer wait for the promise to resolve?

[Nathan-Schwartz]

Yes, for both actions and events a message shouldn't be ack'ed until the promise returned from the handler has resolved. I believe the integration tests verify this behavior, but I haven't personally tested it in a while.

[omerts]

@Nathan-Schwartz Amazing! I am so excited and astonished by this great lob.