Bull: Does Bull support horizontal scaling?

[akashwmp]

Hi @manast,
I"m using bull queue in a microservices architecture under a distributed system. So I got one query whether bull queue supports horizontal scaling because I feel if multiple instances of Nodejs instances running on different machines then they each have got their own bull queue instance. So in a pub sub-model, each bull queue consumer running on different instances will process the message multiple times.

so is that the limitation of the bull queue library?

[rosslavery]

Each worker connects to a common redis instance and claims (locks) a job to prevent the same job being claimed by multiple workers.

In our experience Bull scales very well horizontally with very little effort. We process around 200,000 jobs daily with a horizontally autoscaled worker pool.

[hynra]

Do you mean Round Robin? As long as each worker is connected to the same Redis instance, they will get a different job. That is explained in their documentation: https://docs.bullmq.io/

You can have as many worker processes as you want, BullMQ will distribute the jobs across your workers in a round robin fashion.

[donaldduy-lb]

Each worker connects to a common redis instance and claims (locks) a job to prevent the same job being claimed by multiple workers.

In our experience Bull scales very well horizontally with very little effort. We process around 200,000 jobs daily with a horizontally autoscaled worker pool.

@rosslavery Could you provide more information on how you've set up your horizontal scaling? I'm coming into an issue where my pods are being killed before a job completes. I thought this might be fine because I expected the job to retry (I've set it to 5 retries), but these interrupted jobs are never retried.

[manast]

@donaldduy-lb if the worker dies in the middle of the processing it will be considered a stalled job. maxStalledJobs setting is 1 by default, you may want to increase that.

[manast]

Going to close this issue since it has been answered.