Bull: Multi Instance Concurrency

[guischdi]

My setup

I have a multi instance application maintaining a bull job queue. All application instances off course use the same Redis instance. I configured bull to run a maximum number of 1 concurrent job. But this means that 1 job can be running concurrently PER INSTANCE of the application. So if I'm running two instances of the application, two jobs can be running in parallel.

My usecase

What I want to have is only one job running in parallel. Because the performed operations of the job could possibly cause invalid data, if two jobs are concurrently updating the data.

My questions

• Is it possible to define a concurrency over all application instances?
• If not: would it be possible to implement it in bull?
• Or should the jobs implementation handle the global concurrency (for example by only starting the actual job operations after bulls Queue.getActiveCount() is less than 1)?

[manast]

Not sure I understand your setup, but if you have the "worker" as a separate nodejs process, then you can have as many application instances as you want adding jobs to the queue, and since you only have 1 worker it will process all the jobs in sequence.

[guischdi]

Yes, we could handle it like this with only 1 worker process. But we want to have redundancy. In case one application instance dies, there should still be a worker to handle the jobs. Any solution for this?

[manast]

Ok, so it is more like a failover scenario than pure redundancy. I would say that the solution is to have a mechanism to quickly launch a new worker if one dies. Having more than one will always imply concurrency (besides redundancy).

[guischdi]

Ok. Thx. So a global concurrency is not possible with bull. My second question was, if it would be possible that this is a new feature of bull?

[manast]

probably not because it is really hard to implement a distributed system that works as a single instance, somehow it defeats the point if I understand your case correctly...

[deugene]

Hi. I've had the same problem, so I made a module, named oraq to handle per instance concurrency. And you can use it with Bull and also like a separate queue manager.

Example:

const Bull = require('bull');
const Oraq = require('oraq');

const connection = {
  port: 6379, 
  host: '127.0.0.1'
};
const queue = new Bull('firstBullQueue', {
  redis: connection
});
const limiter = new Oraq({
  connection,
  concurrency: 1
});

queue.on('error', console.error);
queue.on('failed', async (job, err) => {
  console.warn(`Job "${job.data.name}" failed with error "${err.message}"`);
});
queue.process(job => limiter.limit(runJob, {
  jobId: job.id,
  jobData: job,
  lifo: job.opts.lifo
}));

[LeonidEfremov]

Wrong limiter definition:

const limiter = new Limiter({
connection,
concurrency: 1
});

follow this interface

  interface RateLimiter {
    /** Max numbers of jobs processed */
    max: number;
    /** Per duration in milliseconds */
    duration: number;
    /** When jobs get rate limited, they stay in the waiting queue and are not moved to the delayed queue */
    bounceBack?: boolean;
  }