Bull: Question about Jobs Sequence and Attempts

[mauriciocirelli]

Description

Dear,
I am getting started with bull and redis as they fit our requirements pretty well.
I have got the basics working and now I have a couple questions which I hope this community can help me out.

I would like to configure an unlimited number of attempts to process a job in a queue, as, in this scenario, I cannot miss jobs. We will have an admin panel which an admin user may clean up a queue and remove jobs manually in extreme scenarios. Is it possible? If so, how can I configure the attempts parameter to achieve this?

The second question is regarding sequencing jobs. In this application, I need to run a single job from a given queue at a time, as they need to run sequentially. Even if a job fails, I need to retry it and cannot get the next job from the queue. If I understood the docs correctly, the concurrency parameter defines it and it is set to 1 by default. Is it correct or am I missing something?

Thank you for your attention.

Minimal, Working Test code to reproduce the issue.

(An easy to reproduce test case will dramatically decrease the resolution time.)

Bull version

Additional information

[stansv]

Hello. This would be a bit tricky with Bull since

1) when job is in progress it's removed from waiting queue
2) by default failed job will be added to the tail of queue if more attempts available

I'd use something like RabbitMQ for your case: its queue order is guaranteed to be fixed and acknowledgement is required to take next message.

With Bull I can suggest two solutions, both has own flaws. Practically infinity number of attempts can be achieved with setting job attempts to Number.MAX_SAFE_INTEGER. For example, you can use lifo: true and add jobs initially in reverse order (because of lifo):

import Bull from "bull";

const queue = new Bull("test", {});

queue.process(async job => {
  const {step, fail} = job.data;
  console.log("process step " + step + ", fail=" + fail);
  await new Promise((resolve) => { setTimeout(resolve, 500); });
  if(fail && job.attemptsMade < 5) {
      throw new Error("fail");
  }
});

queue.add({ step: "D", fail: false }, { attempts: Number.MAX_SAFE_INTEGER, lifo: true });
queue.add({ step: "C", fail: true }, { attempts: Number.MAX_SAFE_INTEGER, lifo: true });
queue.add({ step: "B", fail: false }, { attempts: Number.MAX_SAFE_INTEGER, lifo: true });
queue.add({ step: "A", fail: true }, { attempts: Number.MAX_SAFE_INTEGER, lifo: true });

Another approach is to put next job to queue on every successful run (either at the end of process callback or inside completed event handler), while still using Number.MAX_SAFE_INTEGER for attempts. So that jobs will be chained: one produce another and so on. But you won't be able to see what's in queue until all chain jobs will be executed.

[mauriciocirelli]

Hi, @stansv

_Practically infinity number of attempts can be achieved with setting job attempts to Number.MAX_SAFE_INTEGER_
This would be a good workaround.

_For example, you can use lifo: true and add jobs initially in reverse order (because of lifo):_
This would not work in our case. Our system is receiving UDP and TCP binary packets from remote sensors on the field, parsing them to JSON and adding to their respective customer's queue (and they want to receive the packets in order and never miss a packet). Thus, I cannot create the jobs "initially", as they are created dynamically.

_So that jobs will be chained: one produce another and so on. But you won't be able to see what's in queue until all chain jobs will be executed._
This would require another queue solution to enqueue jobs in order and pull from them to bull's queue when a bull's job succeeds.

As we are still in the beginning, I will give RabbitMQ a try. It seems that this part is critical for us:
_when job is in progress it's removed from waiting queue
by default failed job will be added to the tail of queue if more attempts available_
If jobs were not removed from the queue to be processed, it would better fit our needs.

Thank you very much for your thoughts.