Bull: Two nodejs processes running bull possible?

[cendrizzi]

Hi, can I safely run bull from two separate processes? Not using cluster, just starting two nodejs processes, each setup to process the queues?

I ask because I'm testing the latest update RC8. I had 16 jobs waiting in a queue that are not recurring (they were added but nothing was able to process them).

I started up two separate instances of my node js job processor and it appears that it was handling the 16 jobs very well. It was taking them on two at a time. However, when I view the actual job record in redis they each have a failed reason like the following:

Missing lock for job 323 finished

The stacktrace shows this originating from node_modules/bull/lib/scripts.js:143:23.

Each one of these sixteen jobs has this same fail reason, even though they did actually complete and do what they were supposed to. Even though they failed they are actually contained in the completed list.

So am I using bull wrong or is this something else. I don't exactly know how locking works on bull.

[manast]

The whole point with bull is to be able to have many separate processes processing jobs concurrently. The error you are describing sounds like a bug, I would like to reproduce it.
Basically If I understand you correctly, first create 16 jobs, then start 2 work processes that process them, right?

[manast]

btw, how long does every job take to be processed? (in case duration matters for the reproducibility of the issue)

[cendrizzi]

The jobs take about a minute (they are actually hitting an outside api and getting records which can change depending on when I hit the API).

I will try and provide a nice piece of code that will reproduce this all nicely.

[cendrizzi]

Okay. I have some code to reproduce this. Hopefully this is a pretty easy way to do this.

https://github.com/cendrizzi/bull-test

I have instructions in the readme.

Further. I noticed that the issue was that they were processing the same job ids between the two processes, so this must be the issue.

[manast]

Ok, it could be caused by this (already fixed) issue: https://github.com/OptimalBits/bull/issues/670

[manast]

anyhow, I am looking at the nice tests you wrote.

[manast]

your test helped me find several critical bugs, thanks for that! 🥇

[lakshmanvassar]

What are those critical bugs? are those fixed?