Bull: Missing lock for job failed when closing queue and job takes more than 30 seconds to complete [BUG]

[misos1]

Description

Seems job which takes more than 30 seconds to complete is not properly finished when is queue closing and in next runs is announces as stalled. Additionally it takes another 5 seconds to gracefully exit after is queue closed which does not happen when job takes less than 30 seconds to complete.

Minimal, Working Test code to reproduce the issue.

``` node js
let Promise = require("bluebird");
let Queue = require("bull");
let queue = new Queue("queue");
queue.process(10, _job => Promise.delay(40000));
queue.on("error", (...errs) => console.log(new Date(), "queue error:", ...errs.reverse()));
queue.on("stalled", job => console.log(new Date(), "stalled", job.id));
(async function()
{
let job = await queue.add({});
console.log(new Date(), "added", job.id);
await Promise.delay(1000);
await queue.close();
console.log(new Date(), "closed");
})();
process.on("exit", () => console.log(new Date(), "exiting"));

Possible output after few runs:

2019-08-04T21:14:37.362Z added 1
2019-08-04T21:15:17.379Z queue error: Error processing job Error: Missing lock for job 1 failed
at Object.finishedErrors (node_modules/bull/lib/scripts.js:182:16)
at node_modules/bull/lib/job.js:268:31
at processTicksAndRejections (internal/process/task_queues.js:82:5)
2019-08-04T21:15:17.382Z closed
2019-08-04T21:15:22.382Z exiting

2019-08-04T21:15:24.398Z added 2
2019-08-04T21:16:04.414Z queue error: Error processing job Error: Missing lock for job 2 failed
at Object.finishedErrors (node_modules/bull/lib/scripts.js:182:16)
at node_modules/bull/lib/job.js:268:31
at processTicksAndRejections (internal/process/task_queues.js:82:5)
2019-08-04T21:16:04.416Z closed
2019-08-04T21:16:09.421Z exiting

2019-08-04T21:16:11.091Z added 3
2019-08-04T21:16:11.096Z stalled 1
2019-08-04T21:16:51.106Z queue error: Error processing job Error: Missing lock for job 1 failed
at Object.finishedErrors (node_modules/bull/lib/scripts.js:182:16)
at node_modules/bull/lib/job.js:268:31
at processTicksAndRejections (internal/process/task_queues.js:82:5)
2019-08-04T21:16:51.107Z queue error: Error processing job Error: Missing lock for job 3 failed
at Object.finishedErrors (node_modules/bull/lib/scripts.js:182:16)
at node_modules/bull/lib/job.js:268:31
at processTicksAndRejections (internal/process/task_queues.js:82:5)
2019-08-04T21:16:51.108Z closed
2019-08-04T21:16:56.108Z exiting

## Bull version
3.10.0
## Additional information
Nothing which I described happens if I change it to 20 seconds:
``` node js
queue.process(10, _job => Promise.delay(20000));

2019-08-04T21:29:58.423Z added 1
2019-08-04T21:30:18.439Z closed
2019-08-04T21:30:18.439Z exiting



md5-1e2cdc017b1d302673c4b89f47ae0f82



2019-08-04T21:30:23.516Z added 2
2019-08-04T21:30:43.531Z closed
2019-08-04T21:30:43.532Z exiting



md5-1e2cdc017b1d302673c4b89f47ae0f82



2019-08-04T21:30:45.734Z added 3
2019-08-04T21:31:05.750Z closed
2019-08-04T21:31:05.750Z exiting

[guidodizi]

Similar case here

[inn0vative1]

I am having a similar issue here. In my case jobs do not even take 30 seconds. I have to types of jobs, one completes very quickly (milliseconds) and another takes a bit longer (still relatively short, a second or two). I find that this error gets thrown more often with the longer running jobs.
The job seems to complete successfully, but this error occurs when the on('complete') callback is called.
Missing lock for job fecf814b-8f54-44bf-ab76-6992f11048ac delayed at Object.finishedErrors (node_modules/bull/lib/scripts.js:182:16) at multi.exec.then.results (node_modules/bull/lib/job.js:293:31) at process._tickCallback (internal/process/next_tick.js:68:7)

[Fossil01]

Same issue.

[guidodizi]

My issue was actually coming from not returning an object on the processing of each job queue. So by returning an object, even an empty object, i manage to avoid the Missing lock for job error.

Hope it helps!

[misos1]

@guidodizi maybe you could create another issue as it does not make much sense to throw such an error in case you are not returning object.

[yan-michael]

Same issue here

[yan-michael]

My issue was actually coming from not returning an object on the processing of each job queue. So by returning an object, even an empty object, i manage to avoid the Missing lock for job error.

Hope it helps!

C

My issue was actually coming from not returning an object on the processing of each job queue. So by returning an object, even an empty object, i manage to avoid the Missing lock for job error.

Hope it helps!

Hey! Can you elaborate on this?

[rj-david]

We are currently experiencing this now when deploying so we need to gracefully close the existing queue

[BogdanStanciu]

Same issue. I return a empty object in process method but it always return a Missing lock for job {id} finished

[cpu100]

What's the reason? Is redis overloaded?

[harshamv]

Has anyone solved this? Please share your solution.

[cpu100]

updated to the newest bull version. seems it solved.

[kodeine]

any updates on this i am on new version but getting the same error.

[cpu100]

@kodeine
Did you notice the CPU load when this error was thrown? My solution is to read /proc/loadavg in the repeat process callback to check whether the current CPU load is too high, if so, return the callback function. Now my problem is completely solved.