Bull: When call queue.close() bull throws Error: Connection is closed.

[mseld]

Description

Handling Graceful Shutdown
Bull throws Error after call queue.close()

Test code to reproduce

// just Press CTRL+C after 1 min while component processing messages
const Queue = require('bull');

let queue = ('mail-queue', 'redis://127.0.0.1:6379');

queue.process(function (job, done) {
     // code ...
});

process.on('SIGINT', function () {
    setTimeout(() => {
        queue.close();
    }, 3000);
});

process.on('SIGTERM', function onSigterm() {
    setTimeout(() => {
        queue.close();
    }, 3000);
});

Bull version

3.3.7&8

Additional information

Error: Connection is closed.
at Redis.sendCommand (/folders/exe/hlr_charging/node_modules/ioredis/lib/redis.js:558:20)
at Redis.zrevrange (/folders/exe/hlr_charging/node_modules/ioredis/lib/commander.js:131:17)
at /folders/exe/hlr_charging/node_modules/bull/lib/getters.js:135:76
at /folders/exe/hlr_charging/node_modules/bull/lib/getters.js:26:18
at arrayMap (/folders/exe/hlr_charging/node_modules/lodash/lodash.js:660:23)
at Function.map (/folders/exe/hlr_charging/node_modules/lodash/lodash.js:9571:14)
at Queue._commandByType (/folders/exe/hlr_charging/node_modules/bull/lib/getters.js:16:14)
at Queue.getRanges (/folders/exe/hlr_charging/node_modules/bull/lib/getters.js:124:30)
at Queue.getJobs (/folders/exe/hlr_charging/node_modules/bull/lib/getters.js:146:17)
at Queue.getFailed (/folders/exe/hlr_charging/node_modules/bull/lib/getters.js:115:17)
at Timeout._onTimeout (/folders/exe/hlr_charging/index.js:83:11)
at ontimeout (timers.js:386:11)
at tryOnTimeout (timers.js:250:5)
at Timer.listOnTimeout (timers.js:214:5)

[manast]

Please complete the issue with a code that reproduces the issue.

[christianbundy]

Looks like OP edited their comment to add repro.

[christianbundy]

@mseld Which version of Redis were/are you using? We're also getting this on Redis 3.2.

cc: @manast

[mseld]

redis 3.2

[christianbundy]

We upgraded to Redis 4 and this problem automatically went away and hasn't come back.

Seems like this may be specific to Redis 3.2.

[derN3rd]

Any news on that for Redis 3.2?
We can't upgrade to Redis 4 and our process keeps crashing when calling queue.close()

EDIT: It seems like it won't crash if there are no active jobs.
We have 10 queues try to stop them like this:

process.on('SIGTERM', () => { // same for sigint
  setTimeout(() => 
    Promise.all([
      q1.close(),
      q2.close(),
      q3.close(),
      q4.close(),
      q5.close(),
      q6.close(),
      q7.close(),
      q8.close(),
      q9.close(),
      q10.close(),
    ]).then(() => process.exit(0))
  , 1000)
})

If some jobs are processing then it will crash otherwise it will shutdown

---

EDIT 2:
Looks like two listeners (one on sigint and one on sigterm) are messing it up.
We use a extra check now:

let isProcessEnding = false

process.on('SIGINT', () => {
  if (!isProcessEnding) {
    isProcessEnding = true
    setTimeout(() => {
      Promise.all([
        q0.close(),
        q1.close(),
        q2.close(),
        q3.close(),
        q4.close(),
        q5.close(),
        q6.close(),
        q7.close(),
        q8.close(),
        q9.close(),
      ]).then(() => {
        console.log('Successfully shut down all queue, because of sigint. Bye!')
        process.exit(0)
      })
    }, 1000)
  }
})

process.on('SIGTERM', () => {
  if (!isProcessEnding) {
    isProcessEnding = true
    setTimeout(
      () =>
        Promise.all([
          q0.close(),
          q1.close(),
          q2.close(),
          q3.close(),
          q4.close(),
          q5.close(),
          q6.close(),
          q7.close(),
          q8.close(),
          q9.close(),
        ]).then(() => {
          console.log(
            'Successfully shut down all queue, because of sigterm. Bye!'
          )
          process.exit(0)
        }),
      1000
    )
  }
})

[manast]

strange that the workaround works, since close() already checks if it is closing and it
if so it does nothing.

[manast]

Actually I see in the stacktrace that the function that produces the exception is getFailed atQueue.getFailed (/folders/exe/hlr_charging/node_modules/bull/lib/getters.js:115:17)
But we never call this function internally, so my guess is that the poster has some timer calling regularly to getFailed when the queue closes the call fails because the connection to redis has already closed. I could add a check that the queue is not closing before executing the method though.

[christianbundy]

If it's helpful, here's our stacktrace from Redis 3.2:

Unhandled rejection Error: Connection is closed.
    at Redis.sendCommand (/usr/src/app/node_modules/ioredis/lib/redis.js:558:20)
    at Script.execute (/usr/src/app/node_modules/ioredis/lib/script.js:24:26)
    at Redis.isFinished (/usr/src/app/node_modules/ioredis/lib/commander.js:164:20)
    at Object.isFinished (/usr/src/app/node_modules/bull/lib/scripts.js:228:29)
    at Timeout._onTimeout (/usr/src/app/node_modules/bull/lib/job.js:432:19)
    at ontimeout (timers.js:427:11)
    at tryOnTimeout (timers.js:289:5)
    at listOnTimeout (timers.js:252:5)
    at Timer.processTimers (timers.js:212:10)

This message repeats indefinitely until you manually close the application.

[manast]

@christianbundy that is a real bug, I have a fix on its way.

[manast]

@mseld Regarding your original bug report, as mentioned above, you probably have a time that is calling getFailed even after the queue has been closed. So this is not a bug in bull.

[aleccool213]

Think this issue can be closed now?

[christianbundy]

Yep, feel free to close!

[mgreenw]

Maybe related: I'm getting a weird race condition when closing my Queues from the jest afterAll() function. It only happens sometimes, and only on a few of the queues. I have tried to trace it in the close() .process but haven't made much progress. Here is the stack I'm getting:

console.error node_modules/jest-jasmine2/build/jasmine/Env.js:289
  Unhandled error

console.error node_modules/jest-jasmine2/build/jasmine/Env.js:290
Error: Connection is closed.
    at close (node_modules/ioredis/built/redis/event_handler.js:102:25)
    at Socket.<anonymous> (node_modules/ioredis/built/redis/event_handler.js:69:20)
    at Object.onceWrapper (events.js:277:13)
    at Socket.emit (events.js:189:13)
    at TCP._handle.close (net.js:597:12)

Any help/suggestions would be appreciated!

[gustawdaniel]

I have the same problem