Bull: Empty and clean jobs

[nanom1t]

Hi,

I'm using bull in my Node.js application. It works good, but sometimes appears issues with repeated jobs especially on application restarts. In my case one or more repeated job don't runs. I've found that I can fix this issue by clearing Redis (flushdb) and restarting application. Then everything works good.

So I've decided to clean all queues on application start. I've found emply and clean methods in documentation. However, it is not clear for me how do they work with Redis. Should they clear Redis database after execution?

I've tried emply / clean methods and they don't clean my Redis db. So I have duplicates:

127.0.0.1:6379[1]> keys *
 1) "bull:vip:repeat"
 2) "bull:notify:repeat:vk-scheduler:notify-scheduler:1506930900000"
 3) "bull:notify:repeat:vk-scheduler:notify-scheduler:1506930600000"
 4) "bull:vip:repeat:scheduler:vip-scheduler:1506942000000"
 5) "bull:vip:repeat:scheduler:vip-scheduler:1506930828000"
 6) "bull:vip:id"
 7) "bull:notify:id"
 8) "bull:notify:repeat"
 9) "bull:notify:repeat:cleaner:notify-cleaner:1506996000000"
10) "bull:notify:repeat:vk-scheduler:notify-scheduler:1506930300000"
11) "bull:vip:repeat:scheduler:vip-scheduler:1506930880000"

In this case vip-scheduler should run two times per day (0 0 2,12 * * *), but there are duplicates in Redis. So I'm not sure how many times it will be executed. Does it matter what stores in Redis database?

queue.add('scheduler', {}, { jobId: 'vip-scheduler', repeat: { cron: '0 0 2,12 * * *' }, removeOnComplete: true, removeOnFail: true });
queue.process('scheduler', vipWorker.scheduler);

Thanks

[manast]

which version of bull are you using?

[nanom1t]

^3.0.0-rc.4

[manast]

ok. Please check with latest version 3.2, there are several issues fixed since that release candidate related to repeatable jobs.

[nanom1t]

Ok. I'll try new version. Thank you!

[manast]

Did it work as expected with the latest version? if so please close this issue.

[nanom1t]

I've tried v3.2.0 and it seems that this issue is not fixed. Also I've tried to reproduce this issue.

I'm using empty method to clean the queues before initialization. So I'm expecting that Redis database will be cleaned before queue initialization.

var queue = new Queue('vip', { redis: redisConfig });
queue.empty().then(function () { 
    queue.add('scheduler', {}, { jobId: 'vip-scheduler', repeat: { cron: '0 0 2,12 * * *' }, removeOnComplete: true, removeOnFail: true });
    queue.process('scheduler', vipWorker.scheduler);

    resolve(queue);
}).catch(function (err) {
     reject(err);
});

Let's see my Redis database after first initialization:

1) "bull:vip:repeat:scheduler:vip-scheduler:1507338000000"
2) "bull:vip:repeat"
3) "bull:vip:id"

Everything works as expected. Then I've changed the cron schedule for this job to '*/5 0 2,12 * * *', restarted application and got next results:

127.0.0.1:6379[1]> keys *
 1) "bull:vip:stalled-check"
 2) "bull:vip:repeat:scheduler:vip-scheduler:1507338000000"
 3) "bull:vip:repeat"
 4) "bull:vip:delayed"
 5) "bull:vip:id"
 6) "bull:vip:repeat:scheduler:vip-scheduler:1507633200000"

Now I have two repeated jobs. So I can suppose that the queue.empty() does not clear Redis db as expected.

Let's just clear queues without adding new jobs or processing:

var queue = new Queue('vip', { redis: redisConfig });
queue.empty().then(function () {    
    resolve(queue);
}).catch(function (err) {
     reject(err);
});

The empty method didn't clear old jobs in queue:

127.0.0.1:6379[1]> keys *
1) "bull:vip:repeat:scheduler:vip-scheduler:1507338000000"
2) "bull:vip:repeat"
3) "bull:vip:id"
4) "bull:vip:repeat:scheduler:vip-scheduler:1507633200000"

Thank you.

[brianlovin]

Subscribing here - I've been wondering how to approach this same issue for a while now, as every application restart kept inserting duplicate repeat jobs into redis. Thanks for looking into this!

[weeco]

Same issue here, haven't figured out exactly what's going on there, but I will try to provide more details once I can. Very annoying.

PS: I am using Bull 3.3.0

[derN3rd]

With version 3.3.6 I can't clear and re-create "cronjobs" on fresh deployments. At the first start it works, but after a restart no new job is added.
Am I doing something wrong or is this the same problem?

const cronjobQueue = new BullQueue('cronjobs', {
  redis: { db: 1, host: process.env.redisip },
})

cronjobQueue.process('newIntervalStats', () =>
  Stats.newIntervalStats()
    .catch(logError)
)
cronjobQueue.process('newTotalStats', () =>
  Stats.newTotalStats()
    .then(() => Stats.clearIntervalStats())
    .catch(logError)
)

/* http for bull-arena and other stuff */
http.listen(8081, () => {
  console.log(`Worker ${version} online on port 8081.`)

  cronjobQueue
    .empty()
    .then(() =>
      Promise.all([
        cronjobQueue.add(
          'newIntervalStats',
          {},
          {
            removeOnComplete: true,
            repeat: {
              cron: '*/1 * * * *',
              tz: 'Europe/Berlin',
            },
          }
        ),
        cronjobQueue.add(
          'newTotalStats',
          {},
          {
            removeOnComplete: true,
            repeat: {
              cron: '0 0 3 * * *',
              tz: 'Europe/Berlin',
            },
          }
        ),
        /* more jobs were here, but I removed them, because the look nearly the same */
      ]).then(() => console.log('Cleaned old and added new cronjobs'))
    )
    .catch(logError)
})

EDIT:
As I can see in the bull-arena web interface, there is no job delayed.
Redis shows the following result (I'm adding 5 jobs in total, but I posted only 2 in my example above):

127.0.0.1:6379[1]> keys bull:cronjobs:*
 1) "bull:cronjobs:repeat:4d8c9fcc2f6327db0e3990c5d803d279:1510538400000"
 2) "bull:cronjobs:repeat:3c0174fdc14b3ad66d5d2eb066a75e8d:1510513260000"
 3) "bull:cronjobs:repeat:3c0174fdc14b3ad66d5d2eb066a75e8d:1510513020000"
 4) "bull:cronjobs:repeat:49cb7983355a7bf58443adefbb109445:1510538400000"
 5) "bull:cronjobs:repeat:08a47f839cdc7ad18d9e7250bfd0d3c0:1510527630000"
 6) "bull:cronjobs:repeat:3c0174fdc14b3ad66d5d2eb066a75e8d:1510514220000"
 7) "bull:cronjobs:repeat:3c0174fdc14b3ad66d5d2eb066a75e8d:1510516200000"
 8) "bull:cronjobs:stalled-check"
 9) "bull:cronjobs:repeat:a14cca50559d5e8b54a2f4a66ac49127:1510538400000"
10) "bull:cronjobs:repeat:4f2428ccb6f2afc073046178c94bb10c:1510512900000"
11) "bull:cronjobs:repeat:3c0174fdc14b3ad66d5d2eb066a75e8d:1510514280000"
12) "bull:cronjobs:id"
13) "bull:cronjobs:repeat"

[manast]

I'll try tomorrow, lets see what I can find...

[derN3rd]

I made a workaround that I'm using in production right now, because I didn't get it to work in any other way:

const redis = require('redis')
const redisClient = redis.createClient({ host: REDIS_IP, db: REDIS_DB })

const completelyCleanQueue = queueName =>
  new Promise((resolve, reject) => {
    redisClient.scan(0, 'MATCH', `bull:${queueName}:*`, (err, data) => {
      if (err) return reject(err)
      if (data[1]) {
        return Promise.all(
          data[1].map(
            entry =>
              new Promise((resolve, reject) =>
                redisClient.del(entry, (err, data) => {
                  if (err) return reject(err)
                  return resolve(data)
                })
              )
          )
        )
      }
      return resolve()
    })
  })

use with caution, I'm not that familiar with redis.

I'm using it in the following way:

myQueue.empty()
.then(() => completelyCleanQueue('myQueueName'))
.then(() => doWhatEverYouWant)
.catch(err => logThatError)

[Freundschaft]

I looked into this and the reason is that queue.empty doesnt actually clear the jobs that are in the queues.
It only clears jobs in the wait and paused queue. (https://github.com/OptimalBits/bull/blob/master/lib/queue.js#L648)
The repeat jobs are stored with a repeat prefix, and in order to clear them, each of them needs to be deleted one by one.
In order to properly work it should run something like the imlpementation from @derN3rd

[manast]

yeah. empty just cleans the "queue", not all the jobs that have been processed by the queue :).
Maybe it should be called differently to avoid confusion, "discard" or something like that.

[Freundschaft]

a solution I used that doesnt involve adding another library is this:

queue.clean(0, 'delayed');
queue.clean(0, 'wait');
queue.clean(0, 'active');
queue.clean(0, 'completed');
queue.clean(0, 'failed');

let multi = queue.multi();
multi.del(queue.toKey('repeat'));
multi.exec();

[Sicria]

@Freundschaft Since queue.clean returns a promise it might be best to add an await for all of those to avoid any issues where you're cleaning anything that might have just been set to run.

[derN3rd]

@Sicria or simply use Promise.all([queue.clean(),...])

[denizdogan]

In my opinion, Queue#empty should actually remove all the entries, including completed, failed, etc. If this is not desirable, the documentation should be updated to reflect this. It currently says:

Empties a queue deleting all the input lists and associated jobs.

The terminology is a bit confusing to me, what is an input list?

[manast]

Thats lists containing jobs that have not yet been processed.

[herrri]

My method of emptying the queue:

const getKeys = async (q) => {
  const multi = q.multi();
  multi.keys('*');
  const keys = await multi.exec();
  return keys[0][1]
}

const filterQueueKeys = (q, keys) => {
  const prefix = `${q.keyPrefix}:${q.name}`;
  return keys.filter(k => k.includes(prefix));
}

const deleteKeys = async (q, keys) => {
  const multi = q.multi();
  keys.forEach(k => multi.del(k));
  await multi.exec();
}

const emptyQueue = async (q) => {  
  const keys = await getKeys(q);
  const queueKeys = filterQueueKeys(q, keys);
  await deleteKeys(q, queueKeys);
}

const queue = new Queue('my-queue');
emptyQueue(queue).then( /* queue now empty */ )

[Freundschaft]

Any reason this isnt put in the library itself and exposed via a simple function?

[pariola]

still no fix?, cause on clean everything stucks

[PierreAndreis]

Redis is using over 1gb of ram because bull keeps the "data" and there is no way to clean up old jobs...

[mrbatista]

@manast any update? This is a critial bug...

[CaptainYarb]

This issue is nearing a year old. Any resolution in sight?

[manast]

Ok, so I am trying to make sense of this issue because in this thread there is a mixture of different issues and there are also false expectations of some apis. I will try to clarify:
queue.empty empties the "queue", meaning that all jobs that are waiting to be processed are discarded, maybe the name is not a very good one, but that is that it does. It is not a wipe all kind of thing.
queue.clean removes jobs from a given "status", for example you can remove all the jobs that are completed, failed, delayed, etc. It would be possible to implement empty with clean by calling clean several times on the "status" a job can be when waiting to be processed, such as "wait", "paused", "delayed", "priority".

Finally, repeatable jobs are a special type of job that creates an entry in the "repeat" zset. As long as an entry representing a given repeatable job is in this set, the job will repeat according to its cron values, so in order to remove it you need to use the queue.removeRepeatable method, as stated in the documentation.

If the same repeatable job is added several times it should result in a noop, otherwise it is a bug. In the version of bull at the time of this writing (3.4.7) there is no know issue regarding removing repeatable jobs.

I am willing to improve the empty and clean functionalities, it would require a major version though.

I will close this thread for now and if based on the information above you find an inconsistent behaviour please open a new issue and I will work on it as soon as possible.

[sainipradeep]

a solution I used that doesnt involve adding another library is this:

queue.clean(0, 'delayed');
queue.clean(0, 'wait');
queue.clean(0, 'active');
queue.clean(0, 'completed');
queue.clean(0, 'failed');

let multi = queue.multi();
multi.del(queue.toKey('repeat'));
multi.exec();

How to clear selected named queue ?

[yundy0605]

You can try this

const Queue = require('bull')
const testQueue = new Queue('test')
testQueue.clean(0, 'delayed')

But I use the library from bull document
The code below I sure that it can work (check and clear the expired data for over 5 sec every 5 sec)

const Queue = require('bull')
const testQueue = new Queue('test')
setInterval(() => {
  testQueue.clean(5000)
}, 5000);

[atwright147]

How about queue.clear() to clear out all jobs from a queue?