Bull: Change schedule for the repeatable job

[tomas-berg]

Team,

Here is the real world scenario:
We have one queue and and repeatable cron-like job, for example, repeat every 5 seconds:

    await queue.add({}, {
        removeOnComplete: true,
        removeOnFail: true,
        repeat: {
            every: 5000
        }
    });

Let's pretend we need to restart our app and change repeat for 10 seconds.
It suggested to remove repeatable jobs by getRepeatableJobs() and remove them by removeRepeatableByKey().

But in fact, previously scheduled job will still be executed one time before the new job schedule will take place.

Nor queue.clean(0) or queue.empty() doesn't help.

It's definitely very confusing and NOT expected scenario.

Is there a right way to do that?

[manast]

If the previous repeatable job was removed, then it should not schedule new jobs after that, if so it is a bug. Can you prepare a minimal case that reproduces the issue?

[tomas-berg]

@manast
Thanks for the quick reply.

I've put some minimal one-file sample here

If you check the code below and run it first time, then stop and change job's data and repeat option and run it second time - the first job will be executed once again. How to achieve case when previously scheduled job will not be executed?

'use strict';

const Bull = require('bull');

const proc = function (job, done) {
    try {
        const { data } = job;
        console.log(`Job data: [${data.payload}]`);
        done();
    } catch (e) {
        console.error(e.message);
        done(e);
    }
};

const queue = new Bull('test', {
    redis: {
        host: '0.0.0.0',
        port: 6379,
        db: 0,
        password: 'secret'
    }
});

(async () => {
    const repeatable = await queue.getRepeatableJobs();

    repeatable.forEach(async (job) => {
        await queue.removeRepeatableByKey(job.key);
    });

    queue.process(proc);
    await queue.add({
        payload: 'j0' // <<< change it next run
    }, {
        removeOnComplete: true,
        removeOnFail: true,
        repeat: {
            every: 1000 // <<< change it next run
        }
    });
})();

Thanks a lot.

[manast]

not sure if it affects the outcome, but in the code above it will add the repeatable job BEFORE you have removed the old jobs. Try to use Promise.all and map instead:

    await Promise.all(repeatable.map((job) => queue.removeRepeatableByKey(job.key)));

[tomas-berg]

@manast
Just tried. The result is just the same. After second run - the old scheduled job executes one time.

[manast]

can you post the updated code?

[tomas-berg]

@manast
sure, please check latest sample

[manast]

this happens because even though you deleted the repeatable job, the next programmed job is still in the delay queue waiting to be executed, a quickaround would be to clean the delayed jobs after removing the repeatable job, if you change the code to this one it will work:

(async () => {
    const repeatable = await queue.getRepeatableJobs();

    await Promise.all(repeatable.map((job) => queue.removeRepeatableByKey(job.key)));

    // repeatable.forEach(async (job) => {
    //     await queue.removeRepeatableByKey(job.key);
    // });

    await queue.clean(0, 'delayed');

    queue.process(proc);
    await queue.add({
        payload: 'j0' // <<< change it next run
    }, {
        removeOnComplete: true,
        removeOnFail: true,
        repeat: {
            every: 1000 // <<< change it next run
        }
    });
})();

but yes, it may be an overlook from my side and I should remove the delayed job associated to the repeatable job when removing the repeatable job. Will mark this a bug.
2

[imsergiobernal]

this happens because even though you deleted the repeatable job, the next programmed job is still in the delay queue waiting to be executed, a quickaround would be to clean the delayed jobs after removing the repeatable job, if you change the code to this one it will work:

(async () => {
    const repeatable = await queue.getRepeatableJobs();

    await Promise.all(repeatable.map((job) => queue.removeRepeatableByKey(job.key)));

    // repeatable.forEach(async (job) => {
    //     await queue.removeRepeatableByKey(job.key);
    // });

    await queue.clean(0, 'delayed');

    queue.process(proc);
    await queue.add({
        payload: 'j0' // <<< change it next run
    }, {
        removeOnComplete: true,
        removeOnFail: true,
        repeat: {
            every: 1000 // <<< change it next run
        }
    });
})();

but yes, it may be an overlook from my side and I should remove the delayed job associated to the repeatable job when removing the repeatable job. Will mark this a bug.
2

What if I want to remove just the delayed job from an specific repeatable job?

[sundowndev]

@imsergiobernal We have the same issue and ended up with this solution :

• Create the job without repeat option

await queue.add({ ... });

• Inside the job, create a duplicate job in the queue with a delay

await job.queue.add(job.data, {
    delay: 5000, // clone this job and execute it in 5 seconds
});

Just return before this statement to prevent another job to be created. So we can stop repeating the job only when needed.

This might be an anti-pattern, but that's the only solution I've come across this far. We needed to prevent this particular job from repeating again, not all jobs from the queue.