Bull: Repeatable jobs with different data has same key

[Nicklason]

I have made two repeatable jobs with the same options, but different data. I would expect Bull to handle it as two seperate jobs, but since Bull does not add the same repeatable job if the options are the same, then I can't.

Bull is smart enough not to add the same repeatable job if the repeat options are the same.

What I did:

queue.add({
    test: 1
}, {
    repeat: {
        every: 1000
    }
});

queue.add({
    test: 2
}, {
    repeat: {
        every: 1000
    }
});

I think it should be handled as two different jobs, because of the different data. I can force Bull to handle them seperately by changing the jobId, because that changes the key of the job in the redis db.

queue.add({
    test: 1
}, {
    repeat: {
        every: 1000
    },
    jobId: "test1"
});

queue.add({
    test: 2
}, {
    repeat: {
        every: 1000
    },
    jobId: "test2"
});

[stansv]

This is works by design, if you don't need to protect from repeatable job duplicates you set custom job ID. But I know this is not well documented yet..

[Nicklason]

Then there should be a setting or something that we can use to protect against duplicate jobs, or Bull should handle them as two seperate jobs because of the different data.

[r3wt]

This is works by design

you can protect from duplicates with a simple sha1 of the jobdata and compare. if its true you did this by design, then your design is bad

[leiwssxl]

queue.add('newJobName',Data,opts).then(function (job) {
console.log(job);
})

queue.process("*",(job, done) => {
// * will handle all jobs
console.log("job " + JSON.stringify(job));
console.log("###########################");
done();
done(Error('error transcoding'));
throw (Error('some unexpected error'));
});
then the redis will have 2 diff repeated job

[Nicklason]

@leiwssxl your example makes no sense what so ever. You are not even making two jobs.

I did try and make two jobs and adding identical names to them, kept the options the same. As I expected, it had the same result as if there was no name. If you have different job names, then it will make different jobs just like if you had different job ids, but then I don't see the reason to go through the trouble of using the job names like you do.

[TheAdamGalloway]

Hi guys, glad I came across this thread as I was really confused by two repeatable jobs not running.

However, could somebody please advice how I would go about removing just one repeatable job? They both seem to have the same key and same repeat options...

Thanks!

[Nicklason]

Hello @TheAdamGalloway

What I did was that I ended up using delays instead of the repeatable option and then have a seperate script that would listen for those jobs to finish, and then add it again with same options. This meant that I wouldn't have to worry about the problems that I faced by having jobs that had the same options.

[TheAdamGalloway]

@Nicklason Do you know if there is any way to retrieve the repeatable job key when running Queue.add? I would like to store it for removing later, but can't seem to find any way of doing so. Thanks

[TheAdamGalloway]

I want to use repeat rather than delays as it will automatically recover the scripts in case the server goes down

[Nicklason]

I want to use repeat rather than delays as it will automatically recover the scripts in case the server goes down

That's why I have a serperate script listen for when the jobs finish or fail and add them back to the queue.

[Nicklason]

queue.add returns a job, the key is stored in the options of the job if I remember correctly.

[TheAdamGalloway]

@Nicklason
I am currently running an application that creates two jobs with different jobIds set in the job options, both run every second. When I call job.toKey() after the job has been returned, it always outputs bull:surveys-1: (Redis prefix followed by queue name, no ID).

Also, job.id gives an ID that is unique to the job instance rather than the repeatable (e.g. repeat:1f04d7e894064acc11a27bf90b0f9e81:1573136133000) so I can't use this to remove the repeatable in future.

[Nicklason]

I found this in the documentation: https://github.com/OptimalBits/bull/blob/master/REFERENCE.md#queueremoverepeatable

[TheAdamGalloway]

I found this in the documentation: https://github.com/OptimalBits/bull/blob/master/REFERENCE.md#queueremoverepeatable

Yes I'm looking to use this, but it currently kills all repeatable jobs that have the same schedule (which is not useful if I have 2 jobs on the same schedule)

Thanks for your help!

It seems that the repetable IDs are generated like this name+':'+jobID+':::'+cron so I should be able to manually generate them when I run the function to create the jobs, then store this value for removing the repeatable later on. It feels like a bit of a workaround, but should work fine I guess.

[Nicklason]

Then this maybe? :) https://github.com/OptimalBits/bull/blob/master/REFERENCE.md#queueremoverepeatablebykey