Bull: Repeatable jobs creates a new ID after each execution

[sriramr98]

Description

When a repeatable job is scheduled with a cron, we get an ID with the pattern repeat:. This ID changes every time the job gets processed regardless of it being a success or failure.
Moreover, if we pass a custom ID to jobOpts when creating the job, the ID for the job is not being replaced but still getting the default repeat: formatted ID.

Minimal, Working Test code to reproduce the issue.

Just a create a new repeatable job and observe the ID using UI

Bull version - 3.14.0

Additional information

I'm using bull to allow users to schedule jobs through an API and cancel them as required. Here, canceling repeating jobs would be problematic because since the ID keeps changing, I can't even find the job using queue.getJob as it returns null. I can't return the key of the repeatable job which is required to remove a repeatable job because the key is not sent back by the API when it's added to the queue.

[tvedtorama]

I'm having a similar problem. I create the repeatable job with an id, and then find a regular job with another id in the queue.

The job in the queue seems to be created from the repeat-spec each time the repeat is processed. Then the payload etc is copied from the last instance into the new.

The repeatable job, found with getRepeatableJobs holds our id, but not the payload.

I try to correlate the two, by reading the jobId from the regular job like this:

(<any>job.opts?.repeat)?.jobId

Then I look for a repeatable job with that ID. If you supplied the jobId when creating the job, this will the jobId in question.

One problem is that the jobId on the regular job is not part of the Typescript definition, leading to a worry that it is undocumented and potentially removed in the future - leaving us without options to correlate.

[StephRicardo]

I am also experiencing this issue, and it has made it challenging to remove repeatable jobs without emptying the queue.

[sriramr98]

The workaround I figured out to clear out jobs is something like this.

When I create a job, depending on the job type, I add a prefix repeated:: or delayed::. You can pass in this ID when creating a Job in JobOpts (This still won't override the autogenerated repeatable job ID. I'll get to that).

When you need to cancel a job if it's a repeatable job (Remember you added repeated:: prefix), you can follow this step.

1. Get All repeatable jobs from the queue.
2. Loop over all jobs and find a job who's key contains the jobId that you created.
3. Cancel that job with the key of the job

    const repeatableJobs = await scheduledQueue.getRepeatableJobs();
    const jobWithId = repeatableJobs.filter(job => job.key.includes(jobId))[0];
    if (jobWithId) scheduledQueue.removeRepeatableByKey(jobWithId.key);

Note that this method will only work if you create a custom ID and pass it in JobOpts when the job is created. It doesn't need the repeated:: prefix. It's just something that I added to easily differentiate delayed jobs from repeating jobs.

I guess bull isn't able to maintain a constant ID for the repeated job because of how bull interacts hiwith Redis.
I hope this solution works for you guys.

[rubenvereecken]

I have the same issue and while the proposed solution would work, I really wouldn't want to iterate over a massive amount of jobs.

Especially because I'm actually using repeatable jobs for one-off tasks because it's a lot easier to schedule repeatable jobs due to the "startTime" field. I suppose I'll stop using repeatable jobs though...

EDIT: You might be able to use Queue#removeRepeatableByKey but I haven't been able to get it to work because I'm not sure what key is.

[manast]

@rubenvereecken first you can get the repeatable jobs with https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queuegetrepeatablejobs and then remove them using Queue#removeRepeatableByKey

[rubenvereecken]

@manast but that's just another O(n) solution isn't it? Whereas presumably something that "gets by key" is O(1)? Though I suppose that's where my lack of Redis knowledge shows because it might well also be O(n) anyway, in which case yeah, might as well.

TODO: Even worse, it turns out getRepeatableJobs gets me all jobs for that queue, even the ones that have already been removed. Or I assume as much, because I have no delayed/active/waiting jobs yet got a long list of results.

[manast]

I think it only returns you the jobs that have not been deleted.