Bull: jobID + repeat

[agustik]

Description

• I can create jobs with custon jobId's
• I can create jobs with "cron"
• BUT I cannot create jobs with special jobID AND repeat on the same job

Minimal, Working Test code to reproduce the issue.

const Queue = require('bull');
const queue = new Queue('test');
const crypto = require('crypto');
function randomString(){
  return crypto.randomBytes(10).toString('hex');
}

async function wrapper(){
  let id = "";
  const data = {
    foo : 'bar',
    date : new Date()
  }

  id = 'My-Special-JOBID--' + randomString();
  const job1 = await queue.add('Adding-with-custom-id', data, {
    jobId : id
  });
  console.log({ name : job1.name, requested : id, got : job1.id, same : job1.id === id })

  id = 'My-Special-JOBID--' + randomString();
  const job2 = await queue.add('Adding-with-custom-id-AND-cron', data, {
    jobId : id,
    repeat : {
      cron : '10 * * * *'
    }
  });

  console.log({ name : job2.name, requested : id,  got : job2.id,  same : job2.id === id })

}
wrapper();

results:

{ name: 'Adding-with-custom-id',
  requested: 'My-Special-JOBID--90b41e8a0a88aa38e45c',
  got: 'My-Special-JOBID--90b41e8a0a88aa38e45c',
  same: true }
{ name: 'Adding-with-custom-id-AND-cron',
  requested: 'My-Special-JOBID--4456d6817cba1fec2986',
  got: 'repeat:62d7dfa8218765a12846082b3a32d28f:1527556200000',
  same: false }

Bull version

3.3.9

Additional information

Is this a expected behavior?

I can of course just loop over all jobs and search for the item .. but that takes a long time :)

[manast]

Mixing normal jobs with jobIds and repeatable jobs is not supported, we have not even considered that case, not sure how difficult it would be to implement though.

[agustik]

Thanks for quick answer. I'll refactor my code and implement a workaround :)

[moltar]

I noticed that, for some reason, the jobId becomes part of the repeat config.

"opts": {
  "repeat": {
    "cron": "3 * * * *",
    "jobId": "cron",
    "count": 1
  },
  ...

I think this is a much needed feature, because otherwise changing the cron schedule creates more jobs rather than update the existing ones. I think the queue should update the job instead.

[chanphillip]

Does it still work? Neither of these works for me. The job ID is still auto generated.

"opts": {
  "repeat": {
    "cron": "3 * * * *",
    "jobId": "cron"
  }
}

or

"opts": {
  "jobId": "cron"
  "repeat": {
    "cron": "3 * * * *",
  }
}

[manast]

The jobId becomes part of the Id, it is not possible to have the same ID for all the repeatable jobs, since every repetition is a new job.

[chanphillip]

@manast Got it.

But what is the best way to maintain repeatable jobs, in case of changing the cron of the repeatable jobs that are already in queues? Right now the original job will be kept if cron has changed.

The only way I can think of is searching the queue for a defined name in data before adding the job, and remove it if cron has changed.

[manast]

why can't you use queue.removeRepeatable() `

[manast]

we do not support update yet, so you will need to remove and create a new one for now.

[chanphillip]

@manast OK. I'll check it out. Thanks!

[stevenolay]

I want to dynamically update the cron period on a running repeat job. My intention is to give a "repeat" job an ID and use a heuristic to determine when the next time the job should run and update the "repeat" job if necessary with a new cron period.

Since my cron duration is calculated dynamically there is not a way for me to provide identical repeat ops to queue.removeRepeatable().

How would I go about removing the repeat job if I don't have the exact settings?

[stevenolay]

I did some playing around. When you supply a jobId to the options like so:
`

const jobId = "my job id";

const data {
  key: "val"
};

const jobOptions: JobOptions = {
  jobId,
  repeat: {
    cron: `*/2 * * * *`,
    tz: 'America/Los_Angeles'
  }
};

chronQueue.add(data, jobOptions);

`

As long as the jobId remains the same it maps to the same key in redis. The key in redis is not my provided jobID but an additional job doesn't get created unless the jobId is changed. Would it be save to assume that whatever key generation algorithm being used by bull is using my jobId in the calculations?

The key in redis does not change if i make any changes to the cron string. Only if I make changes to the jobId.

[sarneeh]

I came here with the same problem. I'd like to use repeatable jobs (add some tasks every X), but use jobId to ensure that these jobs don't overlap (in a case when the job processing takes longer than the repeat interval). Unfortunately, setting a jobId doesn't work and repeat keeps adding jobs anyways.

I've also noticed that the jobId lands in repeat instead of the jobOpts (like @moltar mentioned).

Is it the desired behavior? The jobId documentation doesn't mention that it doesn't work properly with repeat or anything like that:

// jobId: number | string; // Override the job ID - by default, the job ID is a unique
// integer, but you can use this setting to override it.
// If you use this option, it is up to you to ensure the
// jobId is unique. If you attempt to add a job with an id that
// already exists, it will not be added.

[stevenolay]

So repeat jobs have their own formula for constructing a job ID(key).

It is constructed using the job ID you specify and cron repeat options.

Since The job id you set does get factored into the generation of that job
ID. It should be effectively unique.

If you provide the same job ID and different repeat options however, those
two jobs will be treated as independent jobs.

What I had to do to mitigate this issue was keep track of the last known
repeat key generated when my repeat job was instantiated.

If I wanted to target that job I could use getRepeatJobByKey using the
previously generated key. Kill that job. And then generate a new one.

It’s hacky. But the only way to do it with bull.

On Thu, May 30, 2019 at 4:29 PM Jacob Sarnowski [e-mail]
wrote:

I came here with the same problem. I'd like to use repeatable jobs (add
some tasks every X), but use jobId to ensure that these jobs don't
overlap (in a case when the job processing takes longer than the repeat
interval). Unfortunately, setting a jobId doesn't work and repeat keeps
adding jobs anyways.

I've also noticed that the jobId lands in repeat instead of the jobOpts
(like @moltar https://github.com/moltar mentioned).

Is it the desired behavior? The jobId documentation doesn't mention that
it doesn't work properly with repeat or anything like that:

// jobId: number | string; // Override the job ID - by default, the job ID
is a unique
// integer, but you can use this setting to override it.
// If you use this option, it is up to you to ensure the
// jobId is unique. If you attempt to add a job with an id that
// already exists, it will not be added.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/OptimalBits/bull/issues/961?email_source=notifications&email_token=ACLR6YCVTCLVVH6AZIYWPYDPYA2LNA5CNFSM4FCC2DP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWTNYQA#issuecomment-497474624,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACLR6YDOCEAB7B2ENHYEY3DPYA2LNANCNFSM4FCC2DPQ
.

>

Steven O'Brian Layne
Northwestern University 2017
McCormick School of Engineering and Applied Science
Computer Science

[manast]

@sarneeh so the repeatable jobs wont overlap, this should be clarified in the documentation. Until the current repeatable job has not been completed, no new one is scheduled.

[sarneeh]

Until the current repeatable job has not been completed, no new one is scheduled.

@manast But it doesn't work like this. Check this out:

queue.process('test job 1', 10, async job => {
  logger.info('test job 1 started');
  await new Promise(resolve => setTimeout(resolve, 30000));
});

queue.add('test job 1', undefined, {
  jobId: 'unique job id',
  repeat: { every: 1000 }
});

The processor waits 30 seconds and then finishes. But the .add keeps adding jobs until reaching the concurrency (10) limit. As long as you have >1 processors in your queue or one processor with >1 concurrency, the jobs will be overlapping.

[manast]

@sarneeh yes you are completely right, I actually had to check the code because I could not believe it myself. I guess I do not know anymore how my own code works :D.
Having said that, the jobId approach would not work either because even if the repeatable jobs received an unique jobId, then only the first iteration would work since there will exist an old job with the same ID (specially in the overlap case which is the one you want to avoid to begin with).

[sarneeh]

@manast No problem :smile: Just wanted to ensure that I understand this correctly. Thanks for clarifying this up.

[moltar]

@manast:

Is this fixed yet or docs are wrong?

yes you are completely right, I actually had to check the code because I could not believe it myself. I guess I do not know anymore how my own code works :D.

From the docs (very bottom of the screen):

• Bull is smart enough not to add the same repeatable job if the repeat options are the same.
• If there are no workers running, repeatable jobs will not accumulate next time a worker is online.

[oneEyedSunday]

I really would like to have this feature, but I recognize why it may not be feasible. I wonder though if @stevenolay 's formula works in practice. I've resorted to manually creating new jobs on successful processing of former jobs

[manast]

I will close this issue because it is not clear what the issue is about anymore anyway.

[ajwootto]

I think the documentation about repeatable jobs could use some serious updates with some of the information discussed in this thread. It's very unclear how repeatable jobs interact with custom jobIds and why jobs with the same repeatOpts are not treated distinctly.