Bull: Multiple cron job in a queue

[sudhirbitsgoa]

Hi, Thanks for the great library.

Issue steps

create file: cron1.js

var Queue = require('bull');
var cronDelay = new Queue('cronDelay');
cronDelay.add({
  everysec: 'every1sec'
}, {
  delay: 100,
  repeat: {
    cron: '*/1 * * * * *'
  }
}).catch(err => console.log(err))

cronDelay.process(5, (job, done) => {
  console.log('%j', job.data);
  done()
});

this will create a job cronDelay
and adds a cron job of every 5 secs

create file cron2.js

var Queue = require('bull');
var cronDelay = new Queue('cronDelay');
cronDelay.add({
  everysec: 'every5sec'
}, {
  delay: 100,
  repeat: {
    cron: '*/5 * * * * *'
  }
}).catch(err => console.log(err))

cronDelay.process(5, (job, done) => {
  console.log('%j', job.data);
  done()
});

This will add cron job of every 5 sec to the same queue

Issue: when i run

node cron1.js 

it keeps printing

{"everysec":"every1sec"}
{"everysec":"every1sec"}
{"everysec":"every1sec"}
{"everysec":"every1sec"}
{"everysec":"every1sec"}

Now in other terminal run

node cron2.js

It just prints

{"everysec":"every5sec"}

for one time and both the jobs are stopped from then.

Any clue what is wrong.

[weeco]

Might be the same bug reported in #747 I think

[sudhirbitsgoa]

The behaviour is same on
bull 3.3.1
node v4.4.3
redis 4.0.1

[manast]

Can you please recheck with bull 3.3.3?

[sudhirbitsgoa]

In 3.3.3 the issue is still reproduced.
In this case the job with lower repeating time continues to work and other with greater time stops after executing once. I tried with 3 sec and 6 secs as well 5 and 10 sec.

[manast]

weird because I could reproduce the issue with 3.3.2, and understood the reason for it, fixed it, and could not reproduce it anymore. The reason was that the keys were overwriting each other, this is not happening anymore.

[ritter]

Something I've noticed too, is that the Queue.add will return a job when it's successfully added, but I have instances when undefined is returned instead.

[manast]

@ritter that should only happen when adding repeatable jobs. It could be remedied though, but basically it happens then the repeatable job has already been added by the repeat logic.

[ritter]

The issue however is they are not added, I end up with only one repeating job and should have half a dozen. Even though they all have unique names.

[manast]

@ritter please provide a test case, so that I can have a chance to find the problem.

[ritter]

I've attempted a simple test with 2 repeat jobs and I'm not having the issue using v3.3.3 so I'm going to look more in-depth at my code to see what's different.

[ritter]

I'm still trying to find the cause, but what I am finding is that after a repeat job runs, its not re-added to the delayed ZSET, I can see it in the repeat ZSET which prevents me from re-adding the job without removing it first.
And it appears to be missing the job HASH keys under bull:<queue>:repeat:*

[ritter]

@manast how about using this cron as a test: { "cron": "0 0 * * * *" } which I was using for an hourly.

[weeco]

@ritter You can simply use 0 * * * *to start at minute 0 every hour. I am not sure if seconds are supported

[ritter]

{ "cron": "0 * * * *" } is not working, I lose it after it runs first time.
{ "cron": "0 */1 * * *" } is not working either, same result.

[ritter]

@manast job1 runs at the same time as job2, but never comes back. Also, job1 shouldn't even be running when job2 fires off (except at top of the hour).

const repeatJobs = new Queue( "RepeatJobs", queueOpts );
repeatJobs.add( "job1", { "data": "job1" }, { "repeat": { "cron": "0 */1 * * *" } } );
repeatJobs.add( "job2", { "data": "job2" }, { "repeat": { "cron": "*/1 * * * *" } } );

[weeco]

@ritter Then it must be something else, because in general I can setup a job with the given cron and it will execute every hour at minute 0.

Maybe you can provide a full example which does reproduce your setup. Any code issues could be identified there

[ritter]

@weeco I just did, it's the second job in my above example that when it runs also runs job1 and then job1 never comes back.

[manast]

ok. I am working on a small rewrite that is both a simpler and more stable code.

[manast]

You are welcome to review the PR in case I missed something :)

[sudhirbitsgoa]

@manast Thanks for the quick turn around.
Works perfect for me.

[manast]

fixed in 73e4e440a02d69b99cd3869b0e2d81939063fad0