Bull: Recurrent jobs (according to a cron specification)

[manast]

It should be possible to add jobs that will be processed according to some cron specification (once or multiple times)

[linux4kix]

I do this using the later module. http://bunkat.github.io/later/

[matthewmueller]

+1, @linux4kix can you be a little more specific? Are you running queue.process in a setTimeout or something?

[linux4kix]

@matthewmueller Sure.

I am doing it the other way around. I use later to parse the cron schedule of a job and use later.setInterval to add the job to my queue at the specified interval. If you have to run the job immediately then you can use a priority queue.

This is just a quick hack to demostrate.

        var cronSched = later.parse.cron(schedule); 

        var timer = later.setInterval(function() {      
            queue.add(data, opts);                     
            },                                          
            cronSched                                                      
        ); 

[manast]

So maybe I should document this as a pattern usage instead of adding it to bull, since it is so easy and clean to do anyways.

[linux4kix]

It would be easy enough to add a schedule job option and if it is set catch
it in _add and wrap it in a handler similar to this. Although I don't know
if it is worth pulling in another dependency for this feature. I think
either is acceptable.

On Sat, Mar 12, 2016 at 9:10 AM, Manuel Astudillo [email protected]
wrote:

So maybe I should document this as a pattern usage instead of adding it to
bull, since it is so easy and clean to do anyways.

—
Reply to this email directly or view it on GitHub
https://github.com/OptimalBits/bull/issues/252#issuecomment-195695211.

[manast]

I would like to revisit this feature. Basically I think a cron like behaviour is quite useful, and the pattern explained above would work, but it is limited to just one client controlling the crons, and it lacks robustness in the case of failure.
I would like to propose a feature that is scalable, i.e. you can have many workers configured according to a cron job, so if any dies, the cron jobs will continue working, and with at-most-once semantics, i.e, if workers die or new worker are launched, no more than one job that fullfils the cron specification is processed.
My idea is to create a special key for every cron that expires the next time a cron job is to be processed. All workers will try to create a new key according to the cron specification, but only one will succeed, the one that will add the job to the queue, the job can then be processed by any other worker.

[jamesjjk]

@manast unique keys (jobId) are a great way to control repeatable jobs in a more fault tolerent design (i.e. where you operate a cluster/sentinal setup)

[manast]

@jamesjjk yes, you are right. The solution I have in mind is based on the customIds feature already in bull.

[jamesjjk]

@manast I am using a very similar approach, the main issue is with jobs that 'removeOnComplete' - which is not possible in combination with repeatable jobs.

[lpgeiger]

@manast Is this still targeted for 3.0?

My idea is to create a special key for every cron that expires the next time a cron job is to be processed.

[manast]

This is almost implemented, I am writing tests now.

[lpgeiger]

@manast great :) looking forward

[manast]

@jamesjjk removeOnComplete will work since for every repetition a new job is actually created. This has several advantages and fits well in bull's design.

[manast]

PR: https://github.com/OptimalBits/bull/pull/527