Bull: Can jobs be added in bulk?

[brandonmp]

I have a process which generates 100k+ jobs for my queue, and adding them 1-at-a-time is a lot of i/o overhead. this snip:

 for (let i = 0; i < 10000; i++) {
        await testQ.add({ taskID: i, taskMeta: "this is metadata 1" }, { removeOnComplete: true })
    }

took several minutes to complete with a redis server on redislabs.com

Is there a way to construct the jobs locally and add to redis in bulk?

[manast]

There is no way to do this with bull, but should not be very difficult to implement as an addBulk operation, where an array of jobs is sent to redis in one atomic operation, that would increase the performance in at least one order of magnitude:
https://github.com/OptimalBits/bull/blob/master/lib/scripts.js#L73

[sschizas]

@manast any plan to implement this? I think it would be very helpful addition.

[laukaichung]

@manast @n3trino
Any news? I'd love to have this bulk adding feature.
Currently my workaround is using https://github.com/palanik/forloop to make the loop non blocking. Performance isn't bad. Is it a good idea?

forloop(1, 20000000, 10, (i => { queueUpdate.add(i, {delay: 3000, removeOnComplete: true, removeOnFail: true}); }) );

[manast]

you do not need that library in this case, since add is already an asynchronous function. On the other hand you may want to store the resulting promises to know when you are actually done adding jobs, something like this:

var adds = [];
for(var i=0; i < NUM_JOBS;i++){
  adds.push(queueUpdate.add(i, {delay: 3000, removeOnComplete: true, removeOnFail: true});
}

Promise.all(adds).then(function(){
  console.log('done');
});

This issue is about having a bulk method so that we can save roundtrips to redis.

[jrr]

Another reason for this is to be able to transactionally enqueue a bunch of jobs.

[Igor-Lopes]

Hello, are there any updates on this issue ?

[Igor-Lopes]

There is no way to do this with bull, but should not be very difficult to implement as an addBulk operation, where an array of jobs is sent to redis in one atomic operation, that would increase the performance in at least one order of magnitude:
https://github.com/OptimalBits/bull/blob/master/lib/scripts.js#L73

@manast , Is it possible to do this only by creating a new function addBulk in scripts ? Or should I create a new lua script (in commands dir) to add bulk ?

Can you help me ?

[manast]

@Igor-Lopes I think the easiest it so modify the current addJob-6.lua script to accept an array of jobs and just put the current code in a loop, so that priorities, events, etc are all working as in the simple add case.

[Igor-Lopes]

@manast Thanks ! I'll try it.

[holm]

I added a suggestion for how the bulk support could be implemented here: https://github.com/OptimalBits/bull/pull/1419

[Igor-Lopes]

I added a suggestion for how the bulk support could be implemented here: #1419

@holm I think you've found a good solution and it's better than changing the Lua script. Can it be merged @manast ?

[manast]

I think this is a great solution. It is probably not as fast as having it built in LUA but nevertheless its simplicity is very nice, so I can merge this and in the future it can be optimized if needed.