Bull: Way to process broadcast on multiple consumers

[joe-at-startupmedia]

Is there a way to designate a particular job not only
1) Broadcast to all consumers
2) Be processed by all of the consumers
I'm trying to do the opposite of what was resolved here:
Jobs processed more than once #91

[manast]

No. that is not possible. The problem with the broadcast is that you cannot know how many workers should perform the job. Normally standard pubsub is used for broadcasting. I recommend you take a look at redis STREAMS.

[stansv]

There's no such possibility in Bull, you'll have to implement this by your own. I can suggest one of many ways to do this

• once worker is started, create a queue with random name (with uuid inside for example). This is to be used only by this worker.
• use Redis directly to store pointers to these queues. For example, these can be keys with some fixed prefix and TTL set
• use setInterval to renew queue keys while worker process is live
• to submit a job, list all keys by prefix, create Queue instance for each and call add on it. This can require caching and you should take care on releasing resources such as Redis connections used inside these Queue instances
• also you'll probably need to create some scheduled job to remove old queues from Redis — old queues won't have appropriate key because we've set a TTL and dead workers won't prolong key.

[joe-at-startupmedia]

@manast Thanks for the quick response. "The problem with broadcasts is not knowing how many workers there could be"? I'm failing to understand how that is an issue. Who cares if there or 10 consumers or 100 consumers? All I want to make sure is that all consumers receive the message and process it accordingly.

I have looked into Redis streams, in fact I've taken their online university course on the topic. The problem is that I don't want to have to rewrite my entire production queue system to use the streams implementation instead. I would like to get buy with Bull as long as possible before I rewrite our codebase in an entirely different programming language altogether.

Aside from that after reading around and playing with the global events. How would the following not achieve the issue? Let's say I have the following code on all of my consumers:

queue.on('global:completed', function(jobId, result) {
  console.log(`Job ${jobId} completed! Result: ${result}`);
  queue.getJob(jobId).then(function(job) {
    processJob(job);
  });
});

Where processJob is a function that contains business logic to check that a job is never performed more than once on any given consumer?

What issues/shortcoming to you see with this approach?