Bull: Job events on different machines

[floolean]

Hi,
I would like to see following scenario:

• Machine1 creates jobs
• Machine2 processes jobs
• Machine1 listens for job events like 'completed', 'failed', 'progress' and so on.

Is this possible?
Thanks

[manast]

Listen to the event like this:

queue.on('event', listener, true);

[floolean]

sadly, that doesn't seem to work. Any option I need to set maybe?

[manast]

write a test case so that we can reproduce it.

[floolean]

This it the code of Machine1:

var queue           = require('bull');
var testQueue = queue('test',6379,'some.remote.redis', {password:'password'});

testQueue.on('completed', function(job, result){
    console.log('JOB ' + job.jobId + ' completed with result: ' + JSON.stringify(result) );
    createJob();
}, true);

testQueue.on('active', function(job, jobPromise){
    console.log('JOB ' + job.jobId + ' started' );
}, true)

function createJob(){
    testQueue.add({count: count})
    .then(function(job){
        console.log('JOB ' + job.jobId + ' created' );
    });
}

createJob();

The code of Machine2:

var queue           = require('bull');
var testQueue = queue('test',6379,'some.remote.redis', {password:'password'});

testQueue.process(1, function(job,done){
    done(null, {hello:'world'});
    console.log('JOB ' + job.jobId + ' completed' );
});

The events 'active' and 'completed' never get emitted on Machine1.

I noticed that:

`Events are by default local, i.e., they only fire on the listeners that are registered on the given worker, if you need to listen to events globally, just set to true the last argument:

queue.on('completed', listener, true):`

has been removed from the docs in the https://github.com/manast/bull docs. Has the functionality been removed at any time?

[floolean]

Does redis need any special setup to work in this scenario?

[floolean]

I tested both npm modules redis and ioredis, both work with pub/sub on my redis instance. I have no idea whats going wrong, please help.

Thank you!

[v4l3r10]

Same workflow. same problem.
Nodejs last LTS 6.x
bull 2.0
redis 2.8.17

[oliverwoodings]

I'm also seeing the same behaviour.

Node 6.2
Bull 2.1.0

[oliverwoodings]

I did some digging and have found the issue:

• By design, distributed events are emitted with the name of the queue attached i.e. a progress event for a queue called foo would be emitted as progress@foo around the distributed system.
• In order for consumers to be able to subscribe to the events without the name suffix, it looks like the queue is meant to automatically subscribe to the distributed events when it is created and then re-emit the distributed events without the suffix.
• This never happens, however, because listenDistEvent never actually listens to a distributed/global event since it never passes true as a third parameter to this.on
• This means that disturbed doesn't add a subscription for the event

You can work around this until its fixed by (as a consumer) changing the queue subscription from queue.on('process', ..., true) to queue.on('process@<queuename>', ..., true) (replacing ). Doing this means you will add a true global listener.

@manast it seems there are some inconsistencies in how people should listen for global events. The docs tell people to add true to queue.on if you want to listen for a global event, however it does not mention anything about the global events having the queue name suffixed to them. The code, however, makes it seem like as a consumer you never need to pass the third parameter to queue.on, since it automatically subscribes and re-emits global events for you. What should be the correct behaviour here?

[oliverwoodings]

@manast any thoughts on this?

[manast]

This issue is next in my todo list.

[manast]

The proposal I have is that we remove the last parameter "isGlobal", and instead global events will be prefixed with 'global:'. So if you listen to 'global:completed' you listen to all completed events in that queue, but if you only listen to 'completed' you will only receive the events for that worker. What do you think?

[oliverwoodings]

Would a node that created an event emit both a non global and a global
event?

On 12 Mar 2017 20:39, "Manuel Astudillo" [e-mail] wrote:

The proposal I have is that we remove the last parameter "isGlobal", and
instead global events will be prefixed with 'global:'. So if you listen to
'global:completed' you listen to all completed events in that queue, but if
you only listen to 'completed' you will only receive the events for that
worker. What do you think?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/OptimalBits/bull/issues/394#issuecomment-285974678,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAl7C3YrQnHyNvV875R2P4ZtxiNaz9XQks5rlFgLgaJpZM4K87rH
.

[manast]

Yes, the workers will always emit global events. This could be configured in the future to save resources if not needed.