Kue: Documentation on the protocol

[bitliner]

One key feature of queues and distributed systems is to integrate together multiple programming languages.

Actually I see there is only the node.js library, do you plan to extend Kue also to other languages?

Specifically, I would need Kue in Java in order to integrate into a Kue based system also a Java worker.

So I though to replicate in Java the peace of kue.js module to connect to the Redis queue and the worker.js module. Is it the right approach?

Is there anything already under development that goes into this direction?

[behrad]

Actually I see there is only the node.js library, do you plan to extend Kue also to other languages?

No

So I though to replicate in Java the peace of kue.js module to connect to the Redis queue and the worker.js module. Is it the right approach?

Why aren't you delegating the work from a node Kue worker to your Java worker via WebSockets and listen for the done acknowledge from the java process. You need a bi-directional socket connection between the two process.
However you can read the worker.js and find out the BLPOP section and ZPOP operation... that way you can pop a job in your Java code, look at the process method for done ack to free-up worker.

Is there anything already under development that goes into this direction?

I strongly refer you to https://github.com/antirez/disque which I believe would be the next generation of Kue :)

[bitliner]

Hey @behrad ,

I did not think to use Websocket communication because I need to scale on the worker side. With websocket communication I should build a router, while in Kue or in a publish/subscribe pattern this is achieved by default.

Does it make sense?

Thanks for letting me know about disque, it is still in beta though :-(

About the usage of a queue with blocking operations, why did you prefer this choice instead of using publish/subscribe pattern?

[behrad]

I did not think to use Websocket communication because I need to scale on the worker side. With websocket communication I should build a router, while in Kue or in a publish/subscribe pattern this is achieved by default

I think inter-process communication can also be modeled in a clustered way...

About the usage of a queue with blocking operations, why did you prefer this choice instead of using publish/subscribe pattern?

1) Redis pub/sub is only for active subscribers, offline workers loose messages/jobs.
2) pub/sub is for dispatching a single message to many workers which is not the desired behavior for a job queue.

[bitliner]

pub/sub is for dispatching a single message to many workers which is not the desired behavior for a job queue.

Right, I missed this detail.

Is there anything in the roadmap of Kue that will be built?

[behrad]

Nothing at my side, but extra modules are welcome :)

[bitliner]

Thank you very much for your information. I will keep you posted in case I take any decision on it.

[lastlegion]

I've documented the communication between Kue and Redis here: http://ganeshiyer.net/blog/2016/06/29/kue-under-the-hood-using-the-job-que-externally/

This is for people looking to use/extend Kue from outside of Node.