Bull: Python library or reverse engineering Bull protocol

[davedbase]

Hi there,

Our team would like to use Bull for our API layer and for pushing jobs into queues. Our backend processors are written in Python and we were wondering if there was a Python library to directly interface with Redis, process queues and respond accordingly.

If not I was wondering if there was documentation outlining how Bull pushes and updates the Redis keys so that we can write a Python wrapper to perform the same actions. We'd probably be interested in open sourcing the solution.

If we can get support or guidance on the logic/protocol for the Redis structure it would be incredibly helpful as opposed to reading the JS and reverse engineering directly.

Even a basic explanation would be useful to implement the worker processors exclusively. Ultimately the Node side will handle pushing and receiving queue data for distribution back to our end users. If this sounds like an interesting project, we'd love to work on expanding Bulls reach and usability in different environments.

Thanks for any help in advance.

[manast]

If you only want to add jobs from python its quite easy, you can just use the addJob.lua script. There are other commands that you more or less could also call directly from a python redis client https://github.com/OptimalBits/bull/tree/master/lib/commands
If you instead want to process jobs as well, then it is far more complicated, because the logic to process jobs has been refined in many iterations and you would need to implement it almost identical to the node version in order to have a reliable queue. What I would advise if you want to use python to process the jobs is to write instead a simple node websocket server that wraps the queue. The python clients could connect to this server via a websocket and be notified when to process a job and then notify back to the node server when the job has been completed. I am not saying this is a super simple solution but at least I think it is doable.

[bradvogel]

Yeah - using Bull to produce jobs in Node and process them in Python is far harder than the opposite.

Have you looked at Celery? You can use https://github.com/mher/node-celery to produce jobs in Node and then consume then in Python using Celery.

[n1ru4l]

We actually just reverse-engineered a python Worker that supports delayed jobs with asyncio, aredis and converting some of the source code to python. It has not the complete feature set but we might open source it, first we will test it in production.