Rq: Per queue rate limit.

[proofit404]

I think it is useful to have rate limit per each queue on the worker side. If this is desired feature lets discus how it maybe shaped in the API and possible approach to it implementation. If we agree on this at some point then I will provide PR.

[selwin]

Yeah, I think it's about time we work on this feature.

How do you plan to implement this? I was thinking about using a lua script that checks the length of the queue and enqueueing the job in the same transaction but I haven't looked into whether it's hard to implement.

[proofit404]

I don't think lua script can help here.

I define rate limit as an option to worker class. It means don't dequeue from this queue frequently than specified period of time. I think it is possible to implement this feature as part of Queue.dequeue_any logic.

Dequeue any takes a timeout argument. We have a mapping of queues to it rate limits. Somethink like

{
    'foo': '20/s',
    'bar': '5/h',
    'baz': '10/m'
}

Also we have a state inside worker instance which record what queue keys was used earlier and at what time.

So dequeue_any can look at this state, can look at timeout argument and make decission like "Queue.lpop will wait for 5 min, there is bar queue key in the arguments but I already execute 5 tasks from this queue during this hour. I will skip this queue key from arguments until next hour."

[selwin]

Workers are supposed to process as many jobs as possible so implementing rate limit here seems a bit counter intuitive to me. Workers can also dequeue from many different queues so that will complicate things.

The rate limit should be implemented on the Queue side, where it rejects jobs if number of pending jobs on the queue have exceeded a certain number (this is a good place to start since it's so simple). In fact, we already have an open issue for this: https://github.com/nvie/rq/issues/435

[proofit404]

I read connected issue. Those mechanics suppose to do different thing. I'm fine to implement behavior described in this issue as a separate package. But I still want to hear yours opinion on offered implementation.

[selwin]

I think rate limiting would be rather complicated to implement on the worker side. If you want to skip this queue until the next hour, you'll need to control the dequeue timeout too.

[jiajunhuang]

so will rq support rate limit in the future?

[nad2000]

Is there any wait to throttle the message processing rate based on the specified rate limit instead of rejecting the incoming messages? Let's say there is a burst of messages into the queue but the worker picks them up at a steady rate.

[mahnunchik]

Any news?

[cwazuidema]

I would love this feature as well, @nad2000 solution sound great to me.
If possible, limit the process rate of a certain queue would be awesome!

[selwin]

I think this is the right time to consider this both on the worker side and the queue side provided we can find a good way to implement this.

I'll reopen this issue for further discussion.

On the worker side I think this should be relatively straight forward to implement.

[guettli]

AFAIK Celery supports this: https://docs.celeryproject.org/en/stable/userguide/tasks.html#Task.rate_limit
I have not tried it yet. At the moment I decide whether to use rq or celery.

[cwazuidema]

@guettli
Thanks for letting me/us know!

I was in the same boat as you, deciding between rq and celery.
I believe I went for rq for its simplicity.

However a rate limiter on a queue will certainly make rq the better product (for my use case).

That being said, I still you rq for my project and it's working fine.
I just create less workers when I hit the rate limit.