Sidekiq: Feature request: limit number of workers per queue
Hi!
Do you think it's possible add a parameter to limit the number of workers for a given queue?
This would be useful if we have a queue that we don't want to consume too many workers, and, in the case I want, to create a sequencial queue (1 worker).
Thanks!
golive
All 35 comments
See the Advanced Options page of the wiki.
ezkl
on 17 May 2012
You can adjust the concurrency and which queues Sidekiq will process. Beyond that, you need to design your application code and workers to handle sequential and parallel workflows. Maybe there is potential for Sidekiq to extract design patterns or conventions here.
mperham
on 18 May 2012
+1 for for this ability... In my code base I have lots of things that run just fine in parallel, but I have 1 queue that must be processed in order.
It's not that hard to start up a separate process to process that one queue, with the concurrency set to 1 to process that one queue, but seems a little bit wasteful to hear 2 processes running
aselder
on 19 May 2012
I solved the sequential queue problem by writing a server side middleware allowing workers to be wrapped with a semaphore using https://github.com/dv/redis-semaphore. Maybe I should package it as a gem but it's not that many lines to wrap your own in the meanwhile.
betelgeuse
on 10 Jun 2012
Maybe it doesn't help @aselder though because it only makes sure there's one copy running and not that they are processed in insertion order.
betelgeuse
on 10 Jun 2012
I'd love to see someone publish middleware that does this. I don't think it should be in Sidekiq core for now.
mperham
on 17 Jun 2012
Has anyone started writing middlware that does this? I've run into this need, I have a particular job that smashes the CPU/RAM of the machine and I'd like to limit that particular worker to only a few concurrent jobs.
Perhaps this would be easier solved by running two different sidekiq processes that listen to different queues? There would be som RAM overhead but other than that I think it would solve the problem. What are everybody's thoughts?
mariovisic
on 28 Jul 2012
Just wanted to add my voice to the request for this much desired component. Thanks everybody!
mysterryous
on 5 Aug 2012
+1 I've just run into this as I only want a couple of workers running off a _download_ queue. Anyone seen any middleware yet to do this?
philostler
on 5 Dec 2012
I've been running two different sidekiq processes with different queue options (one with 100 threads and another with intensive work with 5 threads). Works fine; only downside is a bit of extra RAM usage due to running an extra process.
mariovisic
on 6 Dec 2012
Sounds like the setup I'm going to have to adopt. It'll work of course but it would be ideal not to have to fire up one or more additional processes for a situation that could be fixed with middleware. If it bugs me enough I might look into writing something
philostler
on 7 Dec 2012
The RAM overhead isn't huge, I believe it's just shy of 100MB for my small/medium sized rails application. If you made some middleware I would be keen to try it out; if I had some spare time I would have attempted this already but 100MB of RAM is cheap/free;
mariovisic
on 7 Dec 2012
Sure, it's nothing to worry about when there's many gigs of RAM on tap. I'm a bit of a perfectionist though and this is just the kind of thing that stops me sleeping at night lol
philostler
on 7 Dec 2012
Unless you're using Heroku, in which case you're looking at an additional
$35 per month.
I'd be eternally grateful if a developer more skilled than myself took on
the challenge. I certainly would put it to use.
On Dec 6, 2012 4:27 PM, "Phil Ostler" [email protected] wrote:
Sure, it's nothing to worry about when there's many gigs of RAM on tap.
I'm a bit of a perfectionist though and this is just the kind of thing that
stops me sleeping at night lol—
Reply to this email directly or view it on GitHubhttps://github.com/mperham/sidekiq/issues/200#issuecomment-11112862.
mysterryous
on 7 Dec 2012
Sure, on heroku it'll cost you extra $$, but if you're on your own hardware, an extra 100MB isn't too much to worry about compared the the RAM savings you can get by running 100 parallel threads using sidekiq.
If I get some spare time I might look into this, but not anytime soon, sorry folks.
mariovisic
on 7 Dec 2012
Some nice fella seems to have created a middelware for this: https://github.com/brainopia/sidekiq-limit_fetch
I'll probably swap to using this.
mariovisic
on 21 Jan 2013
Did somebody try that sidekiq-limit-fetch ? I really need this feature.
juniorjp
on 20 Feb 2013
Yes, I used it for a short time in development. Not had time to use it in anger in a production environment yet but from what I saw it did everything I needed
philostler
on 20 Feb 2013
if you hit any issues I would be glad to help :)
brainopia
on 20 Feb 2013
Thanks for the fast reply, I will try it :+1:
juniorjp
on 20 Feb 2013
This is pretty important to have as an option in sidekiq esp. if you're using 3rd party api's that only allow a particular # of concurrent requests to avoid throttling and immediate failing of API calls. I can think of many instances where you would want to only allow a certain # of workers against a specific queue for example loading lookup tables in a database where other loads are depending on those lookup tables being loaded first.
This is the one area where resque's ability to specify a certain # of workers by queue is better.
henry74
on 6 Mar 2013
@henry74 Limiting concurrent connections to a resource is exactly what a connection pool does.
mperham
on 6 Mar 2013
I'm familiar with collection pooling but it isn't obvious to me based on documentation how I can specify only 4 of the 25 threads should ever be working on a specific queue at any point in time. I believe that is what original feature request is about. It appears the answer is to use a separate gem someone else created for this specific use case.
henry74
on 6 Mar 2013
@henry74 you can use this now, see: https://github.com/brainopia/sidekiq-limit_fetch
It works well, we're using it in production right now
mariovisic
on 6 Mar 2013
Thanks Mariovisic - we'll use this gem. Looks like a good alternative.
henry74
on 7 Mar 2013
Can sidekiq-limit_fetch be used to ensure a queue is processed in order? - i.e. if I set a limit of 1 for the queue, will that ensure the order of the queue will be maintained?
I have some queues that can be processed in parallel and some queues that must be processed in order, 1 item at a time.
tetherit
on 25 Feb 2014
@hackeron yes, the order will be maintained, but exceptions can break the order since the task will be reinserted after retry period.
brainopia
on 26 Feb 2014
Ah, ok, so in order for the order to be maintained, would need to remove the retry middlewhere and handle exceptions myself I guess
tetherit
on 26 Feb 2014
Yeah
brainopia
on 26 Feb 2014
This would be simple to add (some people in this thread have implemented it themselves), and extremely useful for any sufficiently complicated project. Deciding not to include this is a mistake.
alflennik
on 11 Aug 2017
I agree with @alflennik, but I'm probably partial since I need the feature and am not a Sidekiq pro user so I can't use https://github.com/brainopia/sidekiq-limit_fetch.
brandoncc
on 11 Oct 2017
Oh, I totally skipped the first sentence in the Requirements section. I only saw "Requirements" and "sidekiq pro" kicks himself
brandoncc
on 11 Oct 2017
I'd be interested to hear the reasoning why this is unsuitable to be included in core. Genuinely interested because as far as I can see @mperham is pretty damn skilled and there's perhaps something to learn.
I appreciate that sidekiq has succeeded partly due to its great respect for SRP. Still, the fact that I can choose to limit all of sidekiq to one thread seems similar (from a user's view) to the proposed option to limit a queue to 1 thread.
Adamantish
on 24 May 2019
I would love this to be a core feature. Using limit_fetch is not an option, as it's unmaintained, and does not seem to work well with all the latest features of Sidekiq Pro/Ent such as super_fetch.
scottrobertson
on 29 Dec 2019
Because Sidekiq talks directly to Redis, throttling is difficult to implement in a scalable manner. You must use Lua and, with Redis being single threaded, you'll burn a lot of that single core when throttling. Additionally limit_fetch does not work with the reliability feature so you can lose jobs.
Faktory uses a different architecture where it is a single point of access for jobs; it can implement throttling in a safe and efficient manner. Faktory Enterprise will provide a throttling feature and is due for release this month (Jan 2020).
mperham
on 31 Dec 2019
Related issues
edgarjs
·
3Comments
davidcelis
·
3Comments
bartimaeus
·
3Comments
BeRMaNyA
·
3Comments
jlecour
·
4Comments
Most helpful comment
This would be simple to add (some people in this thread have implemented it themselves), and extremely useful for any sufficiently complicated project. Deciding not to include this is a mistake.