Sidekiq: Augment concurrency/weighting to allow for per-queue concurrency

[ryana]

Has anyone ever thought about augmenting the concurrency/weight option on sidekiq to allow for concurrency per queue instead of per process? I understand that a single sidekiq worker can handle multiple queues, but in our use case, there are several classes of jobs that can take 10-40 seconds. These long jobs are not CPU intensive; they are hitting an external endpoint that takes a long time to respond. Because of this, in order to ensure that queues don't get starved, I'm finding that I need to bring up a sidekiq for every queue in the system. This is wasting a lot of memory and is antithetical to the purpose of Sidekiq.

So what I'd like to do is:

sidekiq -Q default -Q incoming_messages,10 -Q outgoing_messages,10

Where the -Q option says: create up to X threads per queue. X defaults to 5, and can be overridden with the comma notation. In order to keep things sane, I think the usage of the -Q option would cause any usage of the -c or -q options to be ignored.

This would have the added benefit of allowing CPU-intensive sidekiqs to help out with IO intensive jobs, since the IO intensive queues (incoming_messages in this case) could be picked up by new threads immediately, whereas in the current model they would have to wait for a CPU-intensive queue worker to finish the job.

We really need something like this, so I'm going to start working on a patch, but wanted to get a few opinions on it first.

Thanks!

[brainopia]

https://github.com/brainopia/sidekiq-limit_fetch – allows to set limits on queues, I think it would serve your purpose

[ryana]

Wow. You just saved me a lot of time. Thanks @brainopia!

[mperham]

Use a connection pool to limit your concurrency to that server.

[brainopia]

connection pool would limit concurrency but would not prevent sidekiq processors from picking up related jobs, so essentially you would get a lot of blocked processors.

[ryana]

Yeah. What @brainopia is suggested is exactly what I'm looking for. Thanks so much for building this. It's great. Gonna have it deployed in a few minutes :)

[ziaulrehman40]

sidekiq-limit_fetch gem has some issues when used with sidekiq-unique-jobs. Also it is not being maintained properly, and does not officially supports sidekiq 5. We need an official solution from sidekiq i think. :/

[mperham]

This is Sidekiq's official solution: https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting

[ziaulrehman40]

@mperham Does this solution play well with sidekiq-unique-jobs?

[mperham]

It works with https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs I cannot and do not make any claims about 3rd party plugins.