Sidekiq: Understanding Queues and Weighting

[snikch]

Hi Mike,

First up – thanks for Sidekiq, it's awesome and I really appreciate the work you've done in creating an awesome job queue.

I ran across an issue today that I'm still unsure of the cause. This might be me completely misunderstanding the way Sidekiq queues work, but I wanted to open up a discussion to check if that's the case.

Previously we had all jobs running on the default queue. Last saturday we had an issue with a bunch of relatively slow jobs causing important jobs to get delayed, so I implemented several queues with different weightings.

Environment

Sidekiq Version: 2.17.0
2 sidekiq processes running on Heroku with concurrency of 3 each

Config file:

---
:queues:
  - [critical, 7]
  - [high, 6]
  - [default, 5]
  - [low, 3]
:concurrency: 3

Jobs

For the sake of this issue, let's say I've got three jobs.

• ProcessRequest - On the default queue
• ProcessDelivery - On the critical queue
• UpdateStats - On the low queue - may take a couple of seconds to process

Each ProcessRequest job on the default queue generates 1-n ProcessDelivery jobs.

Each ProcessDelivery job generates 2 UpdateStats jobs in a very short time (1 immediately, 1 as a result of a webhook from an external service).

Additionally there may be more UpdateStats jobs queued as a result of a webhook from an external service.

What I expected

Based on this ordering and weighting of the queues, I expected Sidekiq to process any job on the critical or high queue before attempting to process a job on the low queue.

What I saw

While monitoring this initially, I believed I saw what I was expecting. critical and default jobs were processed immediately, and the low queue would back up occasionally then get processed when the other queues were empty.

However this morning, I saw that there were around 4,500 jobs enqueued, and when I dug deeper I found that the majority of these were on the default queue. The critical queue was clear, there were ~500 on the low queue and ~4000 on the default queue.

While I watched, jobs on all queues were processed, but the vast majority were on the low queue, despite a heavy backlog in the default queue. At this point the critical queue was kept empty.

Speculation

Weightings were the wrong thing to use and caused the issue. Since a high percentage of all jobs are on the low queue, the weightings may have caused the priority to actually end up quite high for the low queue.

What I don't understand is how Sidekiq actually decides what queue to process next. If I was to remove all the weightings, would that mean that sidekiq checks queues sequentially? i.e. when looking for a job, it would check in critical, process a job if it found one, then start the whole process again - back at critical.

[mperham]

Imagine queues [a,1], [b,2], and [c,3]. This means Sidekiq should pick from c 50% of the time, b 33% of the time and a 16% of the time. Weights prevent starvation so the critical queue doesn't totally halt work on the default or low queues.

It sounds like you want strict ordering. Remove the weights to get this behavior.

[snikch]

Brilliant - as I wrote out the issue this started to become clearer, but I thought I'd get a definitive answer.

Cheers! :beers: