Sidekiq: brpop is impeding performance

[loto]

Hi,

I'm having issues with Redis & Sidekiq. I have to admit that my understanding of the magic between SK & Redis is partial, unfortunately.

My redis server is hosted by RedisCloud. I have one Heroku Standard x1 worker running SK with the following setup:

Procfile

worker: bundle exec sidekiq -C config/sidekiq.yml -i ${DYNO:-1}

config.yml

:concurrency: 1
staging:
  :concurrency: 4
production:
  :concurrency: 7
:queues:
  - [critical, 3]
  - [default, 2]
  - [low, 1]

initializers/sidekiq.rb

Sidekiq.configure_client do |config|
  config.redis = { size: 2 }
end

Here's NewRelic insights on Redis brpop command:

• avg response time: 2s
• min: 4.6ms
• max: 22s
• throughput: 28cpm

It seems that Sidekiq is using the brpopcommand quite a lot!

Is my issue related to an improper configuration of SK and/or Redis or is it a real "issue"?

[jonhyman]

See https://github.com/mperham/sidekiq/issues/2431, Redis 3.0.3 may help you.

[jonhyman]

Actually, it doesn't look like you have a large number of clients. brpop is a blocking command, too, so if you don't have a lot of jobs then each one will block for 1 second. Not sure why your average response time is 2 seconds, you may want to profile in your application, but my guess is that your application has a low volume is jobs and the blocking is fine since Redis will respond with a job when it has one.

[mperham]

What's impending performance is that you have everything turned down as low as possible. 1 worker thread and 2 redis connections (not enough, 3 minimum). brpop is not the performance issue.

[loto]

Thank you for your help, I will tune things up a bit.

[randomm]

Hi!
I'm experiencing something very similar. @L-Oto did you get this to go away? What settings did you use?

[loto]

Unfortunately I didn't got rid of it. Following @mperham advice I tuned up my settings but my Redis still gets a lot of brpop queries with a response time varying from 64ms to 3s with a cpm of 20.

Is this related to my Redis hosting service or to an improper configuration of Sidekiq, I don't know. I guessing there's something between the concurrency value and the number of queues...

[mperham]

Why do you think this is a problem? Sidekiq will normally block on Redis, using BRPOP. That's totally normal.

[onomated]

@L-Oto @randomm I'm having the same worrisome results in my newrelic dashboard:

Most time consuming
Redis brpop
2,290 ms average query time
26.1 cpm throughput

@mperham Does sidekiq block on brpop for the duration of the background job?

[ryansch]

@onomated You _want_ that behavior. That's what sidekiq is doing while it's waiting for work. It's a long running blocking operation because redis can then tell sidekiq about work as soon as it arrives. The alternative would be polling which we certainly don't want.

I see the same thing in my new relic dashboard. No worries.

[onomated]

Thanks for explaining @ryansch

[bf4]

@ryansch

You want that behavior. That's what sidekiq is doing while it's waiting for work. It's a long running blocking operation because redis can then tell sidekiq about work as soon as it arrives. The alternative would be polling which we certainly don't want.
I see the same thing in my new relic dashboard. No worries.

Does that mean that using the same redis db for caching and sidekiq causes sidekiq to block or otherwise impact caching operations?

[mperham]

No. N connections can be blocked while other operations proceed.

But you don't want to use the same Redis instance for caching and jobs. The two systems optimally need completely different persistence configurations: cached data should have no persistence and bg jobs need a level of persistence based on your app and how critical jobs are to it. For example, the jobs db usually should have replication or a save policy enabled.

http://redis.io/topics/lru-cache
http://redis.io/topics/persistence