Sidekiq: Recurring connection_pool.rb error

[kceb]

Ruby version: 2.3.4
Sidekiq / Pro / Enterprise version(s): 5.0.4

Config: I am using puma with 35 threads (RAILS_MAX_THREADS) & 2 workers. We have autoscaling set to be able to reach 4 instances & have a RDS instance that can support ~ 300 connections. I've read previous issues that mention that it may be an issue with my redis pooling config, but I copied the one from the readme.

Sidekiq.rb:

# frozen_string_literal: true
Sidekiq::Web.use(Rack::Auth::Basic) do |user, password|
  [user, password] == [ENV['SIDEKIQ_USER'], ENV['SIDEKIQ_PASSWORD']]
end

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]

redis_conn = proc {
   Redis.new(url: "rediss://:#{ENV['REDIS_AUTH_TOKEN']}@#{ENV['REDIS_HOST']}:#{ENV['REDIS_PORT']}/12", password: ENV['REDIS_AUTH_TOKEN'])
}

Sidekiq.configure_server do |config|
  config.redis = ConnectionPool.new(size: 25, &redis_conn)
  config.logger = Le.new(ENV['LOGENTRIES_TOKEN'], tag: true, ssl: true)  unless Rails.env.test? || Rails.env.development?
end
Sidekiq.configure_client do |config|
  config.redis = ConnectionPool.new(size: 5, &redis_conn)
  config.logger = Le.new(ENV['LOGENTRIES_TOKEN'], tag: true, ssl: true)  unless Rails.env.test? || Rails.env.development?
end

Sidekiq.yml:

# Sample configuration file for Sidekiq.
# Options here can still be overridden by cmd line args.
# Place this file at config/sidekiq.yml and Sidekiq will
# pick it up automatically.
---
:verbose: false
:concurrency: 10

# Set timeout to 8 on Heroku, longer if you manage your own systems.
:timeout: 30

# Sidekiq will run this file through ERB when reading it so you can
# even put in dynamic logic, like a host-specific queue.
# http://www.mikeperham.com/2013/11/13/advanced-sidekiq-host-specific-queues/
:queues:
  - [my_algorithm_queue, 2]
  - default
  - mailers

# you can override concurrency based on environment
production:
  :concurrency: 25
staging:
  :concurrency: 25
sandbox:
  :concurrency: 15

Error:

https://d.pr/free/i/hovgBR

Jobs are still being processed, but this error is worrisome

Are you using an old version?
Not an old major version
Have you checked the changelogs to see if your issue has been fixed in a later version?
Yes
https://github.com/mperham/sidekiq/blob/master/Changes.md
https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md
https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md

[mperham]

Your error report and backtrace do not include the actual error message. I can't know what's wrong.

That said, remove the size: attribute from your connection pools. It's possible that will fix the issue.

[kceb]

Hmm, that's the entire backtrace that was printed in the logs. I will try removing the size: attribute & try to see if I can get a better backtrace via the sidekiq backtrace option

[kceb]

@mperham Only error I see is a Timeout Error: https://d.pr/free/i/YRzD1i

Raised here

I've tried pinging redis in the console via:

      ::Sidekiq.redis do |r|
        res = r.ping
        res == 'PONG' ? '' : "Sidekiq.redis.ping returned #{res.inspect} instead of PONG"
      end

and it returns PONG. Also elasticache web dashboard doesn't show anything that would lead me to believe that it has reached a connection limit.

::Redis.new(url: HealthCheck.redis_url, password: ENV['REDIS_AUTH_TOKEN']).ping also works.

Also, the sidekiq queues are currently empty, so I'm confused why it's printing the backtrace in my logs every couple of seconds.

[mperham]

You are specifying 25 connections but your concurrency is 35. That’s not enough connections. Remove the size key so Sidekiq can manage the size internally (and correctly).

On Jun 13, 2018, at 15:32, kceb [e-mail] wrote:

@mperham Only error I see is a Timeout Error: https://d.pr/free/i/YRzD1i

Raised here

I've tried pinging redis in the console via:

  ::Sidekiq.redis do |r|
    res = r.ping
    res == 'PONG' ? '' : "Sidekiq.redis.ping returned #{res.inspect} instead of PONG"
  end

and it returns PONG. Also elasticache web dashboard doesn't show anything that would lead me to believe that it has reached a connection limit.

Also, the sidekiq queues are currently empty, so I'm confused why it's printing the backtrace in my logs every couple of seconds.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[kceb]

@mperham I did remove it after your initial suggestion above, sorry for not clarifying.

My sidekiq initializer:

# frozen_string_literal: true
Sidekiq::Web.use(Rack::Auth::Basic) do |user, password|
  [user, password] == [ENV['SIDEKIQ_USER'], ENV['SIDEKIQ_PASSWORD']]
end

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]

redis_conn = proc {
   Redis.new(url: "rediss://:#{ENV['REDIS_AUTH_TOKEN']}@#{ENV['REDIS_HOST']}:#{ENV['REDIS_PORT']}/12", password: ENV['REDIS_AUTH_TOKEN'])
}

#Print backtraces when exceptions occur
Sidekiq.default_worker_options = { 'backtrace' => ENV['SIDEKIQ_BACKTRACE'] == 'true' }

Sidekiq.configure_server do |config|
  config.redis = ConnectionPool.new(&redis_conn)
  config.logger = Le.new(ENV['LOGENTRIES_TOKEN'], tag: true, ssl: true)  unless Rails.env.test? || Rails.env.development?
end
Sidekiq.configure_client do |config|
  config.redis = ConnectionPool.new(&redis_conn)
  config.logger = Le.new(ENV['LOGENTRIES_TOKEN'], tag: true, ssl: true)  unless Rails.env.test? || Rails.env.development?
end

[mperham]

Ah, right. You are going too low-level. If you create the connection pool yourself, you must properly size it also. I need to update Sidekiq to verify proper connection pool sizing upon startup.

In other words, configure less and let Sidekiq handle it internally:

redis_conn =  {
   host: ENV['REDIS_HOST'],
   port: ENV['REDIS_PORT'],
   db: 12,
   password: ENV['REDIS_AUTH_TOKEN'],
}

Sidekiq.configure_server do |config|
  config.redis = redis_conn
  config.logger = Le.new(ENV['LOGENTRIES_TOKEN'], tag: true, ssl: true)  unless Rails.env.test? || Rails.env.development?
end
Sidekiq.configure_client do |config|
  config.redis = redis_conn
end

If you set REDIS_URL, you wouldn't need to do any Redis configuration at all.