Sidekiq: ActiveRecord::ConnectionTimeoutError with Heroku

[kelhusseiny]

I've a repeatedly ActiveRecord::ConnectionTimeoutError with Sidekiq in Heroku. here is the full error:

ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.008 seconds); all pooled connections were in use

• I'm using 1 dyno for Sidekiq with size Performance L, with a concurrency of 15
• For the web server, I'm using 1 dyno with 2 puma workers and 15 threads each so the total is (1 * 2 * 15 = 30).
• My DB pool size is 20 (I'm using Standard-0 postgres which have maximum 120 connections).

So how am I getting this error while the connection pool size is greater than the maximum number of connection that could be used? total of (15 + 40 = 45) from 120

I tried to run heroku pg:info while this error is happening and I got this result

Connections:        20/120

here is my configs files:

• sidekiq.yml

concurrency: 15
:queues:
  - myapp

• config/sidekiq.rb

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end
end

Sidekiq.configure_server do |config|
  config.redis = { url: ENV["REDISTOGO_URL"] } if ENV["REDISTOGO_URL"].present?

  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
  end
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV["REDISTOGO_URL"] } if ENV["REDISTOGO_URL"].present?
end

if Rails.env.production?
  Sidekiq.configure_server do |config|
    config.redis = { url: ENV["REDISTOGO_URL"] }
  end

  Sidekiq.configure_client do |config|
    config.redis = { url: ENV["REDISTOGO_URL"] }
  end
end

Sidekiq.default_worker_options = {
  queue: "myapp",
  backtrace: true,
  retry: false
}

• config/puma

workers Integer(ENV["WEB_CONCURRENCY"] || 2)
threads_count = Integer(ENV["RAILS_MAX_THREADS"] || 15)
threads threads_count, threads_count

preload_app!

rackup      DefaultRackup
port        ENV["PORT"]     || 3000
environment ENV["RACK_ENV"] || "development"

on_worker_boot do
  ActiveRecord::Base.establish_connection
end

• database.yml

production:
  pool: 20

[mperham]

No idea.

[nas887]

@mperham I've been seeing this as well. Any new thoughts as time has gone by?

[mperham]

It's likely you have a DB connection leak in your app code.

[alecrubin]

@mperham @nas887 I just started seeing this today too. Perhaps a heroku DB issue?

[LucasCioffi]

@Azzurrio I read this at the bottom of https://devcenter.heroku.com/articles/concurrency-and-database-connections

"If you are using a worker process type and using a background worker library like Sidekiq you may want to have your settings be different in different dyno types. By default Sidekiq uses 25 threads which means that either your database on your worker will need to be 25+ threads or you will need to configure your Sidekiq process to use fewer threads."

The "Background Worker" part of that page explains how to do that:

If you have your Rails app configured from the previous sections, you can set RAILS_MAX_THREADS to a different value. For example:

worker: RAILS_MAX_THREADS=${SIDEKIQ_RAILS_MAX_THREADS:-25} bundle exec sidekiq
In this example you would need to heroku config:set SIDEKIQ_RAILS_MAX_THREADS=25.

Instead if you wanted to change the number of connections in Sidekiq you can boot it with the -c flag and use a different configuration variable such as SIDEKIQ_CONCURRENCY.

worker: bundle exec sidekiq -c ${SIDEKIQ_CONCURRENCY:-5}
In this example we are telling Sidekiq to only use 5 threads to process background jobs.