Sidekiq: Sidekiq worker failed: could not obtain a database connection within 5 seconds (waited 5.003581 seconds)

[krishnanandnb]

We saw this issue in one of our prod servers. While trying to reproduce the issue in local we came across this scenario where Sidekiq may require more db connections than the concurrency
Sample worker
`class HardWorker < BaseWorker

def perform(name, count)
# do something
puts name
puts count
User.first

sleep(10)

end
end`

100.times { HardWorker.perform_async('bob', 5)}

we get this error upon execution
Sidekiq worker failed: could not obtain a database connection within 5 seconds (waited 5.003581 seconds)

Both database pool size and Sidekiq concurrency are 5

Version of sidekiq

sidekiq (4.2.10)
  concurrent-ruby (~> 1.0)
  connection_pool (~> 2.2, >= 2.2.0)
  rack-protection (>= 1.5.0)
  redis (~> 3.2, >= 3.2.1)
sidekiq-pro (3.7.1)
  sidekiq (>= 4.1.5)

Rails

rails (= 3.2.22.5)

[mperham]

That’s a Rails issue where the main thread can take a connection during boot. Bump your pool to 6.

On Sep 9, 2019, at 22:14, Krishnanand nb [e-mail] wrote:

We saw this issue in one of our prod servers. While trying to reproduce the issue in local we came across this scenario where Sidekiq may require more db connections than the concurrency
Sample worker
`class HardWorker < BaseWorker

def perform(name, count)

do something

puts name
puts count
User.first

sleep(10)
end
end`

100.times { HardWorker.perform_async('bob', 5)}

we get this error upon execution
Sidekiq worker failed: could not obtain a database connection within 5 seconds (waited 5.003581 seconds)

Both database pool size and Sidekiq concurrency are 5

Version of sidekiq

sidekiq (4.2.10)
concurrent-ruby (~> 1.0)
connection_pool (~> 2.2, >= 2.2.0)
rack-protection (>= 1.5.0)
redis (~> 3.2, >= 3.2.1)
sidekiq-pro (3.7.1)
sidekiq (>= 4.1.5)
Rails

rails (= 3.2.22.5)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[krishnanandnb]

That’s a Rails issue where the main thread can take a connection during boot. Bump your pool to 6.
…
On Sep 9, 2019, at 22:14, Krishnanand nb @.*> wrote: We saw this issue in one of our prod servers. While trying to reproduce the issue in local we came across this scenario where Sidekiq may require more db connections than the concurrency Sample worker class HardWorker < BaseWorker def perform(name, count) # do something puts name puts count User.first sleep(10) end end 100.times { HardWorker.perform_async('bob', 5)} we get this error upon execution Sidekiq worker failed: could not obtain a database connection within 5 seconds (waited 5.003581 seconds) Both database pool size and Sidekiq concurrency are 5 Version of sidekiq sidekiq (4.2.10) concurrent-ruby (~> 1.0) connection_pool (~> 2.2, >= 2.2.0) rack-protection (>= 1.5.0) redis (~> 3.2, >= 3.2.1) sidekiq-pro (3.7.1) sidekiq (>= 4.1.5) Rails rails (= 3.2.22.5) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Does this happen in all the rails version?

[mperham]

I believe it was fixed in Rails 5.2.

On Sep 10, 2019, at 04:37, Krishnanand nb [e-mail] wrote:

That’s a Rails issue where the main thread can take a connection during boot. Bump your pool to 6.
…
On Sep 9, 2019, at 22:14, Krishnanand nb @.*> wrote: We saw this issue in one of our prod servers. While trying to reproduce the issue in local we came across this scenario where Sidekiq may require more db connections than the concurrency Sample worker class HardWorker < BaseWorker def perform(name, count) # do something puts name puts count User.first sleep(10) end end 100.times { HardWorker.perform_async('bob', 5)} we get this error upon execution Sidekiq worker failed: could not obtain a database connection within 5 seconds (waited 5.003581 seconds) Both database pool size and Sidekiq concurrency are 5 Version of sidekiq sidekiq (4.2.10) concurrent-ruby (> 1.0) connection_pool (> 2.2, >= 2.2.0) rack-protection (>= 1.5.0) redis (~> 3.2, >= 3.2.1) sidekiq-pro (3.7.1) sidekiq (>= 4.1.5) Rails rails (= 3.2.22.5) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Does this happen in all the rails version?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[krishnanandnb]

Thank you did try with connection pool as 6 and did not see any errors. This is fixed then

[tedyangx]

@krishnanandnb Do you know the link to the fix in Rails 5.2 on this issue? (We also encountered this problem in Rails 5.0 in our project.)

[krishnanandnb]

@tedyangx nope. But it has to do with the way rails loads the schema cache and populates the controller to DB model attribute values

[sergey-alekseev]

This was fixed in https://github.com/rails/rails/pull/28057 I suppose, but probably https://github.com/rails/rails/pull/31221 is somewhat related as well. See a discussion in https://github.com/mperham/sidekiq/issues/3879.