Sidekiq: PG::UnableToSend: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

[igorkasyanchuk]

lease help resolve issue with

Message PG::UnableToSend: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. Traceback PG::UnableToSend: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

full log: https://gist.githubusercontent.com/igorkasyanchuk/cc96dba31b0b0f83b77faadf077433e8/raw/70af94105b0a7649b6ad0dde086ac539902d1d3f/gistfile1.txt

we have 2 app servers + 1 db. we have pool 25 and workers 20, rails, sidekiq, pg.

it happen only with sidekiq background workers.

thanks for advices

[mperham]

StackOverflow is the right place to get help with application config issues.

[pmichaeljones]

Hi Mike. I know this is an old issue that has nothing to do with Sidekiq, but this might help others that are looking for the same bug. Congrats on reaching Sidekiq's 6th birthday as well.

I have a super simple Sidekiq job that uploads a file to Filestack, using their Ruby SDK. After they return the URL for the saved file, I save it in my DB. Everything goes smoothly until I try to save the new record in my PostgreSQL database. I've gotten every type of PG error possible: PG::ConnectionBad: PQsocket() can't get socket descriptor:, PG::UnableToSend, PG::ConnectionBad: PQconsumeInput(), PG::UnableToSend: SSL connection has been closed unexpectedly. This probably means the server terminated abnormally before or while processing the request. You name it, I got it.

I've been haunted by this bug for two days and have wasted a solid 10 hours fixing it. In reality, I've learned a ton about AR in those 10 hours. Turns out, in my case, it was because the Parallel gem, which Filestack uses, was finishing its API call, and then closing the AR connection being used.

I did absolutely everything everyone recommended for this bug on SO, adjusting connection pool, reaper frequency, TCP idle and timeout, nothing worked. I eventually hacked the solution below and it worked, but it smelled very bad.

class FileUploadWorker
  include Sidekiq::Worker

  def perform(user_id)
    # Filestack uploading here

    ActiveRecord::Base.connection_pool.release_connection
    ActiveRecord::Base.connection_pool.with_connection do
      # save record in database
    end
  end
end

[Here's the Parallel issue:] (https://github.com/grosser/parallel/issues/62)

Anyways, if any other developers out there stumble upon this comment, take a look if you're using the Parallel gem and either use a different gem or use one of the solutions they propose in Issue 62.

I'm going to write a blog post about this because it was such a bugger to find.

[mperham]

Yep, you are using parallel to fork multiple processes, those processes are all trying to use the same socket at the same time. This is a race condition and leads to unpredictable, weird errors. Don't use parallel without resetting any necessary sockets in each forked copy.