Sidekiq: Workers sometimes can't find records

[chris-lee]

I have a message model with an after_create callback that queues up a sidekiq worker to email the message to the user.

In my rspec test I call .drain on the worker class to send the message and this works most of the time, but sometimes I get ActiveRecord::RecordNotFound errors. Since this worker is created via an after_create callback it should be there (and is in most test runs).

Any idea what's up?

class Message < ActiveRecord::Base
  after_create :send_new_message_email

  def send_new_message_email
    EmailMessageJob.perform_async id
  end
end

class EmailMessageJob
  include Sidekiq::Worker

  def perform(message_id)
    message = Message.find(message_id) <<<<< ActiveRecord::RecordNotFound

    # do stuff
  end
end

ruby 1.9.3 / rails 3.2.7

[mperham]

Use after_commit.

[bensie]

Use after_commit on: :create instead. Sidekiq is finding out about the job before the database record has it committed.

[chris-lee]

Thanks, guys. Unfortunately, that made one test worse and didn't help in another test. I updated my message model to:

class Message < ActiveRecord::Base
  after_commit :send_new_message_email, on: :create

  def send_new_message_email
    EmailMessageJob.perform_async id
  end
end

and now this block is failing:

        expect {
          Message.create(
            body: intro_message_body,
            sender_id: sender.id,
            recipient_id: recipient.id,
          )

        }.to change{ EmailMessageJob.jobs.size }.by(1)

because of:

result should have been changed by 1, but was changed by 0

Another one of my tests that uses selenium is still failing with the same RecordNotFound error.

Did I miss something?

[mperham]

It's essentially impossible to test after_commit in rspec because your tests normally run within one big transaction, thus after_commit is never called.

[chris-lee]

Ah, that makes sense. Shoulda clarified this was in the context of rspec. I'll add a sleep before calling .drain unless there's another way I should be handling that.

Thanks!

[denniskuczynski]

@chris-lee To have rspec examples not run in a transaction you'll have to turn off RSpec's transactional fixtures and use something like the DatabaseCleaner gem to cleanup your test database. I'm using this approach to markup which tests need transaction callbacks to run: http://www.denniskuczynski.com/2012/06/22/changing-individual-test-configuration-based-on-passed-in-options.html

[chris-lee]

Thanks, @denniskuczynski! I'll give that a shot.

[betelgeuse]

@denniskuczynski @chris-lee the option I use is monkey patching from here https://gist.github.com/1169763

[chris-lee]

Turns out the truncation was a bit of a red herring...

When running the full test suite there were other tests that created EmailMessageJobs, but the queue wasn't getting drained at the time. After those tests completed, the Messages associated with the jobs would get cleared out because of the transactions, but the job itself stayed in the redis queue. Later, when I called EmailMessageJob.drain it would try to run jobs for deleted messages. Boom.

My fix was to call EmailMessageJob.jobs.clear before my tests, but I was wondering if there's an equivalent command to clear jobs for all workers rather than one at a time.

[mperham]

Nope, there is no way to clear all jobs for all workers at once.

[runlevel5]

I bumped into the same issue with STI case:

class PrintJob < ActiveRecord::Base
  include Sidekiq::Worker

  after_commit :enqueue_job, :on => :create

  def enqueue_job
    # To be overridden
  end
end

class ChildOnePrintJob < PrintJob
  def perform(job_id)
    job = PrintJob.find(job_id)
    # do something with the job
  end

  private

    def enqueue_job
      Sidekiq::Client.enqueue(self.class, self.id)
    end
end


class ChildTwoPrintJob < PrintJob
  def perform(job_id)
    job = PrintJob.find(job_id)
    # do something with the job
  end

  private

    def enqueue_job
      Sidekiq::Client.enqueue(self.class, self.id)
    end
end

after_commit trick doesn't help at all. Any idea?

[mjc-gh]

You can manually run the after_commit callback for your unit tests by using run_callbacks(:commit). I have something like:

assert_difference('ImageWorker.jobs.size') do
  @img.save
  @img.run_callbacks(:commit)
end

Weebit hacky, but it works for unit test needs.

[sur]

For someone who might still be running in same issue:

Rails' after_commit needs to be used with precautions and understanding.

In my specific case, it was not useful to use the after_commit callback and I had to use the after_save (have to call on both create/update).

The issue can better be handled by using sidekiq's enqueuing:

earlier I had this:

after_save :notify
def notify
  MyMailer.notify(self, param1).deliver_later
end

which was having the above mentioned issues, however the sidekiq's retry mechanism was processing it successfully the next time but I wanted a clean way where it would not raise the error and should not queue in retries.
And this simple tweak fixed it:

after_save :notify
def notify
  MyMailer.notify(self, param1).deliver_later(wait: 30.seconds)
end

~ sur

[nickdowse]

Another option if you're on Rails 3/4 and having issues with the after_commit not running in specs is to use the test_after_commit gem: https://github.com/grosser/test_after_commit. (See also http://stackoverflow.com/questions/22988968/testing-after-commit-with-rspec-and-mocking)

[prashantsirohi04]

I am using rails 4.2 with sidekiq and facing the same issue as the original question, i have used after_commit :on_create and this is not in tests but in actual background job. Any suggestions?