Sentry-ruby: Raven Ruby + ActiveJob can kill BG processing

[mrhead]

We experienced a strange issue where Raven + ActiveJob consumed all of our Redis memory by creating jobs with enormous payload size. This can happen when a background job is scheduled with a model in arguments, but this model doesn't exist when the job is executed AND when SentryJob inherits from ActiveJob.

Environment

• Rails 5.0.2
• sentry-raven 2.3.1
• Sidekiq 4.2.9

Steps to reproduce

# config/initializers/raven.rb
Raven.configure do |config|
  config.dsn = 'http://public:[email protected]/project-id'

  config.async = lambda { |event|
    SentryJob.perform_later(event.to_hash)
  }
end

# app/jobs/sentry_job.rb
class SentryJob < ApplicationJob
  queue_as :default

  def perform(event)
    Raven.send_event(event)
  end
end

# app/jobs/test_job.rb
class TestJob < ApplicationJob
  queue_as :default

  def perform(user)
    # Do something later
  end
end

In rails console:

user = User.create!(name: "John Doe")
TestJob.perform_later(user)
user.destroy!

Start Sidekiq and watch the apocalypse:

bundle exec sidekiq

Example application with steps to reproduce

https://github.com/mrhead/sentry-raven-active-job

Expected result

TestJob should fail with ActiveJob::DeserializationError and SentryJob should successfully report this failure.

Actual result

TestJob fails with ActiveJob::DeserializationError, then SentryJob is scheduled but it fails with ActiveJob::DeserializationError too. This failure event schedules another SentryJob with the payload size double of the original one. And this goes over and over again.

My guess is that SentryJob fails, because it contains a global ID url in the payload and ActiveJob is trying to load that object even if it wasn't directly provided as an argument.

Workaround

Do not use ActiveJob for SentryJob.

[nateberkopec]

Makes sense. Sending enormous payloads to Sentry isn't good either, so it's reasonable that we should do some additional prevention here.

[mrhead]

The thing is that nothing is sent to Sentry at all. Because every SentryJob fails immediately and it also schedules another SentryJob. So now there are two jobs and both of them will fail again and schedule new jobs (with bigger payloads) again.

[nateberkopec]

Also, the real workaround here is "don't use models as background job arguments". But still, Raven can probably do some work here to prevent endless loops.

[nateberkopec]

The only solution I can think of here is to add ActiveJob::DeserializationError to the ignored exception list.

[nateberkopec]

Or perhaps a little better:

class SentryTestJob < ApplicationJob
  queue_as :default
  rescue_from(ActiveJob::DeserializationError) { |e| Rails.logger.error e }

  def perform(some_id
    result = SentryTest.new(some_id).call
  end
end

[mrhead]

@nateberkopec Doesn't this mean that ActiveJob::DeserializationError will be never send to Sentry?

[nateberkopec]

Yes. The best option is probably to use your queue backend directly rather than ActiveJob.

[mrhead]

Yes, that is what we did and everything is working well since then :).

[nateberkopec]

I'm starting to wonder if we shouldn't just ignore ActiveJob::DeserializationError. It's essentially ActiveRecord::RecordNotFound, which we already ignore. Opening new issue.

[ndbroadbent]

Just confirming that this is not an issue if you skip ActiveJob and use Sidekiq directly:

class SentryJob
  include Sidekiq::Worker

  def perform(event)
    Raven.send_event(event)
  end
end

[ndbroadbent]

Oh I just saw this: #701. ActiveJob::DeserializationError is ignored by default now.

[mikecmpbll]

we've had the same issue here but from a different exception. isn't the thing they all have in common that the 2nd exception comes from inside raven? the first exception is often an entirely reasonable exception that you don't want to ignore.

shouldn't raven not capture exceptions that are from within raven (or at least from within raven's exception handling parts ...)? from the naivity of not knowing anything about raven internals or how complicated that might be to implement, that sounds like the right approach to me.