Sentry-ruby: sentry-rails 4.3.0 triggers endless series of transaction traces when using Sentry::SendEventJob

[anderscarling]

Describe the bug

Submitting an error using Sentry::SendEventJob will cause an infinite series of transaction traces to be submitted to sentry.

Error goes away when appending relevant lines in Sentry::Rails::ActiveJobExtensions#capture_and_reraise_with_sentry (those added in #1304) with unless job.class == Sentry::SendEventJob.

To Reproduce

Setup sentry with:

  config.async = lambda do |event, hint|
    Sentry::SendEventJob.perform_later(event, hint)
  end

Cause an exception to be reported.

Expected behavior

Sentry::SendEventJob should report the error, then do no more work.

Actual behavior

For each run of Sentry::SendEventJob a transaction trace will be created, which will trigger another Sentry::SendEventJob to submit the transaction. This will repeat forever.

Environment

• Ruby Version: 3.0.0
• SDK Version: 4.3.0
• Integration Versions (if any):
  
  
  
  • e.g. Rails 6.1
  
  

[st0012]

@anderscarling thanks for reporting this, we've released sentry-rails 4.3.1 to address the issue 🙂

[wladimirgramacho]

@st0012 I just tried to update the sentry-rails gem to 4.3.1 and the erros still remain. I have the same config @anderscarling showed in the description. Here is the sentry error. We are holding onto the 4.2.2 version

[st0012]

@wladimirgramacho do you still receive transactions of SendEventJob after upgrading the gem? the issue you describe looks like a different issue, which probably happens like this:

1. your background jobs also create transactions after the upgrade
2. the high number (I assume) of background job transactions triggers rate limit
3. the SendEventJob worker fails because of the 429 response
4. the exception enqueues another SendEventJob
   
   
   
   • this is a bug because Sentry::ExternalError shouldn't be sent to Sentry. I'll fix it.
   
   
5. 3 and 4 repeats

can you help me confirm the theory by setting this:

  config.traces_sampler = lambda do |sampling_context|
    transaction_context = sampling_context[:transaction_context]

    op = transaction_context[:op]
    if op.match? /request/
      0.3 # set the rate you originally have
    elsif op.match? /active_job/ || op.match? /sidekiq/
      0.01 # set a lower rate for background jobs
    end
  end

and see

• does the rate limiting error occur again?
• do you still see tracing data for SendEventJob

[st0012]

@anderscarling @wladimirgramacho sorry I just found out that the PR was missed from the 4.3.1 release 🤦‍♂️ I'll make 4.3.2 release as soon as possible.

[st0012]

sorry for delay, the original fix has been released in 4.3.2 now.

@wladimirgramacho please let me know if you still see the same problem after the upgrade 🙂

[wladimirgramacho]

@st0012 We have just upgraded and the errors happened again 😕

The error that happens is due to the Sentry::ExternalError. Do you have any updates on this bug fix?

[anderscarling]

It seems to work as expected for me now, so could be a different issue.

[st0012]

@wladimirgramacho the fix is on sentry-ruby's master already. but it'll be released later next week with other corrections.
here's a quick fix for it:

Sentry.init do |config|
  config.excluded_exceptions << "Sentry::ExternalError" # TODO: Remove this line after upgrading sentry-ruby
end