Dd-trace-rb: Infinite loop with some service (eg. Rails) patchers +`Module#prepend` elsewhere

[armstrjare]

The tracer instruments using alias_method in many cases through the Rails stack.
If any other library has used Module#prepend over the same method, this seems to cause an infinite loop.

In our case, we're currently use DD APM + Scout APM at the same time. Scout APM recently swapped to using Module#prepend for Rails 6 compatibility (which seems to now use Module#prepend over some methods that they instrument in their tracer)

eg:

 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
 scout_apm (2.4.24) lib/scout_apm/instruments/action_view.rb:128:in `render_template'
 ddtrace (0.20.0) lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'

[delner]

Thanks for the report. We plan on migrating to a prepend strategy as well: I think that should prevent this from happening. Will add this to the backlog.

[chancelang]

Running into the same issue using jb gem, any workarounds?

14:17:33 worker.1 | 2019-03-22T14:17:33.142Z 9 TID-gog7so38o WARN: /var/bundle/ruby/2.5.0/gems/ddtrace-0.19.0/lib/ddtrace/registry.rb:28:in `synchronize'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/ddtrace-0.19.0/lib/ddtrace/registry.rb:28:in `[]'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/ddtrace-0.19.0/lib/ddtrace/configuration.rb:55:in `fetch_integration'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/ddtrace-0.19.0/lib/ddtrace/configuration.rb:16:in `[]'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/ddtrace-0.19.0/lib/ddtrace/contrib/rails/utils.rb:14:in `normalize_template_name'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/ddtrace-0.19.0/lib/ddtrace/contrib/rails/core_extensions.rb:55:in `render_template_with_datadog'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/jb-0.6.1/lib/jb/action_view_monkeys.rb:48:in `render_template'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/ddtrace-0.19.0/lib/ddtrace/contrib/rails/core_extensions.rb:80:in `render_template_with_datadog'
14:17:33 worker.1 | /var/bundle/ruby/2.5.0/gems/jb-0.6.1/lib/jb/action_view_monkeys.rb:48:in `render_template'
...
...

[delner]

I think the prepend strategy should help, as mentioned above.

(@chancelang Also, side note, but I would highly recommend upgrading to at least 0.19.1. It won't fix this specific issue, but it will add another critical bugfix for an issue in 0.19.0.)

[chancelang]

@delner Thank you very much for the quick response! We had to remove the jb gem to get moving again. Also upgraded to the new version of dd-trace :)

[armstrjare]

Is there any timeline on the prepend strategy being implemented?

I haven’t tested ddtrace with Rails 6 yet - but I know this was why Scout
APM had to make this change as there is framework usage of prepend on
instrumented methods.

Has ddtrace been validated in Rails 6 yet?

On Sat, 23 Mar 2019 at 9:23 AM, chancelang [e-mail] wrote:

@delner https://github.com/delner Thank you very much for the quick
response! We had to remove the jb gem to get moving again. Also upgraded to
the new version of dd-trace :)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/DataDog/dd-trace-rb/issues/723#issuecomment-475768799,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAF3vC6bDf_Vw0ZMquCBHwr3udmhyleRks5vZTu7gaJpZM4b9dQz
.

[delner]

No, it has not been verified with Rails 6.

We do have an overhaul of Rails on our backlog for the near future, but I unfortunately don't have any dates or expectations for when that would happen right now. My sense is if ddtrace-rb doesn't work with Rails 6, we'd likely escalate its priority. I'd be curious to see if that's the case, and if it is, I can raise that to our team.

[vdhpieter]

Any updates? We're still stuck at version 19.1 because of this

[delner]

The overhaul of the Rails integration is in progress, which should set us up to change to a prepend strategy for Ruby 2.0+: that should help resolve this particular problem. I will try to escalate this one after that refactor is done, given we recognize this is really important to many Rails users.

[chadwilken]

Any chance you could cut a hotfix release using the workaround/patch here

[delner]

Unfortunately we can't release a hotfix for this right now because prepend is Ruby 2.0+ and we support Ruby 1.9.3.

However, we have just deprecated Ruby 1.9.3 support in 0.25.0 and will remove it in 0.27.0, at which point we should be able to freely incorporate this prepend strategy. We've actually already begun a refactor for Rails to put us in a position to do just this in 0.27.0, which hopefully we'll be working on within the next month or so.

[delner]

Hey folks, we just released 0.27.0, which updated our Rails patching to use prepend, which should hopefully prevent this from happening. You can try it out, see if it helps.

Although we haven't officially added support for Rails 6 yet, we should start on it very soon, so look for that in a near future release. I'll update you when I have more details on that.

Going to close this for now, but feel free to re-open/comment if this looping issue isn't resolved with the latest version.

[marcotc]

@armstrjare, just to let you know that we've released 0.28.0 with Rails 6 support. Everything should be converted to prepend and tested with the newest version of Rails.

Let us know if the issue persists.