Dd-trace-rb: When on a host that does not have ddtrace setup, the reporter repeatedly tries to send instrumentation

[NullVoxPopuli]

This nearly makes rails c on a staging environment unusable.

Is there a way to disable this 'auto-retry'?

[ufoot]

Thanks for the feedback. Indeed this auto-retrying can quickly fill your logs. OTOH it's legit to retry, the agent could show up after "some time". FYI the enabled flag of the tracer can possibly help you in that case.

Either you disable it by (general case):

Datadog.tracer.enabled = false

Or (Rails specific):

Rails.configuration.datadog_trace = {
  enabled: false
}

This should keep your code instrumented, but won't send data to the agent. A common pattern is to set this flag according to some environment variable, or any contextual info which enables you to differentiate your prod and staging instances. Do not hesitate to contact our support if in doubt.

[NullVoxPopuli]

cool, thanks

I think I may be able to set enabled=false based on

defined? Rails::Console
=> "constant"
defined? Rails::Server
=> nil

I'll try it out, and see how it goes

[NullVoxPopuli]

This works wonderfully:

  Rails.configuration.datadog_trace = {
    auto_instrument: true,
    auto_instrument_redis: true,
    default_service: 'rails-app',
    env: Rails.env,
    enabled: !defined?(Rails::Console)
  }

[sullimander]

I also have this problem when running rake tasks against environments I have instrumented. Anyone have a solution to disable the tracer for Rake tasks?

[palazzem]

Hello @skisulli! does the solution proposed above, helps you to solve the problem? otherwise, can you give us some snippets or more details about your issue?

[sullimander]

The solution above works fine for use in the Rails console, but when I run a rake task that uses the rails environment, I get output like the following. Rails::Console is not defined in a rake task.

In this specific example, I have a rake task that clears a few dynamodb tables that are used only for caching purposes. The command prompts the user if they want to run this in qa, staging, or production. The output below is while the command is waiting for user response. It just keeps printing that error over and over until the command finishes.

$ RAILS_ENV=qa rake dynamodb:clear
Do you really want to run this in qa? (Yes/no)
E, [2017-08-29T10:58:27.841575 #15094] ERROR -- ddtrace: [ddtrace] (/usr/local/opt/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ddtrace-0.8.1/lib/ddtrace/transport.rb:79:in `rescue in post') Failed to open TCP connection to localhost:8126 (Connection refused - connect(2) for "localhost" port 8126)
E, [2017-08-29T10:58:27.842922 #15094] ERROR -- ddtrace: [ddtrace] (/usr/local/opt/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ddtrace-0.8.1/lib/ddtrace/transport.rb:79:in `rescue in post') Failed to open TCP connection to localhost:8126 (Connection refused - connect(2) for "localhost" port 8126)
E, [2017-08-29T10:58:28.847365 #15094] ERROR -- ddtrace: [ddtrace] (/usr/local/opt/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ddtrace-0.8.1/lib/ddtrace/transport.rb:79:in `rescue in post') Failed to open TCP connection to localhost:8126 (Connection refused - connect(2) for "localhost" port 8126)
E, [2017-08-29T10:58:29.852478 #15094] ERROR -- ddtrace: [ddtrace] (/usr/local/opt/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ddtrace-0.8.1/lib/ddtrace/transport.rb:79:in `rescue in post') Failed to open TCP connection to localhost:8126 (Connection refused - connect(2) for "localhost" port 8126)
E, [2017-08-29T10:58:30.857489 #15094] ERROR -- ddtrace: [ddtrace] (/usr/local/opt/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ddtrace-0.8.1/lib/ddtrace/transport.rb:79:in `rescue in post') Failed to open TCP connection to localhost:8126 (Connection refused - connect(2) for "localhost" port 8126)
.....

[palazzem]

Thanks for the logs! Let me think about a possible solution here, because at the moment we offer a way to disable auto instrumentation via env variables: http://gems.datadoghq.com/trace/docs/#label-Disabling+Rails+auto-instrumentation

In the meantime, I'll check if we can do something better to exclude automatically the instrumentation when rake is used (though sometimes developers may want to instrument also rake commands).

Let me know @skisulli if that helps you while we investigate the issue!

[jjoos]

I had this issue too, because we set DISABLE_DATADOG_RAILS for our test environment, the code in /lib/ddagent.rb that sets the configuration isn't excecuted so the option enabled defaults to true and our tests start sending traces.

[palazzem]

@jjoos we're currently fixing the problem with a new way to configure both the Tracer and Rails. Now requiring the ddtrace doesn't have side effects, and the new configuration looks like: https://github.com/DataDog/dd-trace-rb/pull/256

We'll add the same approach for the Tracer so that:

• integrations configuration are delayed
• Tracer configuration is done immediately

Will keep you updated!

[palazzem]

Hello everyone! We've done a huge refactoring with our latest changes and the new configuration block is now available. It is part of our 0.11.0 release that will be shipped in the next couple of days. The problem has been addressed as I wrote in my previous comment. I will write you when the version is released so that you can try it immediately.

Thank you very much!