Dd-trace-rb: Datadog not using hostname but 127.0.0.1

[mhenrixon]

ERROR -- ddtrace: [ddtrace] (/usr/local/bundle/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/client.rb:35:in rescue in send_request') Internal error during HTTP transport request. Cause: Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126) Location: /usr/local/lib/ruby/2.5.0/net/http.rb:939:inrescue in block in connect'

def configure_tracer(config)
  config.tracer          = ::Datadog::Tracer.new
  config.tracer.enabled  = true
  config.tracer.hostname = tracer_params[:hostname]
  config.tracer.port     = tracer_params[:port]
  config.tracer.env      = tracer_params[:env] if config.tracer.respond_to?(:env=)
  config.tracer.transport_options = proc do |t|
    # Hostname, port, and additional options. :timeout is in seconds.
    t.adapter :net_http, tracer_params[:hostname], tracer_params[:port], { timeout: tracer_params[:timeout] }
  end
end

If I do a puts on tracer_params[:hostname] it is set to datadog-agent.datadog but somehow something is still trying to push to 127.0.0.1.

Another weird thing is that unless I set the tracer = the tracer is nil. I am currently on version 0.36.0 of the gem.

[mhenrixon]

So I tried to raise an error inside :

/gem # bin/rspec spec/integration/
Hello world
.#<Thread:0x000055e66672d038@/gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/workers.rb:73 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
    35: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/workers.rb:73:in `block (2 levels) in start'
    34: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/workers.rb:107:in `perform'
    33: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/workers.rb:107:in `loop'
    32: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/workers.rb:108:in `block in perform'
    31: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/workers.rb:55:in `callback_traces'
    30: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/writer.rb:49:in `block in start'
    29: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/writer.rb:76:in `send_spans'
    28: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `send_traces'
    27: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `to_a'
    26: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `each'
    25: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `each'
    24: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `each'
    23: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `each'
    22: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `each'
    21: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:121:in `<<'
    20: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/traces.rb:115:in `block in send_traces'
    19: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/traces.rb:27:in `send_payload'
    18: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/client.rb:22:in `send_request'
    17: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/traces.rb:28:in `block in send_payload'
    16: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/traces.rb:70:in `send_traces'
    15: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/traces.rb:44:in `send_traces'
    14: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/traces.rb:117:in `call'
    13: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/api/endpoint.rb:21:in `call'
    12: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/traces.rb:71:in `block in send_traces'
    11: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/api/instance.rb:27:in `call'
    10: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/adapters/net.rb:31:in `call'
     9: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/adapters/net.rb:42:in `post'
     8: from /gem/vendor/bundle/ruby/2.5.0/gems/ddtrace-0.36.0/lib/ddtrace/transport/http/adapters/net.rb:24:in `open'
     7: from /usr/local/lib/ruby/2.5.0/net/http.rb:609:in `start'
     6: from /usr/local/lib/ruby/2.5.0/net/http.rb:909:in `start'
     5: from /usr/local/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
     4: from /usr/local/lib/ruby/2.5.0/net/http.rb:935:in `connect'
     3: from /usr/local/lib/ruby/2.5.0/timeout.rb:103:in `timeout'
     2: from /usr/local/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'
     1: from /usr/local/lib/ruby/2.5.0/net/http.rb:936:in `block in connect'
/usr/local/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect': Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126) (Errno::ECONNREFUSED)

Even if I hard code the configuration block at the start of everything with the correct HOSTNAME I still get this problem. Why? To me it seems like the argument hostname isn't passed down properly to the http adapter.

[mhenrixon]

The only thing I can think of, is that there is a race condition for something that uses a default value of 127.0.0.1. I've searched our entire code base and can't find a single instance of that or localhost.

Any help greatly appreciated.

[delner]

@mhenrixon This might be related to an issue of a stale tracer instances, one configured with default settings (127.0.0.1) that the instrumentation is using, instead of the properly configured one. If this is a Rails application you can try #1064 to see if it resolves the issue. If that doesn't help, please let us know so we can look at this further.

[mhenrixon]

Unfortunately, it doesn't help me @delner. We have a gem that handles all the configuration of datadog to avoid duplicating the effort in every service. Maybe we should just scrap the gem and copy the single file into each project that needs it.

[mhenrixon]

Does something start when I require 'ddtrace'? That's the only thing I can think of to be honest. Let me try with the environment variables and see if I can bypass the issue with those.

[mhenrixon]

I tried with setting DD_AGENT_HOST=datadog-agent.datadog in the environment variables before require 'ddtrace' unfortunately that also doesn't solve the issue.

Actually now I get:

ERROR -- ddtrace: [ddtrace] (/usr/local/bundle/bundler/gems/dd-trace-rb-0625ccc10682/lib/ddtrace/transport/http/client.rb:35:in rescue in send_request') Internal error during HTTP transport request. Cause: Failed to open TCP connection to datadog-agent.datadog:8126 (getaddrinfo: Name does not resolve) Location: /usr/local/lib/ruby/2.5.0/net/http.rb:939:inrescue in block in connect'

So it seems like this PR might have broken something? https://github.com/DataDog/dd-trace-rb/pull/631

[mhenrixon]

And finally after studying some examples I have it connected and working thanks to the following code:

# === IMPORTANT ===
#     Due to a bug in the ddtrace gem it starts some worker when the gem is required.
#     Unless we make sure these environment variables are set we will log errors
# === IMPORTANT ===
ENV['DD_AGENT_HOST'] ||= 'datadog-agent.datadog'
ENV['DD_TRACE_AGENT_PORT'] ||= 8126
ENV['DD_METRIC_AGENT_PORT'] ||= 8126

require 'ddtrace'
require 'datadog/statsd'

Seems like it shouldn't have to be needed but it does the trick for me.

[delner]

@mhenrixon It shouldn't be required to do that, but that would definitely be an effective workaround, because whatever stale tracer you have producing instrumentation when constructed will inherit those ENV vars by default, if present.

This is definitely a bug; we're aware of a problem with stale tracers, and its probable that your issue is related to this problem. It's being worked on but the fix is not yet ready. If this work around helps in the meantime, then I encourage you to do so.

[delner]

We've opened #1072 to track "stale tracers" which we think is the source of this issue. I'm going to close this issue for now and move the discussion over there. When we get a PR fix up, if said fix doesn't fully resolve this issue, we can re-open this issue again and address its individual circumstances.