Dd-trace-js: Use dd-trace to capture traces and send to jaeger

[shortcircuit3]

Describe the bug

I am able to capture the trace but the network request fails when sending to jaeger.

I have my jaeger container set up in docker compose under the name trace and the port is 6382. When using jaeger-client, i am able to send tracers, but i cant send traces with dd-trace.

web_1               | Encoding trace: [{"trace_id":"1834915692992758874","span_id":"1834915692992758874","parent_id":null,"name":"koa.request","resource":"GET /healthz","error":0,"meta":{"http.url":"http://localhost:8123/healthz","http.method":"GET","span.kind":"server","http.route":"/healthz","http.status_code":"200"},"metrics":{"_sample_rate":1,"_sampling_priority_v1":1},"start":1569338030173337300,"duration":22184814,"service":"api-core","type":"http"}]
web_1               | Adding encoded trace to buffer: 91 8c a8 74 72 61 63 65 5f 69 64 cf 19 76 ee 19 d9 9f 84 5a a7 73 70 61 6e 5f 69 64 cf 19 76 ee 19 d9 9f 84 5a a9 70 61 72 65 6e 74 5f 69 64 c0 a4 6e 61 6d 65 ab 6b 6f 61 2e 72 65 71 75 65 73 74 a8 72 65 73 6f 75 72 63 65 ac 47 45 54 20 2f 68 65 61 6c 74 68 7a a5 65 72 72 6f 72 00 a4 6d 65 74 61 85 a8 68 74 74 70 2e 75 72 6c bd 68 74 74 70 3a 2f 2f 6c 6f 63 61 6c 68 6f 73 74 3a 38 31 32 33 2f 68 65 61 6c 74 68 7a ab 68 74 74 70 2e 6d 65 74 68 6f 64 a3 47 45 54 a9 73 70 61 6e 2e 6b 69 6e 64 a6 73 65 72 76 65 72 aa 68 74 74 70 2e 72 6f 75 74 65 a8 2f 68 65 61 6c 74 68 7a b0 68 74 74 70 2e 73 74 61 74 75 73 5f 63 6f 64 65 a3 32 30 30 a7 6d 65 74 72 69 63 73 82 ac 5f 73 61 6d 70 6c 65 5f 72 61 74 65 01 b5 5f 73 61 6d 70 6c 69 6e 67 5f 70 72 69 6f 72 69 74 79 5f 76 31 01 a5 73 74 61 72 74 d3 15 c7 68 9f ac fe 37 00 a8 64 75 72 61 74 69 6f 6e d3 00 00 00 00 01 52 83 6e a7 73 65 72 76 69 63 65 a8 61 70 69 2d 63 6f 72 65 a4 74 79 70 65 a4 68 74 74 70

web_1               | Request to the agent: {"protocol":"http:","hostname":"trace","port":"6832","path":"/v0.4/traces","method":"PUT","headers":{"Content-Type":"application/msgpack","Datadog-Meta-Lang":"nodejs","Datadog-Meta-Lang-Version":"v11.8.0","Datadog-Meta-Lang-Interpreter":"v8","Datadog-Meta-Tracer-Version":"0.9.1","X-Datadog-Trace-Count":"2"}}
web_1               | Error: Network error trying to reach the agent: connect ECONNREFUSED 172.18.0.7:6832
web_1               |     at ClientRequest.req.on.e (/app/node_modules/dd-trace/src/platform/node/request.js:36:33)
web_1               |     at ClientRequest.emit (events.js:197:13)
web_1               |     at ClientRequest.EventEmitter.emit (domain.js:439:20)
web_1               |     at Socket.socketErrorListener (_http_client.js:399:9)
web_1               |     at Socket.emit (events.js:202:15)
web_1               |     at Socket.EventEmitter.emit (domain.js:439:20)
web_1               |     at emitErrorNT (internal/streams/destroy.js:82:8)
web_1               |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
web_1               |     at processTicksAndRejections (internal/process/next_tick.js:76:17)

My implementation with dd-trace

const opentracing = require('opentracing');
const ddTrace = require('dd-trace').init({
  service: 'api-core',
  debug: true,
  logInjection: true,
});

opentracing.initGlobalTracer(ddTrace);
const tracer = opentracing.globalTracer();

module.exports = tracer;

environment variables

- DD_TRACE_AGENT_HOSTNAME=trace
- DD_TRACE_AGENT_PORT=6832

jaeger container

  trace:
      container_name: jaeger
      environment:
          - COLLECTOR_ZIPKIN_HTTP_PORT=9411
      ports:
          - '5775:5775/udp'
          - '6831:6831/udp'
          - '6832:6832/udp'
          - '5778:5778'
          - '16686:16686'
          - '14268:14268'
          - '9411:9411'
      image: 'jaegertracing/all-in-one:1.6'

Environment

• Operation system:
  Mac
• Node version:
  Node v11.8.0
• Tracer version:
  0.9.1
• Agent version:

[shortcircuit3]

My guess is that "/v0.4/traces" does not exist in jaeger so its failing to send traces.

[rochdev]

Our tracer requires the Datadog agent to work and doesn't support interoperability with Jaeger. You can use Jaeger in parallel by using a Jaeger tracer, but in general this is not recommended unless you are transitioning from Jaeger.

[shortcircuit3]

@rochdev I thought dd-trace is supposed to support the opentracing standard?

https://www.datadoghq.com/blog/opentracing-datadog-cncf/

Seems backwards... 🤔

[rochdev]

OpenTracing is just an API that vendors can implement allowing you to switch between vendors. However, the implementation itself is bound to the vendor. This is different from OpenCensus and OpenTelemetry where the entire implementation is vendor-neutral.

This means that you can switch for example between dd-trace, a Jaeger or a Zipkin tracer and it would work as long as they are all OT compliant because then it means they have the same API. However, each tracer will have its own implementation and its own backend. This means you cannot use for example the Datadog tracer with the Jaeger backend, or the Jaeger tracer with the Datadog backend.

If you are looking for a vendor-neutral implementation that can support any vendor (or even multiple vendors simultaneously) I would recommend looking into OpenTelemetry which will replace both OpenTracing and OpenCensus in the coming months.

[shortcircuit3]

Thank you for the info

On Tue, Sep 24, 2019 at 4:11 PM, Roch Devost < [email protected] > wrote:

OpenTracing is just an API that vendors can implement allowing you to
switch between vendors. However, the implementation itself is bound to the
vendor. This is different from OpenCensus and OpenTelemetry where the
entire implementation is vendor-neutral.

This means that you can switch for example between dd-trace , a Jaeger or a
Zipkin tracer and it would work as long as they are all OT compliant
because then it means they have the same API. However, each tracer will
have its own implementation and its own backend. This means you cannot use
for example the Datadog tracer with the Jaeger backend, or the Jaeger
tracer with the Datadog backend.

If you are looking for a vendor-neutral implementation that can support
any vendor (or even multiple vendors simultaneously) I would recommend
looking into OpenTelemetry ( https://opentelemetry.io/ ) which will replace
both OpenTracing and OpenCensus in the coming months.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub (
https://github.com/DataDog/dd-trace-js/issues/694?email_source=notifications&email_token=AADG34QNX3K5GJEYMUTRGB3QLJX5LA5CNFSM4I2BESH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7PUWWY#issuecomment-534727515
) , or mute the thread (
https://github.com/notifications/unsubscribe-auth/AADG34TJVK3EM35SBD42J3LQLJX5LANCNFSM4I2BESHQ
).

[ab-pm]

I was facing the same question. Can't be that hard, right? Well, no, like @rochdev said the implementations are very different. After digging through the code bases of dd-trace and jaeger-client-node (I :heart: open source), the two vendors have different backend protocols and the libraries use wildly different internal data structures to implement the same OpenTracing interface.

Also the libraries have different feature sets. While both support startSpan(), addTags etc as specified by OpenTracing, Datadog provides best-in-class js library instrumentation (it was even forked by competitors like signalfx and lightstep) and also a scope manager I wouldn't want to live without. So while I could restrict myself to the OpenTracing JS API in my application and then conditionally swap out the datadog tracer by the jaeger one, I don't want to.

I still want to get a local dashboard for performance tracing though that I can use in development, without sending anything to datadog or even giving api keys to developers. Jaeger provides a all-in-one docker image which seemed perfect for this. So I went ahead and hacked the datadog tracer to use the jaeger reporting backend:

const ddTracer = require('dd-trace').init(…);
const jTracer = require('jaeger-client').initTracerFromEnv(…);

const remoteReporter = jTracer._reporter._reporters[1];
ddTracer._tracer._exporter = ddTracer._tracer._processor._exporter = {
    export(spans) { // formatted spans, https://github.com/DataDog/dd-trace-js/blob/v0.28.1/packages/dd-trace/src/format.js#L34-L45
        for (const span of spans) {
            if (blacklistSpanRegex.test(span.resource)) continue;
            remoteReporter.report({
                _operationName: span.name,
                _spanContext: {
                    traceId: Buffer.from(span.trace_id.toBuffer()),
                    spanId: Buffer.from(span.span_id.toBuffer()),
                    parentId: span.parent_id ? Buffer.from(span.parent_id.toBuffer()) : undefined,
                    flags: 0,
                },
                _startTime: span.start / 1e6, // microseconds to seconds
                // _logger: {},
                _duration: span.duration / 1e6, // nanoseconds to milliseconds
                _logs: [],
                _tags: Object.entries({
                    ...span.meta,
                    ...span.error ? {error: true} : null,
                    'resource.name': span.resource,
                    'span.type': span.type,
                    'service.name': span.service,
                }).map(([key, value]) => ({key, value})), // Array<Tag>,
                _references: [],
            });
        }
    }
};

Disclaimer: As can be seen from all the underscore-prefixed property names, this is expected to break at any point… and the hack is incomplete:

• no support for logs in spans. Datadog doesn't seem to implement Span.log anyway, instead structured logging is used to put span ids in log events.
• no support for baggage items. I have no idea how to use them, so I skipped this. Should be possible to add though.
• no support for references. Datadog seems to implement only childOf anyway.
• I have no idea what the flags that jaeger uses are good for, the data structure might be broken or also be missing other important parts.
• closing and flushing the tracer exporter/reporter are not supported, but probably not important.

Still, it works for now! Well, for some definition of "works" at least. Problems I found so far that lessen the experience I had hoped for:

• Jaeger has no concept of resources like Datadog. I've put them in the tags, but there's no UI focus on them
• Jaeger puts the service name in a "process" structure. The whole reporter is initialised with the service name (and tags), so the service name cannot change between spans in the same trace.
• The Jaeger UI is inferior compared to the Datadog UI