Dd-trace-js: Add support for ES modules

[rochdev]

Right now we rely on require to patch modules automatically. We should also support ES modules as well as the esm module.

[luax]

Any news on this? 🙂

We're experiencing some issues instrumenting redis using a node app with only native ES modules (package.json has type "module")

app.js:

import 'dotenv/config.js';
import './ddTracer.js'; // This line must come before importing any instrumented module.
import redis from 'redis';
const app = express()
// redis stuff
....

ddTracer.js:

import trace from 'dd-trace';
const tracer = trace.init({ plugins: true });
export default tracer;

Manually enabling redis does not work either:

tracer.use('redis', { service: 'redis' });

The plugins http-client, dns, tcp and express seemingly works.

Thanks 🙏

[rochdev]

This is still on our roadmap but we're not actively working on it. What is your use case for using ES modules?

[luax]

Thank you @rochdev. I would not say that there is one particular use case but the app is only running modern native Node.js and JS functionality and not any transpiled code.

[provstevi]

We're having a similar issue when using koa and redis.

It seems I can either require and init dd-trace in the main server.js file and get redis tracing but lose endpoints being tagged automatically,
OR
I can import './datadog'; a file with import datadogTrace from 'dd-trace'; and get automatic endpoint tagged as expected, but redis will be disabled with the trace error TypeError: Cannot read property 'prototype' of undefined at Instrumenter.patch (/app/app/node_modules/dd-trace/packages/datadog-plugin-redis/src/index.js:57:35)

For now we're erring on the side of actually getting redis tracing, but I'd really like to not have to manually tag my endpoints, too.

[rochdev]

@provstevi Can you clarify what the issue is when _not_ using import? When using require everything should work out of the box. What is not tagged automatically? (assuming here that tagged means traced)

[provstevi]

OK, to restate, for our services that use koa, we also use esm.

If I have the following, I get everything _except_ redis out of the box, including desired resource names from the route, and other expected plugins, e.g. servicename-http-client, servicename-tcp, servicename-dns.

// startServer.js
require = require('esm')(module);
module.exports = require('./server.js');

// server.js
import './datadog';

// datadog.js
const datadogTrace = require('dd-trace');
const tracer = datadogTrace.init();
const opentracing = require('opentracing');
opentracing.initGlobalTracer(tracer);

Debug reports the following error:

TypeError: Cannot read property 'prototype' of undefined
    at Instrumenter.patch (/app/app/node_modules/dd-trace/packages/datadog-plugin-redis/src/index.js:57:35)
    at Instrumenter.patch (/app/app/node_modules/dd-trace/packages/dd-trace/src/instrumenter.js:219:36)
    at /app/app/node_modules/dd-trace/packages/dd-trace/src/platform/node/loader.js:101:52
    at Array.forEach (<anonymous>)
    at /app/app/node_modules/dd-trace/packages/dd-trace/src/platform/node/loader.js:97:14
    at Map.forEach (<anonymous>)
    at Loader._hookModule (/app/app/node_modules/dd-trace/packages/dd-trace/src/platform/node/loader.js:92:8)
    at Object.Module.require (/app/app/node_modules/require-in-the-middle/index.js:83:30)
    at Object.t (/app/app/node_modules/esm/esm.js:1:279775)
    at n (/app/app/node_modules/esm/esm.js:1:279589)
    at Object.<anonymous> (/app/app/node_modules/redis/lib/debug.js:3:13)
    at Object.<anonymous> (/app/app/node_modules/esm/esm.js:1:251206)
    at Object.<anonymous> (/app/app/node_modules/esm/esm.js:1:287446)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Object.Ph (/app/app/node_modules/esm/esm.js:1:286187)
    at p (/app/app/node_modules/esm/esm.js:1:287490)
TypeError: Cannot read property 'prototype' of undefined
    at Instrumenter.unpatch (/app/app/node_modules/dd-trace/packages/datadog-plugin-redis/src/index.js:60:37)
    at /app/app/node_modules/dd-trace/packages/dd-trace/src/instrumenter.js:229:35
    at Set.forEach (<anonymous>)
    at Instrumenter.unpatch (/app/app/node_modules/dd-trace/packages/dd-trace/src/instrumenter.js:227:20)
    at /app/app/node_modules/dd-trace/packages/dd-trace/src/instrumenter.js:197:14
    at Array.forEach (<anonymous>)
    at Instrumenter.unload (/app/app/node_modules/dd-trace/packages/dd-trace/src/instrumenter.js:196:8)
    at /app/app/node_modules/dd-trace/packages/dd-trace/src/platform/node/loader.js:106:30
    at Map.forEach (<anonymous>)
    at Loader._hookModule (/app/app/node_modules/dd-trace/packages/dd-trace/src/platform/node/loader.js:92:8)
    at Object.Module.require (/app/app/node_modules/require-in-the-middle/index.js:83:30)
    at Object.t (/app/app/node_modules/esm/esm.js:1:279775)
    at n (/app/app/node_modules/esm/esm.js:1:279589)
    at Object.<anonymous> (/app/app/node_modules/redis/lib/debug.js:3:13)
    at Object.<anonymous> (/app/app/node_modules/esm/esm.js:1:251206)
    at Object.<anonymous> (/app/app/node_modules/esm/esm.js:1:287446)
Error while trying to patch redis. The plugin has been disabled.

and I get the following integrations loaded

"integrations_loaded":["http","https","fs","net","dns","[email protected]","http2","[email protected]","[email protected]","[email protected]","[email protected]"]

If I instead do the following, I get _only_ redis out of the box. I added logic to manually tag the endpoints so I got that to mostly work, but no tracing of any other upstream calls, which really clouds my observations:

// server.js
const datadogTrace = require('dd-trace');
const tracer = datadogTrace.init();
const opentracing = require('opentracing');
opentracing.initGlobalTracer(tracer);

and I get the following integrations:

"integrations_loaded":["[email protected]","[email protected]","[email protected]","[email protected]","[email protected]","[email protected]"]

With this config, I had to manually setting the http.route tag to get the resources to show up in APM, and I really don't want to have to try to manually tag the other plugins, but at the moment I have half the picture of what's going on in my service.

In our services that use express, which do not use esm, we are getting redis, express, and the other plugins we expect with no issue.

[provstevi]

I also have a support ticket open for this - 426982

[rochdev]

@provstevi Gotcha, thanks for the clarification! I expected an all or nothing situation, not individual plugins working and other not working. Definitely something we'll be looking into at some point.

One thing we've been trying to understand is the use case for using ESM over CommonJS. Can you share why you're using ESM?

[provstevi]

@rochdev it was a technical decision made by one our teams, and is now baked pretty deeply into several supporting packages. Same reason we have some services using koa and some using express.

NB: ES Modules are stable in node as of 15.3.

[DaedalusX]

We also just ran into this issue as well. A potential workaround would be to migrate to Typescript (there's some tooling for this) but support for ESM would be fantastic until we can do that.

[richard-riverford]

I've also hit this issue. We have some isomorphic code that uses ESM both in Node and in modern browsers (without transpilation), so we just use ESM throughout the project. It would be too much work to revisit that decision now just to get Datadog tracing to work.

[rochdev]

We're working on a plan right now to support ESM soon. Since ESM in Node only supports a single loader hook, it's a bit complicated because it means we need to figure out a way to basically hook into any existing hook. While there is work to address that in Node itself, we still need a workaround that can land beforehand, especially since we support older versions of Node. Supporting this also means a rewrite of some core components of the tracer which will be non-trivial. However, we'll be prioritizing this soon as usage of ESM has been increasing. I will update this issue as progress is made.