Dd-trace-js: TypeError: events.has is not a function

[nickpresta]

Description

When my GraphQL server (using Express, Apollo) starts up, I immediately get the following error:

TypeError: events.has is not a function
    at ClientRequest.addListenerWithTrace (/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:171:17)
    at ClientRequest.addListenerWithTrace [as on] (/node_modules/chunnelx/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:160:24)
    at listenSocketTimeout (_http_client.js:669:9)
    at ClientRequest.setTimeout (_http_client.js:734:3)
    at Object.request (/node_modules/dd-trace/packages/dd-trace/src/platform/node/request.js:43:7)
    at Writer._request (/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/writer.js:71:14)
    at Writer.flush (/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/writer.js:40:12)
    at Timeout.AgentExporter._scheduler.Scheduler [as _onTimeout] (/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/index.js:11:58)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)

The follow environment variables are set:

• DD_RUNTIME_METRICS_ENABLED=true
• DD_TRACE_ANALYTICS_ENABLED=true
• DD_TRACE_ENABLED=true

and the DD config (in-app) looks like:

// # /src/tracer.ts
import tracer from 'dd-trace';
import logger from '~/dependencies/logger';

const application = process.env.APP;
// Set the plugins' service value so that the separator is a `.` instead of a `-`
const service = process.env.WAVE_SERVICE_NAME;

// Initialized in a different file to avoid hoisting
tracer.init({
  env: "prod",
  hostname: '172.17.0.1',
  logger: {
    debug: msg => logger.debug(msg),
    error: msg =>
      msg instanceof Error
        ? logger.error(msg.message, {
            extra: {
              originalError: msg,
            },
          })
        : logger.error(msg),
  },
  service,
  tags: {
    application,
  },
});
tracer.use('dns', {
  service: `${service}.dns`,
});
tracer.use('express', {
  blacklist: ['/.well-known/apollo/server-health'],
});
tracer.use('graphql', {
  analytics: true,
  service: `${service}.graphql`,
});
tracer.use('http', {
  client: {
    service: `${service}.http-client`,
  },
  // Don't set server.service name so that it has no suffix
});
tracer.use('net', {
  service: `${service}.net`,
});
tracer.use('redis', {
  service: `${application}.cache`, // Independent of a specific service
});

export default tracer;

Everything else about the application is a fairly standard Apollo GraphQL application.

Environment

• Operation system: Alpine 3.9.4
• Node version: v10.16.0
• Tracer version: 0.15.0
• Agent version: 6.13.0
• Other packages (possibly related):
  
  
  
  • google-protobuf: 3.9.1
  
  

Package Investigation

• Update on Sept. 16th: I've tried google-protobuf version 3.9.1, 3.9.0, 3.8.0, and 3.7.1 and I'm still seeing the same events.has is not a function error. Going to try downgrading to 0.14.1 of dd-trace.
  
  
  
  • dd-trace version 0.14.1 doesn't seem to work, either.
  
  

Perhaps the WeakMap global is being patched and it doesn't have the has method? I'm going to investigate that.

After doing some debugging, I found the following:

1. It looks like when the eventName is socket (from the http package, I think), the value returned is a plain object ({}).
2. When I change the code here to the following:

let events = this._datadog_events[eventName]
if (!(events instanceof WeakMap)) {
  events = new WeakMap(events.entries)
  this._datadog_events[eventName] = events
}

The error goes away, although I haven't been able to confirm if the code now "works".

I'm having trouble figuring out why this._datadog_events[eventName] will ever return a plain object and not a WeakMap.

Final Update
This error was definitely introduced in 0.14.1, in this PR. Downgrading to 0.14.0 fixes the issue for now.

Thanks!

[rochdev]

@nickpresta Thanks for the thorough investigation! Would you be able to provide a minimal reproduction snippet? I'm unable to reproduce the issue, and as you've mentioned, looking at the code it should not be possible that the variable is ever assigned to anything other than a WeakMap.

[nickpresta]

Hey @rochdev I can take a look at providing a minimal reproduction case later this week. We're currently experiencing the bug in one of our production systems that has many dependencies and code so I'm not sure where to start 😄

Thanks again

[rochdev]

@nickpresta Do you still have this issue with the latest version?

[nickpresta]

Hey @rochdev Apologies for not following up on my last comment.

I've since left the company where I had this problem and am not using DataDog APM anymore 😔

It's probably safe to close this issue if no one else has reported it.

[o-b-one]

In case anyone will run into this issue again:
Try to verify if there are 2 packages installed that consumes different version of dd-trace.
Usually it happen with company internal packages where there are the node_modules of the "common" library and the microservice node_modules.

In case one of them is using dd-trace <=0.14.0 and one of them using dd-trace >=0.14.1 this issue will introduce.
The reason is due to the fact dd-trace switch to work with WeakMap since version 0.14.1.
In case of usage of the two packages there seems to be collision while some of the events initiate with simple object while the 0.14.1 library try to access it has a WeakMap using has, for us it happen each time we used the "request" package.