Dd-trace-js: Memory leak

[UnbrandedTech]

What's the ETA on a KoaJS integration?
We're trying to manually wire up dd-trace but it's causing memory leaks.

The plateaus in the image is when we reverted the commit, after 4 attempts to figure out what was causing the ENOMEM errors.

[UnbrandedTech]

Heres our code that caused the memory leak.

DataDog.js

'use strict'

const { Tags, FORMAT_HTTP_HEADERS } = require('opentracing')

const tracer = require('dd-trace').init({
    service: 'API',
    plugins: false,
    env: process.env.NODE_ENV,
})

tracer.use('mysql2')
tracer.use('redis')

module.exports.tracer = tracer
module.exports.Tags = Tags
module.exports.FORMAT_HTTP_HEADERS = FORMAT_HTTP_HEADERS

DataDogMiddleware.js

'use strict'

const { tracer, Tags, FORMAT_HTTP_HEADERS } = require('../lib/DataDog')

//an array of routes to ignore i.e. '/status' or '/health'
const ignored_routes = require('../lib/constants/ignoredRoutes')

//an array of acceptable methods i.e. 'GET' or 'POST'
const whitelisted_methods = require('../lib/constants/whitelistedMethods')

module.exports = async (ctx, next) => {
    if (whitelisted_methods.includes(ctx.method) && !ignored_routes.includes(ctx.path)) {
        const span = tracer.startSpan('koajs.request', {
            childOf: tracer.extract(FORMAT_HTTP_HEADERS, ctx.request.headers),
            tags: {
                [Tags.SPAN_KIND]: Tags.SPAN_KIND_RPC_SERVER,
                [Tags.HTTP_URL]: ctx.request.href,
                [Tags.HTTP_METHOD]: ctx.request.method
            }
        })

        const scope = tracer.scopeManager().activate(span)

        await next()

        span.setTag('resource.name', ctx.request.path)

        span.setTag('service.name', 'API')
        span.setTag('span.type', 'web')
        span.setTag(Tags.HTTP_STATUS_CODE, ctx.response.statusCode)

        if (ctx.response.statusCode > 500) span.setTag(Tags.ERROR, true)

        span.finish()
        scope.close()
    } else {
        await next()
    }
}

[SethDavenport]

Seeing something very similar in my codebase.

I'm using [email protected] and the heroku buildpack. Here's a screenshot of Heroku's dyno metrics:

That point where the mem starts to climb coincides with the deployment of the agent/buildpack.

In terms of code I'm doing almost the exact same thing as @UnbrandedTech (looks like we both copied and adapted the express middleware for use with Koa 😂 ):

// @flow
import { tracer } from 'initializers/datadog';
import type { Middleware } from 'koa';

// Customized and Koa-ified from the express middleware that ships with dd-trace:
// https://github.com/DataDog/dd-trace-js/blob/master/src/plugins/express.js
//
// See https://datadog.github.io/dd-trace-js/#integrations
// See also https://github.com/koajs/koa/blob/master/docs/api/context.md

const OPERATION_NAME = 'koa.request';

const validateStatus = code => code < 500;

export const datadog: Middleware = async (ctx, next) => {
  const { method, url } = ctx;

  const span = tracer.startSpan(OPERATION_NAME, {
    tags: {
      'http.url': url,
      'http.method': method,
    },
  });

  const scope = tracer.scopeManager().activate(span);

  await next();

  const resourceName = ctx._matchedRoute === '(.*)' ? ctx.url : ctx._matchedRoute;
  span.setTag('resource.name', `${method} ${resourceName}`);
  span.setTag('span.type', 'http');
  span.setTag('http.status', ctx.status);

  if (!validateStatus(ctx.status)) {
    span.setTag('error', true);
  }

  span.finish();
  scope.close();
};

[SethDavenport]

Trying with 0.5.1. Will update once I have more data.

[UnbrandedTech]

Very interested if it works, It took our system down so I'm not willing to test it unless I get a confirmation from the Dev team that's it's been fixed.

[rochdev]

I was able to replicate this and I am currently investigating. I'll let you know as soon as a fix is on the way.

I'll move KoaJS support to a new issue since this discussion is really about the leak and it doesn't seem to be related to the middleware.

[SethDavenport]

Yean 0.5.1 exhibits the same behaviour unfortunately. Thanks for looking into it @rochdev!

[rochdev]

Can you try with 0.5.2-beta.0? It seems to fix it for me.

[holdenmc]

We were having the same issue described here. We just re-deployed with 0.5.2-beta.0 about 15 minutes ago and it's holding steady memory-wise

(edit: been over an hour now and memory use is still at normal levels)

[rochdev]

Fixed by #203

[rochdev]

Released in 0.5.2

[SethDavenport]

Just tried 0.5.2 - memory usage appears to be stable. Thanks for the quick turnaround!