Dd-trace-js: Get multiple redis instances showing up as separate services

[tsheaff]

We have moved from one single redis instance to breaking out into now 4 redis instances specific to different pieces of the application.

However, they're all showing up as a single service in the dashboard.

We're using ioredis. When initializing the different connections to the different redis instances, how can we specify that ddtrace should treat them separately rather than all together?

[tsheaff]

@rochdev

[rochdev]

This is not currently possible unfortunately. It is however a known issue which we would like to address for other data stores as well.

If this is something you need right away, I can offer a workaround, but please keep in mind that it relies on internal APIs that could change in the future.

Since the ioredis integration will use the service name of the tracer by default, you could change this just before your ioredis commands. For example:

const service = tracer._tracer._service
tracer._tracer._service = `${service}-1`
redis.set('foo', 'bar')
tracer._tracer._service = service

This code could then be added to a wrapper around ioredis. For this to work, the service configuration of the ioredis plugin should not be set. This will result in a name similar to myservice-1-redis.

Of course this is less than ideal to say the least, but it would unblock you if you need this functionality right away.

[tsheaff]

Thanks @rochdev that is helpful and we will try this out. Our code is wrapped in a way that will make this rather straightforward in terms of code factoring, subject to one caveat below:

If we're waiting for callbacks to occur, will that cause any problems? For example:

redis.get('foo', (err, value) => {
  doSomethingWithValue();
});

Things can obviously occur in between the get() and the doSomething(), including other calls to redis (either the same redis server or another). Will that create problems? There's also the possibility, which may be undocumented, of the callback being called before the next line of the caller (described here)

Planned solution:

this.ddTracer = require('dd-trace').init(); // only called once per redis instance

...

const service = this.ddTracer._tracer._service
this.ddTracer._tracer._service = `${service}-1`
redis.get('foo', (err, value) => {
  doSomethingWithValue();
});
this.ddTracer._tracer._service = service;

Hopefully that works and the tracer._tracer._service only needs to be correct at the moment we call the .get(), .set(), .hsetall() etc. on ioredis's instance rather than throughout the call and until the callback comes back. Otherwise it seems impossible until there's more native support for this.

Thanks for the prompt reply @rochdev I'll reply here once we confirm that it works properly.

[rochdev]

Yes, it should work because everything between the moment redis.get() is called and the span creation happens synchronously. Also, any other calls to a Redis command cannot be done before this code path finishes, which means the service name reset also always has the chance to run.

Of course it does mean that you will need to be very careful with updates to the tracer for the time being. Even though I don't think this internal code will change before we release a proper way to do this, it could still happen.

[tsheaff]

OK that sounds great @rochdev we'll test it out on our dev tier and let you know how it goes. Looks like it won't be too bad. We'll make this code defensive to avoid any null pointer exceptions even if the API changes. I'll post our full code once it's confirmed to be working on production, for posterity.

[rochdev]

@tsheaff What information would you use to differentiate between different Redis clients?

[tsheaff]

@rochdev we have a wrapper over our redis connection that we initialize with various fields. one of these is a name which is a string so we use that.

[rochdev]

What I mean is, for example if this was a built-in option on the plugin, how would you expect to be able to select the correct service name?

For example:

tracer.use('redis', {
  // what should go here?
})

I don't know if there is any information on the Redis client itself that would allow to select a name.

[tsheaff]

Yeah it's a good question. I'm not super familiar with ioredis under the hood. It's just a javascript object tho so I'd imagine we could set a property on the ioredis instance? Not sure how you've handled that (if at all) in other cases. Open to suggestions.

The this.ddTracer._tracer._service is working perfectly for us for now!

[tekmaven]

@rochdev does this still work in v0.10.1?

[rochdev]

@tekmaven Yes, it should still work as of today.