Dd-trace-js: Service map looks odd with out-of-box plugins

[stephenh]

We're using ddtrace-js in a new project with all out-of-the-box plugins and while the traces are generally really useful / great to get all of this for free, the service map looks strange:

I.e. things like graphql-service-fs and graphql-service-graphql are not separate services, they're just internal modules/spans created internally within the plugins.

Granted, graphql-service-postgres is a real external service/database, as well as graphql-service-aws-cognito. And graphql-service-tcp is making calls to "other things". But I'd expect this to be more like:

• graphql-service-postgres would, instead of being "all calls to any pg database", use the db.instance that is already on the span to have a service name that is the specific database name
• graphql-service-tsp would, instead of being "any random tcp call", use the span's tcp.remote.host value to specifically identify the external service.

FWIW, all of the APM data is very useful, and the service map "looking a little weird" is not a deal breaker, just curious if there are plans to nudge the plugins to 1) create more "logical based on this span's specific destination" names (i.e. a given db host name, given tcp host name) and 2) not create "separate services" like the graphql plugin suffixing -graphql to the top-level service name which makes it look like a separate service when really it is not.

(I also see in the docs that the plugins can have all of their service names customized, which I could do for the pg plugin (assuming I am connecting to only a single pg instance, so can set it globally / at the plugin level) / graphql plugin / etc.)

[stephenh]

(Er, didn't mean to add the bug label to this, looks like I can't edit/remove it?)

[rochdev]

We're using ddtrace-js in a new project with all out-of-the-box plugins and while the traces are generally really useful / great to get all of this for free, the service map in particular looks strange:

This is a known issue that we're working on to improve how grouping of resources is done without relying exclusively on the service name.

I.e. things like graphql-service-fs and graphql-service-graphql are not separate services, they're just internal modules/spans created internally within the plugins.

The main reason for this is to ensure the resources from each integration are grouped and aggregated together. As mentioned above, we are working on improving this. If you don't need this information at all, it's possible to rename these services to the same name as the application service, and they will no longer appear separated. When doing this, it's important to consider the impact it will have on errors, metrics and the service page since some information will be lost for each. In general I recommend to keep the default for these, at least for now, even if it creates more noise in the service map.

graphql-service-postgres would, instead of being "all calls to any pg database", use the db.instance that is already on the span to have a service name that is the specific database name

It's possible to change the service name right now, but not dynamically like this. I think there could be a few ways to improve this:

1) As you proposed, an option to use the instance name instead of a static name. This would be similar to the splitByDomain option for http. Maybe something like splitByInstance?
2) Accept a function for the service name that could set it dynamically.
3) Add a span hook to postgres and similar to allow altering the service name and any other span metadata before the span is added to the trace.

graphql-service-tsp would, instead of being "any random tcp call", use the span's tcp.remote.host value to specifically identify the external service.

Similar to the above, I think all 3 options would work in this case as well.

FWIW, all of the APM data is very useful, and the service map "looking a little weird" is not a deal breaker, just curious if there are plans to nudge the plugins to 1) create more "logical based on this span's specific destination" names (i.e. a given db host name, given tcp host name) and 2) not create "separate services" like the graphql plugin suffixing -graphql to the top-level service name which makes it look like a separate service when really it is not.

We are exploring relying more in raw data to group things in the UI and the backend instead of relying solely on the service name. The goal being that the service name would then be exactly that: the name of the service and nothing more. Then other tags would be used to replicate what we have today and even extend beyond what is currently possible.

(I also see in the docs that the plugins can have all of their service names customized, which I could do for the pg plugin (assuming I am connecting to only a single pg instance, so can set it globally / at the plugin level) / graphql plugin / etc.)

Yes but it can only be done statically at config time. To change it dynamically will require changes in the tracer (see above for 3 options to handle this).

[stephenh]

Hi @rochdev , thanks for the reply.

This is a known issue that we're working on to improve how grouping of resources

Awesome. Out of curiosity is there an issue / something I could follow to hear about progress on this? Or are these sort of changes less "ddtrace issue/release notes" and more "datadog marketing blog / something"?

similar to the splitByDomain option for http. Maybe something like splitByInstance

Ah nice! I had not seen splitByDomain...I think a splitByInstance would be great as well.

I'm also tempted to think these sort of "split by x b/c this is obviously a separate/remote service that the span 'knows' the name of" should default to on, because currently they are not very discoverable and (I assume/my bias) seems like "what most people would expect". Admittedly it would be a breaking change.

In general, you've answered my question, which is that yeah this is actively being considered/worked on/etc.

Thanks for the splitByDomain hint. I think a splitByInstance would also be great but a different/specific ticket, so feel free to close this one out.

Thanks!

[rochdev]

Out of curiosity is there an issue / something I could follow to hear about progress on this? Or are these sort of changes less "ddtrace issue/release notes" and more "datadog marketing blog / something"?

Yes this is definitely not a dd-trace issue and is platform-wide. Unfortunately I don't have a timeline and it will most likely take a while given the implications, but it's definitely something being worked on.

Ah nice! I had not seen splitByDomain...I think a splitByInstance would be great as well.

I'll think about how to make this kind of configuration available across all integrations and potentially make it more flexible to match our long-term plans.

I'm also tempted to think these sort of "split by x b/c this is obviously a separate/remote service that the span 'knows' the name of" should default to on, because currently they are not very discoverable and (I assume/my bias) seems like "what most people would expect". Admittedly it would be a breaking change.

Yes, unfortunately it's a breaking change and also differs from the expectation of the platform. It's also only one possible dimension across many possible ones, so it's not necessarily always expected. I think a more flexible way to set the service name, and maybe a global config to set the pattern across every integrations could go a long way to improve this until the underlying limitation is resolved.

I'll keep this issue open since while the question was answered, the issue is not resolved.