Dd-trace-js: How to sample APM traces by HTTP Endpoint URL?

[MartinLoeper]

Is there an option to configure APM trace sampling for analyzed spans which are emitted for express requests?

We have an HTTP endpoint which is called very often and returns a 401 error code.
We would like to retain APM traces which contain non-401 status at a high sampling rate and others with a low sampling rate.

Is this possible in dd-trace-js?
If no, is it possible to configure this at the dd agent level?

[rochdev]

It depends at that point you know that it will be a 401. You can control the sampling priority to force the agent to keep or drop a trace, but the decision must happen before any propagation to external services. How to do that is explained in our docs here.

The reason why the decision must be done before propagation happens is because we propagate the decision to make sure that the entire distributed trace is kept or dropped to avoid incomplete traces.

[MartinLoeper]

In our scenario, the decision could be made in the very first service that starts the trace - before any propagation to other services or the dd agent.

I am basically looking for something like the "sampling rules" in AWS X-Ray SDK. [1]
You can use them to set a "rate" property which is then enforced by the SDK. [2]

Is there something similar built into the dd-trace-js library?

[1] https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sampling.html
[2] https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nodejs-configuration.html#xray-sdk-nodejs-configuration-sampling

[rochdev]

We do have sampling rules, but they don't currently support arbitrary tags. It's still possible to do it but it will require more code as you would need to write a custom sampling logic, but it can be done with the constructs from the above documentation. Please let me know if you need any help to implement the custom logic. If it can be random with different rates it should be fairly straightforward to combine the sampling priority tags with Math.random() to get the desired behavior.

[MartinLoeper]

Thanks for describing what is currently supported and offering help in implementing new stuff! I am currently considering if it pays off to invest the time to build a custom sampling logic or if I should just use DD_APM_IGNORE_RESOURCES to ignore the route altogether. It would be ideal to do the following:

• If traffic does not match a specific route, no sampling should be performed by the dd-trace-js library (i.e. spans should be forwarded to the dd agent as always).
• If traffic matches a specific route (e.g. /new_feature), sampling should be performed based on HTTP status codes:
  
  
  
  • keep the trace if error code == 4xx OR error code == 5xx
  
  
  • sample at rate 1% for error code == 2xx OR error code == 3xx
  
  

What do you think @rochdev? From your experience: Is this a reasonable approach for ignoring uninteresting traffic in APM that is primarily driving up the APM bill?
Does it make sense to put effort into this and standardize it for the dd-trace-js lib, so others can use it too?

[rochdev]

It sounds like you want to control what gets analyzed and not what is actually sent, sorry for the misunderstanding. These sampling mechanisms are different since we don't bill for sent traces, but we do bill for analyzed traces. This means you don't necessarily want to drop the traces, just stop them from being analyzed.

With that in mind, the approach I described above wouldn't work. You would have to instead change the analytics sampling decision right before the request is ended. The benefit is that it will be easier to implement, but the downside is that decision is not propagated, so upstream/downstream services will make their own decision. If you do have upstream/downstream services for these requests you should implement the logic there as well.

One way to do it would be to add a span hook on the request and set the analytics sampling decision there. So for example, assuming Express is used:

const { ANALYTICS } = require('dd-trace/ext/tags')
const RATE = 0.01

// right after tracer.init()
tracer.use('express', {
  analytics: true,
  hooks: {
    request: (span, req, res) => {
      if (res.statusCode < 400) {
        return span.setTag(ANALYTICS, RATE)
      } // else use the default which is `1` when analytics is enabled
    }
  }
})

Please let me know if that works for your use case!

[rochdev]

Does it make sense to put effort into this and standardize it for the dd-trace-js lib, so others can use it too?

While this would be ideal, I think the end result would be a configuration that allows you to pass your own sampler, which would be very similar to the above. While it would be somewhat cleaner, this is probably not something that will happen soon since the hooks can be used for that purpose.

[MartinLoeper]

Thanks @rochdev! I will try this out and reply whether it worked!

[andrewsouthard1]

Hey @MartinLoeper let us know if you have any questions, if not going to close this out.