Dd-trace-js: Send container_id as APM tag

[MartinLoeper]

We are using the dd-trace-js library inside a docker container running node.js on AWS Fargate.
Is it possible to tag the APM spans with the Fargate container id?

I noticed that we somehow tag our logs with an attribute called container_id and it would be nice to tag all APM spans accordingly.

[rochdev]

What is your use case? We already report the container ID for correlation purpose but not as a tag, so it may or may not work for you depending on your requirements.

[MartinLoeper]

My use case is to enable specific APM facets under the containers section:

Currently they are not working.
What I do not understand: The container section seems to be included in the APM UI by default.

[rochdev]

My understanding is that these should be added by the agent based on the container ID reported by the tracer. Let me verify that this is actually the case, in which case there might be a bug.

[rochdev]

I wasn't able to get a definitive answer internally, but from the above code and the tracer which sends the container ID of the process to the agent, these should be added automatically out of the box. I would recommend contacting support since they will be able to take a closer look into your agent configuration, and into what is actually received and sent by the agent to determine if some data is missing at any point during flushing.

You could also first verify that the logic used by the tracer works properly by replicating the few lines used to grab the container ID:

const containerInfo = require('container-info').sync() || {}
const containerId = containerInfo.containerId

If this doesn't return the ID, the issue is likely a bug in the tracer, and otherwise it would be more likely to be an issue downstream in either the agent or the backend.

[MartinLoeper]

Thanks @rochdev. I'm going to raise a support case and report back here.

[MartinLoeper]

@rochdev I printed the container-info as requested by you in the comment above.
I think I found the issue and asked for clarification upstream [1].
Maybe this is something that has to be fixed inside the container-info package?!

TLDR; The container-info package is not able to retrieve the container id for our containers running on AWS Fargate.

[1] https://github.com/aws/amazon-ecs-agent/issues/1119#issuecomment-729133165

[rochdev]

Definitely something that should be fixed upstream, but also something we'll need to otherwise fix in the tracer. Maybe container-info could be configurable to allow customizing the regular expressions so that custom environments like Fargate are easier to support externally. I'll have to look into it.

[rochdev]

From what I can tell, with the cgroup change it's no longer possible to grab the container ID from it. It looks from the docs that it is now available in a metadata file instead that is only available if explicitly enabled.

@MartinLoeper Would support for that file out of the box in the tracer work for your use case?

[rochdev]

The above approach will be supported when https://github.com/Qard/container-info/pull/10 lands.

[MartinLoeper]

@MartinLoeper Would support for that file out of the box in the tracer work for your use case?

Yes, transferring the container ID correctly would be the way to go for us.
If ECS would not have changed the container ID format, the current approach would also work.
Thus, fixing the container ID extraction in container-info by using the metadata file would be fine.

The above approach will be supported when Qard/container-info#10 lands.

Looks good to me! Good job! Thanks @rochdev!

[rochdev]

It looks like this approach won't work with the current version of the agent. We're currently looking at fixing this at the agent level with the goal of including the fix in the next version of the agent. I will update this issue as we make progress.