Aws-app-mesh-roadmap: Feature Request: Add option to enable Datadog tracer for App Mesh on Amazon ECS (and Fargate)

[CharlyF]

Tell us about your request
👋 It would be great if users could enable the Datadog Tracer on App Mesh on Amazon ECS (Fargate or EC2).
It is one of the tracers that has been upstreamed in envoy and we recently worked with the Amazon EKS team to improve the injector to support the Datadog Tracer on App Mesh on AWS EKS, we received a requests to have this available in ECS as well.

Which integration(s) is this request for?
Fargate, ECS (App Mesh overall).

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Ultimately, we would like to have the equivalent of ENABLE_ENVOY_XRAY_TRACING to enable the Datadog tracer. Following the implementation in the App Mesh injector in Amazon EKS, this could be ENABLE_ENVOY_DATADOG_TRACING that would just inject a similar template as the one for XRay (see details below).

We could also solve this by allowing to mount a file to the task/container, which has been reported here - It would allow for a broader/more future proof integration with other vendors/tracers (as users would only need to mount the extra config and use the ENVOY_STATS_CONFIG_FILE env var of envoy). What I am suggesting is a more straight forward solution that would be consistent with App Mesh on Amazon EKS.

Are you currently working around this issue?
I took the walkthrough for a spin and made some adjustments to gauge how hard it would be to introduce such a feature. I am not able to suggest a code change as it would require a access to the code creating the aws-appmesh-envoy image.

Investigation details:
Looking at the configuration injected in envoy as a result of using ENABLE_ENVOY_XRAY_TRACING:

bash-4.2$ cat envoy_tracing_config.yaml
tracing:
  http:
    name: envoy.xray
    config:
      daemon_endpoint: "127.0.0.1:2000"

Which makes sense (using 127.0.0.1) as containers in the same task are part of the same network namespace.
I just added the following configuration to enable the Datadog tracer:

tracing:
  http:
    name: envoy.tracers.datadog
    config:
      collector_cluster: datadog_agent
      service_name: envoy
static_resources:
  clusters:
  - name: datadog_agent
    connect_timeout: 1s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: datadog_agent
      endpoints:
      - lb_endpoints:
        - endpoint:
           address:
            socket_address:
             address: 127.0.0.1
             port_value: 8126

And created a custom envoy container image simply by:

FROM 840364872350.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.12.2.1-prod
COPY envoy.yaml /tmp/envoy-extra.yaml

Then, updating the spec of the task for the envoy container to use:
ENVOY_STATS_CONFIG_FILE=/tmp/envoy-extra.yaml the Datadog tracer was enabled in the envoy side car.
Updating the code of colorteller and colorgateway to emit traces to the Datadog Agent and switching the Xray daemon to the Datadog Agent container, we were able to get everything to work:

Additional context

More than happy to provide more details about my investigation, or contribute to the codebase if necessary,

[rexnp]

hi @CharlyF, I'll be working on this issue. We will support these two new variables
ENABLE_ENVOY_DATADOG_TRACING : default 0
DATADOG_TRACER_PORT: default 8126
to generate the envoy datadog tracing config. This would also mirror the x-ray variables: https://docs.aws.amazon.com/app-mesh/latest/userguide/envoy.html

[CharlyF]

That's great - Thank you for sharing this update.

[rexnp]

hi @CharlyF, based on our meeting today I will prepare a dev image with this feature to share with some members at Datadog. Could you shoot me a mail at [email protected] so I can whitelist certain AWS accounts to gain access to that image? The official image will roll out with AppMesh's Envoy release in about a couple weeks

[rexnp]

@CharlyF we just shipped our latest Envoy image v1.13.1.0 that contains the datadog tracer support covered in this issue. The Envoy user guide is also updated to explain how to use the two newly added env variables for datadog tracer:https://docs.aws.amazon.com/app-mesh/latest/userguide/envoy.html