Aws-app-mesh-roadmap: Bug: Datadog service names are hard-coded to "envoy"

[jvosantos]

If you want to see App Mesh implement this idea, please upvote with a :+1:.

Tell us about your request
What do you want us to build?
I would like to be able to set the service name when integrating with datadog for tracing. Envoy Datadog Http Traccer allows such configuration as documented here

Which integration(s) is this request for?
This could be Fargate, ECS, EKS, EC2, Kubernetes, something else.

My main case is ECS, but it could apply to the above ones.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

I want to have datadog correctly configured to have a more smoother experience.

Are you currently working around this issue?
How are you currently solving this problem?

I'm not.

Additional context
Anything else we should know?

IMHO you could either do it with an environment variable such as DD_SERVICE or you could set the service name on the datadog tracing configuration using the environment variable APPMESH_RESOURCE_CLUSTER.

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[jseiser]

We have this same issue with DD/App Mesh now.

Anything App mesh, just shows up as service envoy, environment none. Making it impossible to determine which traffic belongs to which environment and application.

[joshuabaird]

If I'm understanding correctly, you can wrap the AWS-provided Envoy image to specify a custom service like so:

tracing:
  http:
    name: envoy.tracers.datadog
    config:
      collector_cluster: datadog_agent
      service_name: envoy-${APP_NAME}

For what it's worth, this was required for DD integration back when I first looked at App Mesh a year or so ago. Maybe the DD configuration has been streamlined since.

[jseiser]

@joshuabaird

If you install app mesh via: https://github.com/aws/eks-charts/tree/master/stable/appmesh-controller
and datadog via: https://github.com/DataDog/helm-charts/tree/master/charts/datadog

You actually cant even get a working setup. We are currently running a special appmesh/envoy image from AWS, to just get it to integrate with DD at this point.

I dont believe either of those deployment styles, provide for doing anything to the appmesh envoy image, except things that can be done as annotations on your actual applications's pod. Like, ignore egress ports.

[joshuabaird]

@jseiser Ah, yeah.. we run ECS, and we can specify whatever Envoy image we want to use. I'm not sure how it works for k8s/EKS.

[jvosantos]

Depending on what base image of envoy you are running, the datadog c++ tracer dependency can be configured using environment variables.

https://docs.datadoghq.com/tracing/setup_overview/proxy_setup/?tab=envoy#environment-variables

It's what I'm currently doing to avoid having my own custom envoy image for my service mesh.

[joshuabaird]

@jvosantos How are you getting Envoy metrics to DD (dogstatsd)?

[jseiser]

@jvosantos

We are currently running an unreleased image from AWS.

I do not see any way to utilize ENV vars, on our envoy side-car.

If you install app mesh via: https://github.com/aws/eks-charts/tree/master/stable/appmesh-controller
and datadog via: https://github.com/DataDog/helm-charts/tree/master/charts/datadog

[jvosantos]

@joshuabaird
DD is not dogstatsd. Since you're using ECS you can deploy the datadog agent as a daemon service in your ECS cluster and add dockerlabels to all envoy sidecars to have datadog scrape automatically the metrics from envoy. There's some configuration also needed in order to envoy push the traces to the local datadog agent and have datadog agent scrape the metrics from envoy.

Datadog container_definition:

[
  {
    "name": "datadog-agent",
    "image": "datadog/agent:latest",
    "cpu": 128,
    "memory": 256,
    "memoryReservation": 256,
    "essential": true,
    "portMappings": [
      {
        "containerPort": 8126,
        "hostPort": 8126,
        "protocol": "tcp"
      }
    ],
    "dockerLabels": {
      "com.datadoghq.ad.logs": "[{\"source\": \"datadog-agent\", \"service\": \"datadog-agent\"}]"
    },
    "environment": [
      {
        "name": "DD_SITE",
        "value": "datadoghq.eu"
        },
      {
        "name": "DD_APM_NON_LOCAL_TRAFFIC",
        "value": "true"
      },
      {
        "name": "DD_ECS_COLLECT_RESOURCE_TAGS_EC2",
        "value": "true"
      },
      {
        "name": "DD_LOGS_ENABLED",
        "value": "true"
      },
      {
        "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
        "value": "true"
      },
      {
        "name": "DD_APM_DD_URL",
        "value": "https://trace.agent.datadoghq.eu"
      }
    ],
    "secrets": [
      {
        "name": "DD_API_KEY",
        "valueFrom": "arn:aws:ssm:eu-west-1:123456789012:parameter/ecs/datadog-agent/datadog_api_key"
      }
    ],
    "healthCheck": {
      "command": ["CMD-SHELL", "agent health"],
      "startPeriod": 15,
      "interval": 30,
      "timeout": 5,
      "retries": 3
    },
    "mountPoints": [
      {
        "sourceVolume": "docker-sock",
        "containerPath": "/var/run/docker.sock",
        "readOnly": true
      },
      {
        "sourceVolume": "proc",
        "containerPath": "/host/proc",
        "readOnly": true
      },
      {
        "sourceVolume": "cgroup",
        "containerPath": "/host/sys/fs/cgroup",
        "readOnly": true
      },
      {
        "containerPath": "/opt/datadog-agent/run",
        "sourceVolume": "pointdir",
        "readOnly": false
      }
    ],
    "volumesFrom": []
  }
]

Container definition of envoy

[
  {
    "name": "envoy",
    "dockerLabels": {
      "com.datadoghq.ad.logs": "[{\"source\": \"envoy\", \"service\": \"foo\"}]",
      "com.datadoghq.tags.env": "dev",
      "com.datadoghq.tags.service": "foo",
      "com.datadoghq.ad.instances": "[{\"stats_url\": \"http://%%host%%:9901/stats\"}]",
      "com.datadoghq.ad.check_names": "[\"envoy\"]",
      "com.datadoghq.ad.init_configs": "[{}]"
    },
    "image": "840364872350.dkr.ecr.eu-west-1.amazonaws.com/aws-appmesh-envoy:v1.16.1.0-prod",
    "essential": true,
    "cpu": 128,
    "memory": 256,
    "memoryReservation": 256,
    "portMappings": [
      {
        "containerPort": 9901,
        "hostPort": 9901,
        "protocol": "tcp"
      },
      {
        "containerPort": 15000,
        "hostPort": 15000,
        "protocol": "tcp"
      },
      {
        "containerPort": 15001,
        "hostPort": 15001,
        "protocol": "tcp"
      }
    ],
    "healthCheck": {
      "command": [
        "CMD-SHELL",
        "curl -s http://localhost:9901/server_info | grep state | grep -q LIVE"
      ],
      "startPeriod": 10,
      "interval": 5,
      "timeout": 2,
      "retries": 3
    },
    "user": "1337",
    "ulimits": [
      {
        "softLimit": 15000,
        "hardLimit": 15000,
        "name": "nofile"
      }
    ],
    "environment": [
      {
        "name": "APPMESH_RESOURCE_ARN",
        "value": "arn:aws:appmesh:eu-west-1:753823221368:mesh/validation/virtualNode/foo"
      },
      {
        "name": "APPMESH_RESOURCE_CLUSTER",
        "value": "foo"
      },
      {
        "name": "ENABLE_ENVOY_DATADOG_TRACING",
        "value": "1"
      },
      {
        "name": "DATADOG_TRACER_PORT",
        "value": "8126"
      },
      {
        "name": "DATADOG_TRACER_ADDRESS",
        "value": "datadog-agent.svc.cluster.local" // <- service discovery for the datadog agent
      },
      {
        "name": "ENABLE_ENVOY_STATS_TAGS",
        "value": "1"
      },
      {
        "name": "DD_ENV",
        "value": "stg"
      },
      {
        "name": "DD_SERVICE",
        "value": "foo"
      },
      {
        "name": "ENVOY_LOG_LEVEL",
        "value": "info"
      }
    ],
    "mountPoints": [],
    "volumesFrom": []
  }
]

@jseiser

The datadog tracer library that is integrated in envoy looks for the environment variables DD_SERVICE, DD_ENV and DD_VERSION that you can use to configure what service it refers instead of envoy until this issue is addressed. Take a look above.

here's a link to datadog documentation about their tracing library and what env variables you might have available
https://docs.datadoghq.com/tracing/setup_overview/setup/cpp/?tab=otherenvironments#environment-variables
here's a link if you enjoy going through code
https://github.com/DataDog/dd-opentracing-cpp/search?q=DD_SERVICE

[lavignes]

Im going to relabel this as a bug. Ideally every service in the service map should not be hard-coded to the name "envoy". This makes the trace data very hard to use and reason about.

I'll make sure we update this so that we can support DD_SERVICE and have a more sensible default in our next Envoy image release.

[rexnp]

This is now available- starting from App Mesh Envoy image version 1.18.3.0, you can provide an optional DD_SERVICE environment variable in the Envoy container to configure the service name for the Datadog tracer config. We also updated the default behavior such that if you don't specify this environment variable, we will use the Mesh and Virtual Node or Virtual Gateway names to generate the service name (as opposed to the previous behavior where every service was just named envoy). Doc update for this new variable to follow.

[suniltheta]

App Mesh docs are updated, so closing this issue.