Datadog-agent: Retrieve pod's name on Kubernetes with AutoDiscovery

[ueokande]

Describe what happened:

I propose an ability to retrieve pod's name on Kubernetes with AutoDiscovery. The AutoDiscovery currently does not allows to retrieve pod's name from tags.

For example, the following settings discover and observe NGINX status on each pods via AutoDiscovery:

annotations:
  ad.datadoghq.com/nginx.check_names: '["nginx"]'
  ad.datadoghq.com/nginx.init_configs: '[{}]'
  ad.datadoghq.com/nginx.instances: |
    [ { "nginx_status_url": "http://%%host%%:10081/nginx_status" } ] 

Currently, I can observe only pod's deployment name, namespace, and template hash. Additionally the host tag on the metric indicates Kubernetes node's name (EC2 instance name).

In that situation, I cannot aggregate metrics by NGINX pods when the pods placed on same node. The current workaround to avoid this issue, I adds template variable %%host%% as a tag. But that is confusing for me due to the variant indicates pod's IP.

annotations:
  ad.datadoghq.com/nginx.instances: |
    [   
      {   
        "nginx_status_url": "http://%%host%%:10081/nginx_status",
        "tags": [ "host_ip:%%host%%" ]
      }   
    ]   

Describe what you expected:

Now, I propose a new idea to retrieve pod's name on AutoDiscovery:

• Set pod's hostname (pod's name) to host tag.
• Add pod's name as a tag such as kube_pod_name.

Additional environment details (Operating System, Cloud provider, etc):

• Amazon EKS 1.13
• DataDog version: datadog/agent:6.13.0-jmx

[mfpierre]

Hey @ueokande thanks for reaching out,

There's actually already an option in the agent to add the pod_name tag to all auto-discovered checks metrics by setting the DD_CHECKS_TAG_CARDINALITY option to orchestrator.

Please refer to the official documentation https://docs.datadoghq.com/tagging/assigning_tags/?tab=agentv6

[ueokande]

@mfpierre
Thank you for your response!
I can observe pod's name of the metric.