Charts: [stable/datadog] Missing dogstasd and apm port from service

[endzyme]

Missing Functionality
During the 2.x refactor it looks like you removed some functionality from the chart templates. There used to be a service.yaml file, which could direct dogstatsd.port traffic to pod endpoints of the daemonset or a smaller subset of a deployment. This functionality is now missing and I cannot find out why. Nothing was mentioned about this drop in functionality and I am curious if it is intentional.

• Commit where the stable/datadog/templates/service.yaml was removed: 6d2fc82c220ad13e0d74df69275fc98f6085ccf1

The new file (stable/datadog/templates/agent-services.yaml) only includes support for the cluster agent port on 5005 but doesn't appear to support sending statsd to port 8125 nor apm port 8126.

As a separate issue, it would appear you can no longer deploy the solo statsd agents as a deployment. I'm also curious to understand if this is intentional. If so, the README should be reconciled.

Any help or insight would be appreciated :) Thanks!

[grzegorzgr]

This looks to be a bug. Helm chart doesn't open a hostport in current version. Had to manually edit daemonset.

[endzyme]

@clamoriniere @CharlyF

[Simwar]

Hi @endzyme

This service has been removed because running the agent with this service alongside a dogstasd deployment was considered as running the agent in degraded mode.

Explanations:

• the pods of this dogstatsd deployment will run on random nodes, and metrics you send to the service will be sent in a round-robin fashion to these pods.
  Since the dogstatsd server (the agent) adds the hostname of the host it runs on to metrics it receives, metrics that are sent to the dogstatsd service will have the wrong host name. That makes querying them difficult even if you don’t care about the host name because the wrong host tags get added to the metric. If your query uses a host tag instead of a pod tag, the value will be wrong in a non-obvious way.

• metrics that are sent to a dogstatsd server running in the agent on the same host as the application pod can benefit from origin detection, so they get pod and container tags automatically if origin detection is enabled. That doesn’t work if they’re sent to a random dogstatsd server running on a different host, behind a service.

Let us know if you have further questions!

Simon

[grzegorzgr]

@Simwar what about the missing APM.
Removed hostPort 8126 will prevent metrics from being pushed to the agent.
I spend a day trying to make it work until I figured that there was no listener on a destination port.

[endzyme]

Hi @endzyme

This service has been removed because running the agent with this service alongside a dogstasd deployment was considered as running the agent in degraded mode.

Explanations:

• the pods of this dogstatsd deployment will run on random nodes, and metrics you send to the service will be sent in a round-robin fashion to these pods.
  Since the dogstatsd server (the agent) adds the hostname of the host it runs on to metrics it receives, metrics that are sent to the dogstatsd service will have the wrong host name. That makes querying them difficult even if you don’t care about the host name because the wrong host tags get added to the metric. If your query uses a host tag instead of a pod tag, the value will be wrong in a non-obvious way.
• metrics that are sent to a dogstatsd server running in the agent on the same host as the application pod can benefit from origin detection, so they get pod and container tags automatically if origin detection is enabled. That doesn’t work if they’re sent to a random dogstatsd server running on a different host, behind a service.

Let us know if you have further questions!

Simon

Does this mean the only supported implementation for kubernetes is to run statsd on hostPort, and to send statsd stats to DD_AGENT_HOST set to status.hostIP?

i.e. -

spec:
      containers:
      - name: ...
        image: ...
        env:
          - name: DD_AGENT_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP

I am failing to see another way to send custom metrics to a specific host, without using useHostNetwork:true. The docs in values.yaml are also misleading - still citing references to a service endpoint.

https://github.com/helm/charts/blob/master/stable/datadog/values.yaml#L506

Please help me understand the path forward and what the best practice is to produce custom metrics to statsd port on the agents.

[CharlyF]

👋 Using the downward API is the best practice we recommend following indeed, whether it is for statsd or for traces. This is to ensure that datagrams are sent to the local instance, to get proper tagging.
I see the hostPort option listed with .Values.datadog.apm.port for apm, am I missing something @grzegorzgr ?
Thanks for the feedback on the clarity of the docs - we're going to fix that.

Best,
.C

[grzegorzgr]

@CharlyF not a single cased container loads for some reason.

from templates/daemonset.yaml:

  containers:
    {{- include "container-agent" . | nindent 6 }}
    {{- if .Values.datadog.apm.enabled }}
      {{- include "container-trace-agent" . | nindent 6 }}
    {{- end }}
    {{- if .Values.datadog.processAgent.enabled }}
      {{- include "container-process-agent" . | nindent 6 }}
    {{- end }}
    {{- if .Values.datadog.systemProbe.enabled }}
      {{- include "container-system-probe" . | nindent 6 }}
    {{- end }}

running helm upgrade --install --namespace datadog --wait -f values.yaml --set datadog.apm.enabled=true datadog stable/datadog will only build the main agent container but no trace-agent. Tested the same for system-probe and it didn't work. I am yet to find out why this happens.

Helm Version: v2.12.3

[edit] - it may be an issue with my helm version

[clamoriniere]

Hello @grzegorzgr
I have tested with v2.16.7 and v3.2.0.
For both versions, the following command works fine:
helm upgrade --install --namespace datadog --wait --set datadog.apiKey=**** --set datadog.apm.enabled=true datadog stable/datadog

the only difference that I see is in your command, you also have a reference to a values.yaml file, but it should not be a problem.

which version of the chart are you using?
Could you run: helm search datadog is the version is inferior to 2.2.12 you can run the command: helm repo update

[grzegorzgr]

stable/datadog 2.2.12 7 Datadog Agent

Ok turned out it was a helm issue - noticed both client and server are running an outdated version which has a known bug of not including nested charts properly