Datadog-agent: Datadog agent doesn't scrape OpenMetrics of kubernetes-external-secrets controller

[toshi0607]

Please help me configure to scrape metrics from kubernetes-external-secrets controller.

Describe what happened:

I want to scrape prometheus metrics from a kubernetes-external-secretes controller, but it doesn't work.

metrics that I want to scrape

• kubernetes_external_secrets_sync_calls_count
• kubernetes_external_secrets_last_sync_call_state

from https://github.com/external-secrets/kubernetes-external-secrets#metrics

Describe what you expected:

Datadog agent collects metrics

Steps to reproduce the issue:

I set up the controller like this.

podAnnotations:
  ad.datadoghq.com/kubernetes-external-secrets.check_names: '["openmetrics"]'
  ad.datadoghq.com/kubernetes-external-secrets.init_configs: "[{}]"
  ad.datadoghq.com/kubernetes-external-secrets.instances: |
    [{
      "prometheus_url": "http://%%host%%:3001/metrics",
      "namespace": "external-secrets",
      "metrics": [
        "kubernetes_external_secrets_sync_calls_count",
        "kubernetes_external_secrets_last_sync_call_state",
      ]
    }]

$ kubectl get pod external-secrets-kubernetes-external-secrets-xxxxxx -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    ad.datadoghq.com/kubernetes-external-secrets.check_names: '["openmetrics"]'
    ad.datadoghq.com/kubernetes-external-secrets.init_configs: '[{}]'
    ad.datadoghq.com/kubernetes-external-secrets.instances: |
      [{
        "prometheus_url": "http://%%host%%:3001/metrics",
        "namespace": "external-secrets",
        "metrics": [
          "kubernetes_external_secrets_sync_calls_count",
          "kubernetes_external_secrets_last_sync_call_state",
        ]
      }]
    kubernetes.io/psp: eks.privileged
...

spec:
  containers:
  - env:
...
    name: kubernetes-external-secrets
    ports:
    - containerPort: 3001
      name: prometheus
      protocol: TCP

In datadog agent pod, agent configcheck shows prometheus check, but it only includes namespace: datadog.cluster_agent, not external-secrets.

Additional environment details (Operating System, Cloud provider, etc):

• Datadog runs on Amazon EKS
• Agent 7.24.1 - Commit: cf2a7e6 - Serialization version: v4.46.0 - Go version: go1.14.7

[Simwar]

Hi @toshi0607

Thanks for reaching out!

The configuration you shared seems good, although it seems that you have one extra comma in the metrics parameter:
"kubernetes_external_secrets_last_sync_call_state",

Could you try to remove this extra comma and see if the agent picks up the annotations?

If this doesn't do the trick, please email our support team so they can investigate further with specific details from your environment.

Thank you!

[toshi0607]

Hi @Simwar, really thank you for the suggestion. After removing the comma, datadog agent started to collect metrics and our dashboard shows the metrics at last 🎉

In this case, what is the best way to debug? I compared docs and configs, tried agent commands in datadog-agent pods, and kubectled the logs of the datadog-agent. Please tell me the way to detect config error directly like checking the specific log, etc if available!