Datadog-agent: 400 bad request from Kubernetes pod

[dignajar]

I'm trying to monitor my Kubernetes cluster which is running in a private subset and to get access to Datadog I have a Squid proxy.

Describe what happened:
Here is the log of the pod.

[ AGENT ] 2019-03-14 13:02:52 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check memory
[ AGENT ] 2019-03-14 13:02:52 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check memory
[ AGENT ] 2019-03-14 13:02:53 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check kubernetes_apiserver
[ AGENT ] 2019-03-14 13:02:53 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check kubernetes_apiserver
[ AGENT ] 2019-03-14 13:02:54 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check io
[ AGENT ] 2019-03-14 13:02:54 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check io
[ AGENT ] 2019-03-14 13:02:55 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check docker
[ AGENT ] 2019-03-14 13:02:55 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check docker
[ AGENT ] 2019-03-14 13:02:56 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check cpu
[ AGENT ] 2019-03-14 13:02:56 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check cpu
[ AGENT ] 2019-03-14 13:02:56 UTC | INFO | (pkg/collector/corechecks/net/ntp.go:180 in queryOffset) | There was an error querying the ntp host 3.datadog.pool.ntp.org: read udp 100.124.0.6:48229->85.25.148.4:123: i/o timeout
[ AGENT ] 2019-03-14 13:02:56 UTC | INFO | (pkg/collector/corechecks/net/ntp.go:151 in Run) | Failed to get clock offset from any ntp host
[ AGENT ] 2019-03-14 13:02:56 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check ntp
[ AGENT ] 2019-03-14 13:02:57 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check kube_dns
[ AGENT ] 2019-03-14 13:02:57 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check kube_dns
[ AGENT ] 2019-03-14 13:02:58 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check uptime
[ AGENT ] 2019-03-14 13:02:58 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check uptime
[ AGENT ] 2019-03-14 13:02:59 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check network
[ AGENT ] 2019-03-14 13:02:59 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check network
[ AGENT ] 2019-03-14 13:03:00 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check load
[ AGENT ] 2019-03-14 13:03:00 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check load
[ AGENT ] 2019-03-14 13:03:01 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check kubelet
[ AGENT ] 2019-03-14 13:03:01 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check kubelet
[ AGENT ] 2019-03-14 13:03:02 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check file_handle
[ AGENT ] 2019-03-14 13:03:02 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check file_handle
[ AGENT ] 2019-03-14 13:03:03 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check disk
[ AGENT ] 2019-03-14 13:03:03 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check disk
[ AGENT ] 2019-03-14 13:03:04 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check kube_dns
[ AGENT ] 2019-03-14 13:03:04 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check kube_dns
[ AGENT ] 2019-03-14 13:03:06 UTC | INFO | (pkg/collector/runner/runner.go:264 in work) | Running check ntp
[ AGENT ] 2019-03-14 13:03:06 UTC | INFO | (pkg/collector/runner/runner.go:330 in work) | Done running check ntp
[ AGENT ] 2019-03-14 13:03:06 UTC | ERROR | (pkg/forwarder/transaction.go:175 in Process) | Error code "400 Bad request" received while sending transaction to "https://6-10-1-app.agent.datadoghq.com/api/v1/check_run?api_key=<REDACTED>": <html><body><h1>400 Bad request</h1>
[ AGENT ] Your browser sent an invalid request.
[ AGENT ] </body></html>
[ AGENT ] , dropping it
[ AGENT ] 2019-03-14 13:03:06 UTC | ERROR | (pkg/forwarder/transaction.go:175 in Process) | Error code "400 Bad request" received while sending transaction to "https://6-10-1-app.agent.datadoghq.com/api/v1/series?api_key=<REDACTED>": <html><body><h1>400 Bad request</h1>
[ AGENT ] Your browser sent an invalid request.
[ AGENT ] </body></html>
[ AGENT ] , dropping it

Describe what you expected:

Steps to reproduce the issue:
Deploy the yaml files from https://docs.datadoghq.com/agent/kubernetes/

Add the environment variables to the deployment descriptor to use the squid.

          - name: DD_PROXY_HTTP
            value: http://<REDACTED>:3128
          - name: DD_PROXY_HTTPS
            value: http://<REDACTED>:3128

Docker image in the deployment descriptor.

- image: docker.ocean.pmicloud.biz/datadog/agent:6.10.1

I changed the API-KEY in the secret.

In the Squid I have an ACL to these endpoints.

.agent.datadoghq.com
process.datadoghq.com
app.datadoghq.com

Additional environment details (Operating System, Cloud provider, etc):
Kubernetes 1.11.7
AWS

[guyromb]

+1

[dignajar]

I found the problem, the API key need to be hashed base64 in the deployment yaml.

api-key: "<YOUR_BASE64_ENCODED_DATADOG_API_KEY>"

Would be good in the documentation add just the command/note how to do it.

$ echo -n "<API-KEY>" | base64

Regards

[xlucas]

Hello,

Yes Kubernetes secrets are base64 encoded. Instructions have been added to the documentation here and here.

So I guess we can close this issue now.

Thanks!