Datadog-agent: Spamming Error: Missing target pids from mounts since v6.11.0 (AWS, ECS, Docker)

[piotrb]

Output of the info page (if this is a bug)
agent.status.txt

Describe what happened:
Ever since version 6.11.0 getting a lot of spam in logs about:

Error: Missing target pids from mounts

Describe what you expected:
No spam?

Steps to reproduce the issue:

This is reproable by just running the check docker command on the amazon ecs ami we're running:

The output looks like this:

/opt/datadog-agent/bin/agent/agent check docker
Multiple check instances found, running each of them
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
Error: Missing target pids from mounts
=== Series ===
And then a bunch of json for the actual docker containers

Additional environment details (Operating System, Cloud provider, etc):
AWS
Amazon ECS AMI (latest or near latest)

[thisdougb]

Just to add some error messages, I am seeing both of these continuously on 6.11.

2019-04-24 06:18:58 UTC | PROCESS | ERROR | (pkg/util/containers/metrics/cgroup_detect.go:53 in cgroupFilePath) | Missing target pids from mounts

2019-04-24 06:19:00 UTC | CORE | ERROR | (pkg/util/containers/metrics/cgroup_detect.go:53 in cgroupFilePath) | Missing target pids from mounts

I have both v6.10 and v6.10.02 running without this error, on various recent AMI versions.

[danvaida]

I'm also seeing this with 6.11.0

2019-05-03 08:45:50 UTC | CORE | ERROR | (pkg/util/containers/metrics/cgroup_detect.go:53 in cgroupFilePath) | Missing target pids from mounts

Hi @thisdougb 👋

[jatinngupta]

2019-05-20 08:16:51 UTC | CORE | ERROR | (pkg/util/containers/metrics/cgroup_detect.go:53 in cgroupFilePath) | Missing target pids from mounts
2019-05-20 08:16:54 UTC | PROCESS | ERROR | (pkg/util/containers/metrics/cgroup_detect.go:53 in cgroupFilePath) | Missing target pids from mounts

Only seeing these errors in agent logs

[dagisus]

we are getting the same issue using de 6.11.3, we had to rollback to 6.10.2 to avoid the log spam. any updates on this issue???

[thisdougb]

@dagisus the fix seems to be coming in the 6.12 release: https://github.com/DataDog/datadog-agent/milestone/52?closed=1

[dagisus]

we are getting the same issue using de 6.11.3, we had to rollback to 6.10.2 to avoid the log spam. any updates on this issue???

we solved the issue by updating the ECS AMI to the latest. it seems that in previous versions some cgroup data is not stored in the default /cgroup/ and the documented /sys/fs/cgroup/ is also empty, we moved the cluster AMI to the latest to this date for us-east-1 ami-0f812849f5bc97db5 which has docker version 18.06.1-ce and Amazon agent version 1.28.1 running Amazon Linux 2. Now we are running the latest 6.11.3 with no spam or issues. Hope this helps.

[DylanLovesCoffee]

Going to close this out since the fix was released with 6.12.0