Datadog-agent: Permission denied when collecting docker.container.open_fds

[renchap]

Describe what happened:
Agent is configured with a simple container monitoring config:

instances:
- url: unix://var/run/docker.sock
logs:
- type: docker
  service: docker
  source: docker


init_config:

Logs are full of lines like:
```2020-02-17 15:16:42 UTC | CORE | WARN | (pkg/collector/corechecks/containers/docker.go:272 in Run) | Failed to get file desc length for pid 14963, container cd4410ca734e: open /proc/14963/fd: permission denied
2020-02-17 15:16:42 UTC | CORE | WARN | (pkg/collector/corechecks/containers/docker.go:272 in Run) | Failed to get file desc length for pid 29853, container cd4410ca734e: open /proc/29853/fd: permission denied
2020-02-17 15:16:42 UTC | CORE | WARN | (pkg/collector/corechecks/containers/docker.go:272 in Run) | Failed to get file desc length for pid 30028, container cd4410ca734e: open /proc/30028/fd: permission denied

Those path in /proc can not be read by the agent due to their permissions:

ls -ld /proc/30028/fd

dr-x------ 2 messagebus ssh 0 Feb 17 08:01 /proc/30028/fd
```

Describe what you expected:
Datadog does not generate hundred of lines like this.
It could either:

• not collect this metric if it can not access /proc/*/fd
• not log one line each time it fails
• provide a way to disable this metric collection
• use something similar to try_sudo from the process check, which has the exact same issue

Steps to reproduce the issue:
Setup Docker monitoring on a Debian system, launch some containers.

Additional environment details (Operating System, Cloud provider, etc):
Debian 10.2
Docker version 19.03.5, build 633a0ea838
Agent 7.16.1 - Commit: 02e0969 - Serialization version: 4.15.0 - Go version: go1.12.9

[andreausu]

Hi, this has been happening to us as well after enabling docker containers monitoring.

docker.yaml config:

init_config:
instances:
  -
    collect_container_size: true
    collect_disk_stats: true
    collect_exit_codes: true

Additional environment details (Operating System, Cloud provider, etc):
Amazon Linux AMI release 2018.03 4.14.165-103.209.amzn1.x86_64
Agent 7.17.0 - Commit: 6eb25b4 - Serialization version: 4.19.0 - Go version: go1.12.9
Docker version: 18.09.9-ce

Adding dd-agent to the list of sudoers as suggested for the process integration hasn't fixed the issue.

[renchap]

Adding dd-agent to the list of sudoers as suggested for the process integration hasn't fixed the issue.

This is normal as the Docker check does not support the sudo find-based method that the process check supports:
https://github.com/DataDog/datadog-agent/blob/642b251f9f7bccd03c76352fec3b7eb07f606c18/pkg/collector/corechecks/containers/docker.go#L273

Which calls:
https://github.com/DataDog/datadog-agent/blob/642b251f9f7bccd03c76352fec3b7eb07f606c18/pkg/util/containers/metrics/proc.go#L21-L34

[asciifaceman]

Running into this as well, Ubuntu 18.04

Agent 7.18.0 - Commit: d88154b - Serialization version: 4.25.0 - Go version: go1.12.9

Docker version 19.03.8, build afacb8b7f0

docker.d/conf.yml

init_config: null
instances:
-   collect_events: true
    collect_exit_codes: true
    collect_disk_stats: false
    collect_container_size: false

datadog.yml

config_providers:
-   name: docker
    polling: true
listeners:
-   name: docker

2020-03-30 21:10:53 UTC | CORE | WARN | (pkg/collector/corechecks/containers/docker.go:275 in Run) | Failed to get file desc length for pid 10528, container f04adeaeb836: open /proc/10528/fd: permission denied
2020-03-30 21:10:53 UTC | CORE | WARN | (pkg/collector/corechecks/containers/docker.go:275 in Run) | Failed to get file desc length for pid 10591, container f04adeaeb836: open /proc/10591/fd: permission denied
2020-03-30 21:10:53 UTC | CORE | WARN | (pkg/collector/corechecks/containers/docker.go:275 in Run) | Failed to get file desc length for pid 9752, container 4b5c5f763bb5: open /proc/9752/fd: permission denied

Can't seem to find a way to disable it either, kind of a nightmare for the journal

[ahmed-mez]

Hi,

Thank you for bringing this to our attention!
Indeed, these errors shouldn't be logged as warnings see https://github.com/DataDog/datadog-agent/pull/5350
Hopefully the fix will be included in Agent 7.20.

Thanks!

[ahmed-mez]

closing this issue as https://github.com/DataDog/datadog-agent/pull/5350 is merged and will be in Agent 7.20