Datadog-agent: Error running check disk: [{"message": "[Errno 2] No such file or directory: '/host/proc/filesystems'"

[amlwwalker]

Trying to use datadog/agent to log the std out of the other containers into datadog event logs inside a docker-compose file, running locally and on AWS ECS.

Describe what happened:
I am using the agent inside docker, and specifically using convox to deploy my infrastructure. convox.yml is very close to a docker-compose.yml. I have datadog agent running inside the compose file as such

    datadog:
      agent: true
      privileged: true
      environment:
        - DD_API_KEY=${DATADOG_API_KEY}
        - DD_APM_ENABLED=true
        - DD_LOGS_ENABLED=true
        - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      image: datadog/agent:latest
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - /proc/:/host/proc/:ro
        - /cgroup/:/host/sys/fs/cgroup:ro

and I have the following errors in my logs.

2018-07-14 16:10:04 UTC | ERROR | (runner.go:277 in work) | Error running check disk: [{"message": "[Errno 2] No such file or directory: '/host/proc/filesystems'", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/base.py\", line 294, in run\n    self.check(copy.deepcopy(self.instances[0]))\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 43, in check\n    self.collect_metrics_psutil()\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 90, in collect_metrics_psutil\n    for part in psutil.disk_partitions(all=True):\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/__init__.py\", line 1839, in disk_partitions\n    return _psplatform.disk_partitions(all)\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py\", line 1000, in disk_partitions\n    with open_text(\"%s/filesystems\" % get_procfs_path()) as f:\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py\", line 194, in open_text\n    return open(fname, \"rt\", **kwargs)\nIOError: [Errno 2] No such file or directory: '/host/proc/filesystems'\n"}]
2018-07-14 16:10:04 UTC | WARN | (cgroup.go:510 in parseCgroupMountPoints) | No mountPoints were detected, current cgroup root is: /host/sys/fs/cgroup/

I am running localy for testing at the moment (OSX with docker). I can see in the logs that datadog agent is successfully detecting the other containers

Describe what you expected:

Hopefully to see logs from my other containers in datadog events!

Steps to reproduce the issue:

I expect, to create a docker-compose file, stick a container in there (web app for example) and copy the above docker-compose configuration for datadog/agent. Then check the docker logs.

If that doesn't work, it would mean using convox locally and doing the above, however explaining how to use convox is beyong this

Additional environment details (Operating System, Cloud provider, etc):

[sgbett]

Similar issue, not using logs just trying to get APM running on a rails app in local docker setup (on a Mac).

Rails app all works fine, added this to my docker-compose.yml to spin up a datadog agent as part of docker cluster:

datadog:
image: datadog/agent:latest
environment:
- DD_API_KEY=[my key]
- DD_HOSTNAME=0.0.0.0
- DD_APM_ENABLED=true
ports:
- "8126:8126"

in docker logs the error I am getting is....

datadog_1 | [ AGENT ] 2018-07-23 20:24:29 UTC | ERROR | (runner.go:277 in work) | Error running check disk: [{"message": "[Errno 2] No such file or directory: '/host/proc/filesystems'", "traceback": "Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/base.py\", line 294, in run\n self.check(copy.deepcopy(self.instances[0]))\n File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 43, in check\n self.collect_metrics_psutil()\n File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 90, in collect_metrics_psutil\n for part in psutil.disk_partitions(all=True):\n File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/__init__.py\", line 1839, in disk_partitions\n return _psplatform.disk_partitions(all)\n File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py\", line 1000, in disk_partitions\n with open_text(\"%s/filesystems\" % get_procfs_path()) as f:\n File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py\", line 194, in open_text\n return open(fname, \"rt\", **kwargs)\nIOError: [Errno 2] No such file or directory: '/host/proc/filesystems'\n"}]

Then the agent reports....

datadog_1 | [ TRACE ] 2018-07-23 20:24:57 INFO (service_mapper.go:59) - total number of tracked services: 0

and

datadog_1 | [ TRACE ] 2018-07-23 20:25:07 INFO (receiver.go:324) - no data received

though this looks promising if I can fix the other issue (I think!)....

datadog_1 | [ AGENT ] 2018-07-23 20:25:43 UTC | INFO | (transaction.go:121 in Process) | Successfully posted payload to "https://6-3-3-app.agent.datadoghq.com/api/v1/series?api_key=*************************0c030"

[sgbett]

Further to previous comment, I was able to resolve by adding the relevant mount points to my docker-compose.yaml as follows:

+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /proc/:/host/proc/:ro
+      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

[zhouzhuojie]

Why is - /proc/:/host/proc/:ro required? What's the use of it, and what's the downside if containers are started without any mount points?

[rogaha]

same problem here.

[ab]

This appears to be caused by datadog config procfs_path defaulting to /host/proc when running inside a container. You can override this, though.

I set this in the environment, which seemed to do the trick:

DD_PROCFS_PATH=/proc

I believe setting procfs_path: /proc in datadog.yaml would also work.

It looks like this was actually just fixed on master by #5168, so the next version of datadog-agent may have this fixed:

https://github.com/DataDog/datadog-agent/pull/5168/files#diff-091ace02b49a46f55f2586cfec89a160R228