Datadog-agent: "unable to get disk metrics" spam from the agent when running as a container

[joshuabaird]

I am running the DD agent as a container. I'm getting the following log message spam every few seconds:

[ AGENT ] 2018-05-23 18:45:54 UTC | WARN | (datadog_agent.go:135 in LogMessage) | (disk.py:105) | Unable to get disk metrics for /host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links: '/host/proc/sys/fs/binfmt_misc'

Some additional information:

docker_host$ ls -l /host/proc/sys/fs/binfmt_misc
ls: cannot access '/host/proc/sys/fs/binfmt_misc': No such file or directory

datadog_agent_container$ DISK='/host/proc/sys/fs/binfmt_misc' /opt/datadog-agent/embedded/bin/python -c 'import psutil; import os; print [part.fstype for part in psutil.disk_partitions(all=True) if part.mountpoint == os.environ["DISK"]][0]'
autofs

datadog_agent_container$ ls -l /host/proc/sys/fs/binfmt_misc/
ls: cannot access '/host/proc/sys/fs/binfmt_misc/': Too many levels of symbolic links

datadog_agent_container$ mount | grep autofs
systemd-1 on /host/proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=0,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1855)

Is adding custom config in /etc/datadog-agent/conf.d/disk.d/disk.yaml (to ignore autofs filesystems) the only way to fix this? Any other suggestions?

Additional environment details (Operating System, Cloud provider, etc):
CoreOS 1688.5.3 running an ECS agent using datadog/agent:latest

[truthbk]

Hi @joshuabaird, thanks for reporting this. Trying to understand the nature of the Too many levels of symbolic links error, in an effort to do so, could you run the following couple commands and share the output (not sure if you might want to redact anything from it):

Host:

find /proc/sys/fs/binfmt_misc -type l -exec ls {} \;
find -L /proc/sys/fs/binfmt_misc

Container:

find /host/proc/sys/fs/binfmt_misc -type l -exec ls {} \;
find -L /host/proc/sys/fs/binfmt_misc

I wouldn't be surprised if we encounter some failures running those commands.

Your issue appears to be similar to https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1555760 and https://bugzilla.redhat.com/show_bug.cgi?id=986339 and possibly related to systemd - so the errors encountered in the logs are probably legit system errors (on that note, restarting the service might help sudo service binfmt-support restart). It's most likely the former. It might also be valuable to get the output for:

systemctl -a | grep binfmt

[joshuabaird]

Here you go:

Host:

ip-10-184-94-198 ~ # find /proc/sys/fs/binfmt_misc -type l -exec ls {} \;

ip-10-184-94-198 ~ # find -L /proc/sys/fs/binfmt_misc
/proc/sys/fs/binfmt_misc
/proc/sys/fs/binfmt_misc/register
/proc/sys/fs/binfmt_misc/status

Container:

root@98058964d05c:/# find /host/proc/sys/fs/binfmt_misc -type l -exec ls {} \;
find: '/host/proc/sys/fs/binfmt_misc': Too many levels of symbolic links

root@98058964d05c:/# find -L /host/proc/sys/fs/binfmt_misc
/host/proc/sys/fs/binfmt_misc
find: '/host/proc/sys/fs/binfmt_misc': Too many levels of symbolic links

ip-10-184-94-198 ~ # systemctl -a | grep binfmt
  proc-sys-fs-binfmt_misc.automount                                                                                                                                   loaded    active   running   Arbitrary Executable File Formats File System Automount Point
  proc-sys-fs-binfmt_misc.mount                                                                                                                                       loaded    active   mounted   Arbitrary Executable File Formats File System
  systemd-binfmt.service                                                                                                                                              loaded    inactive dead      Set Up Additional Binary Formats

ip-10-184-94-198 ~ # systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/usr/lib/systemd/system/systemd-binfmt.service; static; vendor preset: disabled)
   Active: inactive (dead)
Condition: start condition failed at Tue 2018-05-29 12:53:00 UTC; 33s ago
           ├─ ConditionDirectoryNotEmpty=|/lib/binfmt.d was not met
           ├─ ConditionDirectoryNotEmpty=|/usr/lib/binfmt.d was not met
           ├─ ConditionDirectoryNotEmpty=|/usr/local/lib/binfmt.d was not met
           ├─ ConditionDirectoryNotEmpty=|/etc/binfmt.d was not met
           └─ ConditionDirectoryNotEmpty=|/run/binfmt.d was not met
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/Documentation/binfmt_misc.txt

Hmm - yeah, I'll keep digging on the OS side.

[joshuabaird]

The fix to this particular issue was to add a new Require to the systemd unit that starts the datadog-agent container:

ip-10-184-94-198 system # grep Require datadog.service
Requires=proc-sys-fs-binfmt_misc.mount

ip-10-184-94-198 system # docker exec -it dd-agent bash
root@5828dbb260dd:/# find -L /host/proc/sys/fs/binfmt_misc
/host/proc/sys/fs/binfmt_misc
/host/proc/sys/fs/binfmt_misc/register
/host/proc/sys/fs/binfmt_misc/status