Datadog-agent: Too many levels of symbolic links occur again in datadog-agent:6.11.0

[meoooh]

Output of the info page (if this is a bug)

(Paste the output of the info page here)

Describe what happened:

CORE | WARN | (pkg/collector/py/datadog_agent.go:148 in LogMessage) | (disk.py:114) | Unable to get disk metrics for /host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links: '/host/proc/sys/fs/binfmt_misc'

Describe what you expected:
No error

Steps to reproduce the issue:

Additional environment details (Operating System, Cloud provider, etc):
docker inspect

...
"Mounts": [
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/sys/fs/cgroup",
                "Destination": "/host/sys/fs/cgroup",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/proc",
                "Destination": "/host/proc",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
]
...

amazonlinux - ami-0c57dafd95a102862

docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<MY_API_KEY> \
              -e DD_APM_ENABLED=true \
              datadog/agent:latest

[rltvty]

also seeing this

[xlucas]

Hello and thanks for reporting this!

I'm able to reproduce this after the container instance boot when systemd-binfmt is in an inactive/dead state.

This service unit file comes with the following conditions:

ConditionPathIsReadWrite=/proc/sys/
ConditionDirectoryNotEmpty=|/lib/binfmt.d
ConditionDirectoryNotEmpty=|/usr/lib/binfmt.d
ConditionDirectoryNotEmpty=|/usr/local/lib/binfmt.d
ConditionDirectoryNotEmpty=|/etc/binfmt.d
ConditionDirectoryNotEmpty=|/run/binfmt.d

Interestingly, by making sure one of these is met or by simply removing them, then restarting the service, the symlink recursion issue is gone.

The most viable approach is to ignore that mount point in the disk check by uncommenting these lines.

An example of passing check configuration to the agent with docker is detailed here.

Hope this helps!

[ndbroadbent]

Hi @xlucas, I'm still seeing these warnings in my logs, and I'm not sure how to change the configuration for the docker agent. I'm using Convox, and I have this in my convox.yml (which is similar to docker-compose.yml):

datadog:
    agent:
      ports:
        - 8125/udp
        - 8126/tcp
    image: datadog/agent:latest
    environment:
      - DD_API_KEY
      - DD_APM_ENABLED=true
      - DD_PROCESS_AGENT_ENABLED=true
    privileged: true
    scale:
      cpu: 128
      memory: 128
    volumes:
      - /sys/fs/cgroup/:/host/sys/fs/cgroup/
      - /proc/:/host/proc/
      - /var/run/docker.sock:/var/run/docker.sock

How can I modify this config to remove these warnings:

WARN | (pkg/collector/python/datadog_agent.go:118 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:75) | Unable to get disk metrics for /host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links: '/host/proc/sys/fs/binfmt_misc'

Thanks!

[xlucas]

Hey @ndbroadbent. I'm not familiar with Convox but what you want to do is to configure the disk check as I was mentioning it in my answer.

In the case of a docker environment, you can do that by adding that file https://github.com/DataDog/integrations-core/blob/master/disk/datadog_checks/disk/data/conf.yaml.default as disk.yaml to a host folder, let's say /home/user/conf.d/disk.yaml, uncomment the specified section, then share this as a volume with your container (docker run [...] -v /home/user/conf.d:/conf.d [...]).

Configuration files present in the container under /conf.d will get copied to /etc/datadog-agent/conf.d before the agent starts. This folder is where the agent will look at to find checks configuration.

[xlucas]

I believe we can close this issue now!

[ianwalter]

I'm getting this error running the agent on Kubernetes. Surely there is a simple way to stop the agent from spamming the logs with this warning?

[caberdoz]

I'm getting this error running the agent on Kubernetes. Surely there is a simple way to stop the agent from spamming the logs with this warning?

I am having the same issue in our Kubernetes cluster and that's pretty annoying. I did some investigation and I don't see how to fix this.

[arthurio]

There doesn't seem to be an obvious way to fix it with the helm chart either. Any guidance?

[johnpekcan]

  volumeMounts:
    - name: emptydir
      mountPath: /etc/datadog-agent/conf.d/disk.d/
  volumes:
    - name: emptydir
      emptyDir: {}

This helped me, give it a go!

[arthurio]

Thanks for sharing your tip @johnpekcan! I actually found another way to make it work:

datadog:
  # See: https://github.com/DataDog/datadog-agent/issues/1961#issuecomment-446918067
  confd:
    disk.yaml: |-
      init_config:

      instances:
        - use_mount: false
          file_system_blacklist:
            - autofs$
          mount_point_blacklist:
            - /proc/sys/fs/binfmt_misc
            - /host/proc/sys/fs/binfmt_misc

[johnpekcan]

Ooh, much more surgical! Thank you 😇

[minnieko]

uncommenting these lines.

Is there a way to pass it as an ENV Variable?
https://docs.datadoghq.com/agent/docker/?tab=standard#installation ?

[jmchuster]

Would definitely be a lot nicer if we could pass in mount_point_blacklist as an environment variable. That works much more nicely than mounting the conf.d file, even if you are on kubernetes (which we are not) and can do so via config. That might even obviate the need for defaulting blacklists (https://github.com/DataDog/integrations-core/issues/2492), just need to add in the documentation common things that various customers might want to blacklist depending on their setup.

[jmchuster]

Our workaround ending up being to explicitly mount the folder. So, we originally just had the mount /proc/:/host/proc/:ro. So we then had to add /proc/sys/fs/binfmt_misc/:/host/proc/sys/fs/binfmt_misc/:ro to get around the symlink issue.