bcc tools inside a docker container?

[saobao913]

Trying to run 'tcplife' inside a docker container with latest stable release v0.6.0
bcc tools built from sources.

Ubuntu bionic
4.15.0-24-generic #26-Ubuntu SMP Wed Jun 13 08:44:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Docker version 17.12.1-ce, build 7390fc6

# python3 ./tcplife -w

1) It SUCCEEDS if I add --privileged
But this is excessive

2) It FAILS if I add only CAP_SYS_ADMIN with the following error:

**open(/sys/kernel/debug/tracing/kprobe_events): Permission denied**
Traceback (most recent call last):
  File "./tcplife", line 479, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 329, in __init__
    self._trace_autoload()
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 1010, in _trace_autoload
    self.attach_kprobe(event=fn.name[8:], fn_name=fn.name)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 575, in attach_kprobe
    raise Exception("Failed to attach BPF to kprobe")
Exception: Failed to attach BPF to kprobe

Thoughts?

[saobao913]

In general, what are the implications of running bcc tools inside Linux containers?

[pchaigno]

BPF programs for kprobes and tracepoints require privileges. They can at least read kernel memory and leak kernel pointers; do you really want an unprivileged user to be able to do that?

[Julio-Guerra]

Useful to build and start a "debugging container" with all the tools inside. But so far I couldn't receive anything form a "sibling container". But it works with "non containerized" processes.

I wonder if I should use the process id docker top gives.

For now, I simply mount into the "debugging container the required" /lib/modules/<version> folders to be able to compile BPF programs, /sys/kernel/debug for debugfs and /proc for procfs. What else should I do?

I'll probably end up mounting every cgroup-related path to try.

[yzhao1012]

Hi All, sorry to post on an old issue.

Was this resolved? I tried to mount the various directory, and still the kprobe attach failed:
docker run -it --rm --name=test -v /sys:/sys:rw -v $PWD/bazel-bin/src/stirling/:/pl/src/pixielabs.ai/pixielabs -v /lib/modules:/lib/modules -v /usr/src:/usr/src --cap-add=SYS_PTRACE --cap-add=SYS_ADMIN test:latest bash
// Code
BPF bpf;
bpf.attach_kprobe(...);
// This still failed with
// open(/sys/kernel/debug/tracing/kprobe_events): Permission denied
root@bda878f536a4:/pl/src/pixielabs.ai/pixielabs# ls -ld /sys/kernel/debug/tracing/kprobe_events-rw-r--r-- 1 root root 0 May 1 15:46 /sys/kernel/debug/tracing/kprobe_events
// I am running as root:
root@bda878f536a4:/pl/src/pixielabs.ai/pixielabs# whoami
root

Any idea?

[alban]

@yzhao1012 does it work with --privileged? Just to make sure some other capability is not missing.

I can run bcc scripts in a container successfully using this kind of commands:

docker run --rm -v /sys/kernel/debug:/sys/kernel/debug -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/fs/bpf:/sys/fs/bpf --privileged myimagename

[yzhao1012]

Yeah! --privileged works for me, after I added it to my command line.
Thanks!

[yzhao1012]

A brief summary:
I tried --cap_add ALL, without --previleged, and still runs into the same issue. According to https://docs.docker.com/v17.09/engine/reference/commandline/run/:

"""
The --privileged flag gives all capabilities to the container, and it also lifts all the limitations enforced by the device cgroup controller. In other words, the container can then do almost everything that the host can do. This flag exists to allow special use-cases, like running Docker within Docker.
"""

So it seems the limitation is from "the device cgroup controller", as ALL capabilities do not enable the access.

[alban]

So it seems the limitation is from "the device cgroup controller", as ALL capabilities do not enable the access.

I suspect it might be a LSM (AppArmor or SELinux?) instead of the device cgroup controller: --privileged also affects LSMs.