Bpftrace: Document bpftrace and kernel lockdown mode

[psanford]

I ran into some issues after upgrading my ubuntu 18.04 kernel to the latest HWE 5.0 version. This issue is mostly to document what happened in case others run into similar problems.

After upgrading my ubuntu 18.04 system to the latest hwe kernel (5.0.0-25-generic) from 4.15 bpftrace stopped working for me. Running it would result in the following:

$ sudo bpftrace -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }'
Error creating printf map: Operation not permitted
Creation of the required BPF maps has failed.
Make sure you have all the required permissions and are not confined (e.g. like
snapcraft does). `dmesg` will likely have useful output for further troubleshooting

Sadly no log messages made it into the dmesg output and I wasn't running in a restricted environment.

After some more experimentation I discovered that kprobes also weren't working for me anymore. Using brendangregg/perf-tools execsnoop errored:

$ sudo ./execsnoop 
Tracing exec()s. Ctrl-C to end.
ERROR: adding a kprobe for execve. Exiting.

This did result in a useful error in dmesg:

[  102.156090] Lockdown: execsnoop: Use of kprobes is restricted; see man kernel_lockdown.7

It turns out that newer ubuntu kernels have applied a version of Matthew Garrett's kernel lockdown patch series[1] that hasn't yet been merged to the mainline kernel. This restricts a number of kernel APIs when you boot from UEFI secure boot with a signed kernel. Those APIs include kprobe and bpf.

There's a sysrq key combination you can send to disable lockdown mode. First make sure that sysrq is enabled:

echo 1 | sudo tee /proc/sys/kernel/sysrq

Then press alt-sysrq-x to disable lockdown mode (on my Lenovo T480s keyboard Fn-s sends the sysrq key).

You should then see the following in dmesg:

[ 1330.151812] sysrq: SysRq : Disabling Secure Boot restrictions
[ 1330.151827] Lifting lockdown

It sounds like if the lockdown patches do get merged into mainline it will probably be less restrictive by default than what the ubuntu version of the patch series does.

[psanford]

I'm going to close this as I don't think there's anything to be done in bpftrace.

[fbs]

his restricts a number of kernel APIs when you boot from UEFI secure boot with a signed kernel. Those APIs include kprobe and bpf.

If a version of lockdown is going to end up in mainline it would be nice if we could detect this mode and generate an appropriate error, like we do for running without root.

[dailyherold]

@psanford thank you for this debugging! I've spend last two nights trying to figure out why my freshly 19.10 install was having so much trouble with bpftrace and bcc. Had been trying all install methods from build-from-source, to docker, to snap packages, across multiple versions and tags. Disabling lockdown mode did it for me and now all the tools working. Good idea to check perf-tools which generated the helpful dmesg, which I have also been watching but not getting anything helpful.

❯ uname -r
5.3.0-29-generic

[brendangregg]

Since Ubuntu 19.04 (5.0 kernel) BPF is broken when Lockdown is enabled, which is for UEFI Secure Boot. That won't be the case on, say, AWS EC2 virtual machines.

I've asked Canonical for their recommended way of disabling Lockdown.

[brendangregg]

I've written a note about lockdown here: https://github.com/iovisor/bcc/issues/2565#issuecomment-584476552

[Hi-Angel]

Note: if anyone running into the same error after you've installed bpftrace through snap, you need to execute a snap connect bpftrace:system-trace.

[miaogen123]

Note: if anyone running into the same error after you've installed bpftrace through snap, you need to execute a snap connect bpftrace:system-trace.

great thanks~