Bcc: Per-cpu array element size limit (or total size limit)

[yzhao1012]

We use a per-CPU array with size 1, and a custom struct as the data buffer for submitting data to perf buffer:

define MAX_MSG_SIZE 131072 // 128KiB

struct buffer_t {
char msg[MAX_MSG_SIZE];
// Some other fields that is negligible
};
BPF_PERCPU_ARRAY(data_buffer, struct buffer_t, 1);

But when we set the size of the element to a larger value, let's say 128kib, BPF would fail to create the array:
could not open bpf map: data_buffer_heap, error: Cannot allocate memory

I thought there is no limit on the per cpu array size, right?

Did I miss something?

[yonghong-song]

Maybe this is due to locked memory limit?

[willfindlay]

Hi, @yonghong-song I am also having the same issue. Supposing the issue is locked memory limit, what would be the proper way to circumvent this?

[pchaigno]

@willfindlay Try ulimit -l unlimited. See the ulimit man page.

[willfindlay]

@willfindlay Try ulimit -l unlimited. See the ulimit man page.

Thanks for the reply @pchaigno . I've tried with ulimit -l unlimited as well as using the resource module to set the rlimit in the script directly, which (as I understand) should be equivalent. I've also tried messing with the allow_rlimit flag in the BPF constructor. Nothing seems to work.

Here is a minimal example:
bigdata.py

#! /usr/bin/env python3

import ctypes as ct
import os, sys, signal, time, resource

from bcc import BPF
from bcc.libbcc import lib

if __name__ == "__main__":
    print(resource.getrlimit(resource.RLIMIT_MEMLOCK))
    resource.setrlimit(resource.RLIMIT_MEMLOCK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))
    print(resource.getrlimit(resource.RLIMIT_MEMLOCK))

    # note that I get the same result when setting allow_rlimit=True
    bpf = BPF(src_file="bigdata.c", allow_rlimit=False)

    while True:
        try:
            time.sleep(1)
        except KeyboardInterrupt:
            sys.exit(0)

bigdata.c

typedef struct
{
    u8 test[300 * 300];
}
big_data;

BPF_PERCPU_ARRAY(testificate, big_data, 1);

TRACEPOINT_PROBE(raw_syscalls, sys_enter)
{
    int key = 0;

    big_data *d = testificate.lookup(&key);

    return 0;
}

And here is an example of my shell output:

(-1, -1)
(-1, -1)
could not open bpf map: testificate, error: Cannot allocate memory
Traceback (most recent call last):
  File "./bigdata.py", line 13, in <module>
    bpf = BPF(src_file="bigdata.c", allow_rlimit=False)
  File "/usr/lib/python3.7/site-packages/bcc/__init__.py", line 343, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module b'bigdata.c'

I'm guessing locked memory limit is not the culprit then? Am i missing something?

[yzhao1012]

Same here, I tried ulimit -l unlimited, and have the following:
`

ulimit -a

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 126778
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited`

Max locked memory is already unlimited, but the same error still appears.

Note that we this error happens inside and outside of a docker container, just in case it matters.

[willfindlay]

I can also confirm that I have tried it on a few past bcc releases (as far back as 7.0.0), with the same results. Tested on Linux 5.2.9-arch1-1-ARCH and Linux 5.3.0-arch1-1-ARCH.

When I get home today, I am going to test with some older kernels in virtual machines.

[yonghong-song]

There is a limitation for element size for per-cpu map.
The limitation is (in linux/mm/percpu.c)

        if (unlikely(!size || size > PCPU_MIN_UNIT_SIZE || align > PAGE_SIZE ||
                     !is_power_of_2(align))) {
                WARN(do_warn, "illegal size (%zu) or align (%zu) for percpu allocation\n",
                     size, align);
                return NULL;
        }

and

#define PCPU_MIN_UNIT_SIZE               PFN_ALIGN(32 << 10)

So the maximum percpu map element size is 32KB. Anything larger than that will have allocation errors.

[willfindlay]

Thanks for the info. I suppose this would present another excellent use case for adding bpf_spinlock to tracing programs so that a regular bpf_array could be used instead.

[yonghong-song]

@willfindlay Indeed, bpf_spinlock should help in such cases.

[willfindlay]

I've created a pull request to change the documentation to hopefully clarify this #2527 .

[yzhao1012]

Very nice!

Could someone file another issue for https://github.com/iovisor/bcc/issues/2519#issuecomment-534724355 and https://github.com/iovisor/bcc/issues/2519#issuecomment-534735957

I myself do not have any context on how spinlock works in this scenario.

But it sounds like it could be fixed somewhere, such that the limit could be removed?

[willfindlay]

@yzhao1012 I created an issue for spinlock a while ago. It turns out it's actually a Linux kernel problem, not a bcc problem, as bpf_splinlock is actually disabled for tracing programs.

So what we have to do is make our case that bpf_spinlock support for tracing programs would be a useful feature (which we have essentially done here). Somebody just needs to actually make the necessary changes to the kernel and get them merged upstream.

[yzhao1012]

@willfindlay Thanks Will! Do you mind to share the issue you created, I suppose it's created somewhere in the kernel's tracking systems? It would be great if we can learn more context, and possibly contribute further details for the objective of fixing the issue.

[willfindlay]

@yzhao1012 I actually misremembered. It was in the comments on another issue in bcc. I'll post the link here. If you're interested in trying to get this working I'd be happy to help however I can.

[yzhao1012]

Thanks again, I'll need to reach the issue to gain more context. Will reach out if something could be contributed from my side.