Cilium: BPF load fail: invalid bpf_context access

[Jianlin-lv]

General Information

• Cilium version : the latest master branch
• Kernel version
  uname -a
  Linux net-arm-thunderx2-04 4.20.16-042016-generic #201903132232 SMP Wed Mar 13 23:09:29 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

• Orchestration system version in use
  kubectl version
  Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2",

• Link to relevant artifacts (policies, deployments scripts, ...)

• Generate and upload a system zip:

curl -sLO https://git.io/cilium-sysdump-latest.zip && python cilium-sysdump-latest.zip

cilium-sysdump-20200907-231129.zip

How to reproduce the issue

kubectl apply -f quick-install.yaml

Log info:

level=warning msg="+ clang -O2 -g -target bpf -std=gnu89 -nostdinc -emit-llvm -Wall -Wextra -Werror -Wshadow -Wno-address-of-packed-member -Wno-unknown-warning-option -Wno-gnu-variable-sized-type-not-at-end -Wdeclaration-after-statement -I. -I/run/cilium/state/globals -I/var/lib/cilium/bpf -I/var/lib/cilium/bpf/include -D__NR_CPUS__=224 -DENABLE_ARP_RESPONDER=1 -DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB '-DNODE_MAC={.addr={0x7a,0xa2,0x75,0xd6,0xb3,0xcf}}' -DCALLS_MAP=cilium_calls_overlay_2 -c /var/lib/cilium/bpf/bpf_overlay.c -o -" subsys=datapath-loader
level=warning msg="+ llc -march=bpf -mcpu=v2 -mattr=dwarfris -filetype=obj -o bpf_overlay.o" subsys=datapath-loader
level=warning msg="+ tc qdisc replace dev cilium_vxlan clsact" subsys=datapath-loader
level=warning msg="++ tc filter show dev cilium_vxlan ingress" subsys=datapath-loader
level=warning msg="++ grep -v 'pref 1 bpf chain 0 $\\|pref 1 bpf chain 0 handle 0x1'" subsys=datapath-loader
level=warning msg="+ '[' -z '' ']'" subsys=datapath-loader
level=warning msg="+ cilium-map-migrate -s bpf_overlay.o" subsys=datapath-loader
level=warning msg="+ set +e" subsys=datapath-loader
level=warning msg="+ tc filter replace dev cilium_vxlan ingress prio 1 handle 1 bpf da obj bpf_overlay.o sec from-overlay" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="BTF debug data section '.BTF' rejected: Invalid argument (22)!" subsys=datapath-loader
level=warning msg=" - Length:       22780" subsys=datapath-loader
level=warning msg="Verifier analysis:" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="magic: 0xeb9f" subsys=datapath-loader
level=warning msg="version: 1" subsys=datapath-loader
level=warning msg="flags: 0x0" subsys=datapath-loader
level=warning msg="hdr_len: 24" subsys=datapath-loader
level=warning msg="type_off: 0" subsys=datapath-loader
level=warning msg="type_len: 1608" subsys=datapath-loader
level=warning msg="str_off: 1608" subsys=datapath-loader
level=warning msg="str_len: 21148" subsys=datapath-loader
level=warning msg="btf_total_size: 22780" subsys=datapath-loader
level=warning msg="[1] PTR (anon) type_id=2" subsys=datapath-loader
level=warning msg="[2] STRUCT __sk_buff size=184 vlen=32" subsys=datapath-loader
level=warning msg="\tlen type_id=3 bits_offset=0" subsys=datapath-loader
level=warning msg="\tpkt_type type_id=3 bits_offset=32" subsys=datapath-loader
level=warning msg="\tmark type_id=3 bits_offset=64" subsys=datapath-loader
level=warning msg="\tqueue_mapping type_id=3 bits_offset=96" subsys=datapath-loader
level=warning msg="\tprotocol type_id=3 bits_offset=128" subsys=datapath-loader
level=warning msg="\tvlan_present type_id=3 bits_offset=160" subsys=datapath-loader
level=warning msg="\tvlan_tci type_id=3 bits_offset=192" subsys=datapath-loader
level=warning msg="\tvlan_proto type_id=3 bits_offset=224" subsys=datapath-loader
level=warning msg="\tpriority type_id=3 bits_offset=256" subsys=datapath-loader
level=warning msg="\tingress_ifindex type_id=3 bits_offset=288" subsys=datapath-loader
level=warning msg="\tifindex type_id=3 bits_offset=320" subsys=datapath-loader
level=warning msg="\ttc_index type_id=3 bits_offset=352" subsys=datapath-loader
level=warning msg="\tcb type_id=5 bits_offset=384" subsys=datapath-loader
level=warning msg="\thash type_id=3 bits_offset=544" subsys=datapath-loader
level=warning msg="\ttc_classid type_id=3 bits_offset=576" subsys=datapath-loader
level=warning msg="\tdata type_id=3 bits_offset=608" subsys=datapath-loader
level=warning msg="\tdata_end type_id=3 bits_offset=640" subsys=datapath-loader
level=warning msg="\tnapi_id type_id=3 bits_offset=672" subsys=datapath-loader
level=warning msg="\tfamily type_id=3 bits_offset=704" subsys=datapath-loader
level=warning msg="\tremote_ip4 type_id=3 bits_offset=736" subsys=datapath-loader
level=warning msg="\tlocal_ip4 type_id=3 bits_offset=768" subsys=datapath-loader
level=warning msg="\tremote_ip6 type_id=7 bits_offset=800" subsys=datapath-loader
level=warning msg="\tlocal_ip6 type_id=7 bits_offset=928" subsys=datapath-loader
level=warning msg="\tremote_port type_id=3 bits_offset=1056" subsys=datapath-loader
level=warning msg="\tlocal_port type_id=3 bits_offset=1088" subsys=datapath-loader
level=warning msg="\tdata_meta type_id=3 bits_offset=1120" subsys=datapath-loader
level=warning msg="\t(anon) type_id=8 bits_offset=1152" subsys=datapath-loader
level=warning msg="\ttstamp type_id=10 bits_offset=1216" subsys=datapath-loader
level=warning msg="\twire_len type_id=3 bits_offset=1280" subsys=datapath-loader
level=warning msg="\tgso_segs type_id=3 bits_offset=1312" subsys=datapath-loader
level=warning msg="\t(anon) type_id=12 bits_offset=1344" subsys=datapath-loader
level=warning msg="\tgso_size type_id=3 bits_offset=1408" subsys=datapath-loader
level=warning msg="[3] TYPEDEF __u32 type_id=4" subsys=datapath-loader
level=warning msg="[4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)" subsys=datapath-loader
level=warning msg="[5] ARRAY (anon) type_id=3 index_type_id=6 nr_elems=5" subsys=datapath-loader
level=warning msg="[6] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)" subsys=datapath-loader
level=warning msg="[7] ARRAY (anon) type_id=3 index_type_id=6 nr_elems=4" subsys=datapath-loader
level=warning msg="[8] UNION (anon) size=8 vlen=1" subsys=datapath-loader
level=warning msg="\tflow_keys type_id=9 bits_offset=0" subsys=datapath-loader
level=warning msg="[9] PTR (anon) type_id=54" subsys=datapath-loader
level=warning msg="[10] TYPEDEF __u64 type_id=11" subsys=datapath-loader
level=warning msg="[11] INT long long unsigned int size=8 bits_offset=0 nr_bits=64 encoding=(none)" subsys=datapath-loader
level=warning msg="[12] UNION (anon) size=8 vlen=1" subsys=datapath-loader
level=warning msg="\tsk type_id=13 bits_offset=0" subsys=datapath-loader
level=warning msg="[13] PTR (anon) type_id=55" subsys=datapath-loader
level=warning msg="[14] Invalid kind:13" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Prog section '2/17' rejected: Permission denied (13)!" subsys=datapath-loader
level=warning msg=" - Type:         3" subsys=datapath-loader
level=warning msg=" - Attach Type:  0" subsys=datapath-loader
level=warning msg=" - Instructions: 1119 (0 over limit)" subsys=datapath-loader
level=warning msg=" - License:      GPL" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Verifier analysis:" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Skipped 34167 bytes, use 'verb' option for the full verbose log." subsys=datapath-loader
level=warning msg="[...]" subsys=datapath-loader
level=warning msg="=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R8=inv(id=0) R9=inv0 R10=fp0,call_-1 fp-48=????mmmm fp-56=mmmmmmmm fp-88=????00mm fp-96=??mmmmmm fp-104=mmmmmmmm fp-112=????00mm fp-120=00000000 fp-128=0000mmmm fp-136=00000000 fp-144=00000000 fp-152=00000000 fp-160=00000000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=00000000" subsys=datapath-loader
level=warning msg="925: (07) r1 += 1" subsys=datapath-loader
level=warning msg="926: (7b) *(u64 *)(r0 +0) = r1" subsys=datapath-loader
level=warning msg=" R0=map_value(id=0,off=0,ks=8,vs=16,imm=0) R1_w=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R8=inv(id=0) R9=inv0 R10=fp0,call_-1 fp-48=????mmmm fp-56=mmmmmmmm fp-88=????00mm fp-96=??mmmmmm fp-104=mmmmmmmm fp-112=????00mm fp-120=00000000 fp-128=0000mmmm fp-136=00000000 fp-144=00000000 fp-152=00000000 fp-160=00000000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=00000000" subsys=datapath-loader
level=warning msg="927: (79) r1 = *(u64 *)(r0 +8)" subsys=datapath-loader
level=warning msg=" R0=map_value(id=0,off=0,ks=8,vs=16,imm=0) R1_w=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R8=inv(id=0) R9=inv0 R10=fp0,call_-1 fp-48=????mmmm fp-56=mmmmmmmm fp-88=????00mm fp-96=??mmmmmm fp-104=mmmmmmmm fp-112=????00mm fp-120=00000000 fp-128=0000mmmm fp-136=00000000 fp-144=00000000 fp-152=00000000 fp-160=00000000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=00000000" subsys=datapath-loader
level=warning msg="928: (0f) r1 += r7" subsys=datapath-loader
level=warning msg="929: (7b) *(u64 *)(r0 +8) = r1" subsys=datapath-loader
level=warning msg=" R0=map_value(id=0,off=0,ks=8,vs=16,imm=0) R1_w=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R8=inv(id=0) R9=inv0 R10=fp0,call_-1 fp-48=????mmmm fp-56=mmmmmmmm fp-88=????00mm fp-96=??mmmmmm fp-104=mmmmmmmm fp-112=????00mm fp-120=00000000 fp-128=0000mmmm fp-136=00000000 fp-144=00000000 fp-152=00000000 fp-160=00000000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=00000000" subsys=datapath-loader
level=warning msg="930: (05) goto pc+121" subsys=datapath-loader
level=warning msg="1052: (18) r7 = 0xffff80bd66ee7000" subsys=datapath-loader
level=warning msg="1054: (bf) r1 = r6" subsys=datapath-loader
level=warning msg="1055: (bf) r2 = r7" subsys=datapath-loader
level=warning msg="1056: (b7) r3 = 1" subsys=datapath-loader
level=warning msg="1057: (85) call bpf_tail_call#12" subsys=datapath-loader
level=warning msg="1058: (b7) r0 = 2" subsys=datapath-loader
level=warning msg="1059: (05) goto pc-124" subsys=datapath-loader
level=warning msg="936: (95) exit" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="from 923 to 1041: R0=inv0 R6=ctx(id=0,off=0,imm=0) R7=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R8=inv(id=0) R9=inv0 R10=fp0,call_-1 fp-48=????mmmm fp-56=mmmmmmmm fp-88=????00mm fp-96=??mmmmmm fp-104=mmmmmmmm fp-112=????00mm fp-120=00000000 fp-128=0000mmmm fp-136=00000000 fp-144=00000000 fp-152=00000000 fp-160=00000000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=00000000" subsys=datapath-loader
level=warning msg="1041: (7b) *(u64 *)(r10 -184) = r7" subsys=datapath-loader
level=warning msg="1042: (b7) r1 = 1" subsys=datapath-loader
level=warning msg="1043: (7b) *(u64 *)(r10 -192) = r1" subsys=datapath-loader
level=warning msg="1044: (bf) r2 = r10" subsys=datapath-loader
level=warning msg="1045: (07) r2 += -56" subsys=datapath-loader
level=warning msg="1046: (bf) r3 = r10" subsys=datapath-loader
level=warning msg="1047: (07) r3 += -192" subsys=datapath-loader
level=warning msg="1048: (18) r1 = 0xffff809ea9594800" subsys=datapath-loader
level=warning msg="1050: (b7) r4 = 0" subsys=datapath-loader
level=warning msg="1051: (85) call bpf_map_update_elem#2" subsys=datapath-loader
level=warning msg="1052: safe" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="from 570 to 894: R0=inv(id=0) R1=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R2=inv(id=0,umax_value=8,var_off=(0x0; 0x8)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0) R8=map_value(id=0,off=0,ks=14,vs=56,imm=0) R9=inv0 R10=fp0,call_-1 fp-48=????mmmm fp-56=mmmmmmmm fp-88=????00mm fp-96=??mmmmmm fp-104=mmmmmmmm fp-112=????00mm fp-120=00000000 fp-128=0000mmmm fp-136=00000000 fp-144=00000000 fp-152=00000000 fp-160=00000000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=00000000" subsys=datapath-loader
level=warning msg="894: (b7) r8 = 0" subsys=datapath-loader
level=warning msg="895: (bf) r1 = r8" subsys=datapath-loader
level=warning msg="896: (67) r1 <<= 32" subsys=datapath-loader
level=warning msg="897: (77) r1 >>= 32" subsys=datapath-loader
level=warning msg="898: (b7) r2 = 1" subsys=datapath-loader
level=warning msg="899: (15) if r1 == 0x2 goto pc+1" subsys=datapath-loader
level=warning msg="900: (b7) r2 = 0" subsys=datapath-loader
level=warning msg="901: (77) r1 >>= 31" subsys=datapath-loader
level=warning msg="902: (4f) r1 |= r2" subsys=datapath-loader
level=warning msg="903: (15) if r1 == 0x0 goto pc+27" subsys=datapath-loader
level=warning msg="931: (b7) r1 = 0" subsys=datapath-loader
level=warning msg="932: (63) *(u32 *)(r6 +12) = r1" subsys=datapath-loader
level=warning msg="invalid bpf_context access off=12 size=4" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Error filling program arrays!" subsys=datapath-loader
level=warning msg="Unable to load program" subsys=datapath-loader
level=warning msg="+ RETCODE=1" subsys=datapath-loader
level=warning msg="+ set -e" subsys=datapath-loader
level=warning msg="+ cilium-map-migrate -e bpf_overlay.o -r 1" subsys=datapath-loader
level=warning msg="+ return 1" subsys=datapath-loader
level=error msg="Error while initializing daemon" error="exit status 1" subsys=daemon
level=fatal msg="Error while creating daemon" error="exit status 1" subsys=daemon

This issue is triggered by loading bpf_overlay.o:

tc filter replace dev cilium_vxlan entry prio 1 handle 1 bpf da obj bpf_overlay.o sec from-overlay

I added the "-g" option when compiling the bpf program, and got the disassembly file of bpf_overlay.o
bpf_overlay_objdump.log

The relevant disassembly information is as follows:

 5572 0000000000001d18 LBB2_149:
 5573      931:       r1 = 0
 5574 ;       ctx->queue_mapping = aggregate;
 5575      932:       *(u32 *)(r6 + 12) = r1
 5576 ;       return redirect(ifindex, flags);
 5577      933:       r1 = r9
 5578      934:       r2 = 0
 5579      935:       call 23

The corresponding source code should be:

tail_rev_nodeport_lb4
->edt_set_aggregate
    ->ctx->queue_mapping = aggregate;

Please correct me if I have an error here.

I deployed the same cilium image on both the old kernel version(Linux net-arm-amp-01 4.18.20 #1 SMP Wed May 29 16:40:00 CST 2019 aarch64 aarch64 aarch64 GNU/Linux) and the new kernel version(Linux jianlin 5.9.0-rc1+ #3 SMP Thu Aug 20 09:05:37 UTC 2020 aarch64 aarch64 aarch64), and it works normally.
These are all tests done on Arm64.

I don’t know why this error occurs, please give some hints.
Thanks.

[Jianlin-lv]

quick-install-9-7.yaml.log

Attach quick install yaml file

[pchaigno]

The ctx->queue_mapping field is only writable from BPF since commit 74e31ca ("bpf: add skb->queue_mapping write access from tc clsact") which was released as part of Linux 5.1:

$ git describe --contains 74e31ca850c1c
v5.1-rc1~178^2~17^2~22

We already have a check that the kernel supports bpf_sk_release() which was released as part of Linux 5.0. We could maybe simply require Linux 5.1 instead. @borkmann What do you think?
https://github.com/cilium/cilium/blob/c39cce931c212e21c3ed3921d861825cad9ad702/pkg/bandwidth/bandwidth.go#L61-L65

[borkmann]

The ctx->queue_mapping field is only writable from BPF since commit 74e31ca ("bpf: add skb->queue_mapping write access from tc clsact") which was released as part of Linux 5.1:

$ git describe --contains 74e31ca850c1c
v5.1-rc1~178^2~17^2~22

We already have a check that the kernel supports bpf_sk_release() which was released as part of Linux 5.0. We could maybe simply require Linux 5.1 instead. @borkmann What do you think?
https://github.com/cilium/cilium/blob/c39cce931c212e21c3ed3921d861825cad9ad702/pkg/bandwidth/bandwidth.go#L61-L65

Ah yes, that makes sense. I can send a fix. Thx!