ERROR Loading BPF program and maps into Kernel #44

nickschuetz · 2022-01-07T00:11:24Z

Note that this is a VM on a Mac M1 running Ubuntu 21.10 (GNU/Linux 5.13.0-23-generic aarch64). I compiled this bee binary on the same vm it's running on.

$ file /usr/local/bin/bee
/usr/local/bin/bee: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=7d6I66htAooEyvvjafcO/4smkmG1QZgmhRnqbyK1N/AcgWG-x3EfW5StWmnIUX/_YCuHdBV4WDMGPDjTZML, not stripped

sudo bee run ghcr.io/solo-io/bumblebee/tcpconnect:0.0.9
 SUCCESS  Fetching program from registry: ghcr.io/solo-io/bumblebee/tcpconnect:0.0.9
  ERROR   Loading BPF program and maps into Kernel
Error: program tcp_v4_connect: load program: invalid argument: trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.
R1 type=ctx expected=fp
; int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
2022/01/07 00:17:28 exiting: program tcp_v4_connect: load program: invalid argument: trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.
R1 type=ctx expected=fp
; int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

This same example runs on amd64 w/o issue.

The text was updated successfully, but these errors were encountered:

yuval-k · 2022-01-07T22:55:08Z

weird, works on my linux (5.15.11-200.fc35.x86_64)... could be an arm thing? are the other probes working?

lgadban · 2022-01-08T15:34:27Z

reposting here for posterity, can you try removing the BPF_F_NO_PREALLOC field here: https://github.com/solo-io/bumblebee/blob/main/examples/tcpconnect/tcpconnect.c#L20

and then rebuild the probe locally?

nickschuetz · 2022-01-10T17:44:18Z

Same result:

# bee run tcpconnect
 SUCCESS  Fetching program from registry: tcpconnect
  ERROR   Loading BPF program and maps into Kernel
Error: program tcp_v4_connect_ret: load program: invalid argument: R1 type=ctx expected=fp
; int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
2022/01/10 17:23:12 exiting: program tcp_v4_connect_ret: load program: invalid argument: R1 type=ctx expected=fp
; int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

nickschuetz · 2022-01-10T18:50:47Z

Thanks @lgadban for pointing this out: https://nakryiko.com/posts/bpf-core-reference-guide/#guarding-potentially-failing-relocations

nickschuetz · 2022-01-10T19:23:38Z

After including vmlinux_arm.h:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux_arm.h

and attempting a build I get:

# ./builder/build.sh ./examples/tcpconnect/tcpconnect.c tcpconnect.o
./examples/tcpconnect/tcpconnect.c:107:5: error: no member named 'di' in 'struct pt_regs'
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
<scratch space>:42:1: note: expanded from here
___bpf_kprobe_args1
^
/usr/include/bpf/bpf_tracing.h:355:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:54:32: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) ((x)->di)
                          ~~~  ^
./examples/tcpconnect/tcpconnect.c:113:5: error: no member named 'ax' in 'struct pt_regs'
int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:411:20: note: expanded from macro 'BPF_KRETPROBE'
        return ____##name(___bpf_kretprobe_args(args));                     \
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:395:2: note: expanded from macro '___bpf_kretprobe_args'
        ___bpf_apply(___bpf_kretprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
<scratch space>:58:1: note: expanded from here
___bpf_kretprobe_args1
^
/usr/include/bpf/bpf_tracing.h:393:36: note: expanded from macro '___bpf_kretprobe_args1'
        ___bpf_kretprobe_args0(), (void *)PT_REGS_RC(ctx)
                                          ^~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:61:29: note: expanded from macro 'PT_REGS_RC'
#define PT_REGS_RC(x) ((x)->ax)
                       ~~~  ^
2 errors generated.

yuval-k · 2022-01-10T19:34:48Z

Ahh ok this makes sense; a krpobe that uses the arguments needs to be recompiled for arm, as it uses different registers.
I'm surprised recompiling didn't work, as it should pick up the right arch. @lgadban , maybe we are not passing the right ARCH defines?

yuval-k · 2022-01-10T19:35:34Z

bee should probably validate the architecture of an image before running and emit a warning/error

nickschuetz · 2022-01-10T19:37:12Z

Changing the build.sh from

clang-13 -g -O2 -target bpf -D__TARGET_ARCH_x86 -Wall -c $1 -o $2

to

clang-13 -g -O2 -target bpf -D__TARGET_ARCH_ARM -Wall -c $1 -o $2

worked.

$ ./builder/build.sh ./examples/tcpconnect/tcpconnect.c tcpconnect.o
./examples/tcpconnect/tcpconnect.c:107:5: warning: implicit declaration of function 'PT_REGS_PARM1' is invalid in C99 [-Wimplicit-function-declaration]
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
    ^
/usr/include/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^
/usr/include/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:42:1: note: expanded from here
___bpf_kprobe_args1
^
/usr/include/bpf/bpf_tracing.h:355:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^
./examples/tcpconnect/tcpconnect.c:107:5: warning: cast to 'void *' from smaller integer type 'int' [-Wint-to-void-pointer-cast]
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:42:1: note: expanded from here
___bpf_kprobe_args1
^
/usr/include/bpf/bpf_tracing.h:355:25: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~
./examples/tcpconnect/tcpconnect.c:113:5: warning: implicit declaration of function 'PT_REGS_RC' is invalid in C99 [-Wimplicit-function-declaration]
int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
    ^
/usr/include/bpf/bpf_tracing.h:411:20: note: expanded from macro 'BPF_KRETPROBE'
        return ____##name(___bpf_kretprobe_args(args));                     \
                          ^
/usr/include/bpf/bpf_tracing.h:395:2: note: expanded from macro '___bpf_kretprobe_args'
        ___bpf_apply(___bpf_kretprobe_args, ___bpf_narg(args))(args)
        ^
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:58:1: note: expanded from here
___bpf_kretprobe_args1
^
/usr/include/bpf/bpf_tracing.h:393:36: note: expanded from macro '___bpf_kretprobe_args1'
        ___bpf_kretprobe_args0(), (void *)PT_REGS_RC(ctx)
                                          ^
./examples/tcpconnect/tcpconnect.c:113:5: warning: cast to 'void *' from smaller integer type 'int' [-Wint-to-void-pointer-cast]
int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:411:20: note: expanded from macro 'BPF_KRETPROBE'
        return ____##name(___bpf_kretprobe_args(args));                     \
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:395:2: note: expanded from macro '___bpf_kretprobe_args'
        ___bpf_apply(___bpf_kretprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:58:1: note: expanded from here
___bpf_kretprobe_args1
^
/usr/include/bpf/bpf_tracing.h:393:28: note: expanded from macro '___bpf_kretprobe_args1'
        ___bpf_kretprobe_args0(), (void *)PT_REGS_RC(ctx)
                                  ^~~~~~~~~~~~~~~~~~~~~~~
4 warnings generated.

But now when I run it I get:

$ go run -exec sudo ./bee/main.go run tcpconnect.o
 SUCCESS  Fetching program from file: tcpconnect.o
  ERROR   Loading BPF program and maps into Kernel
Error: program tcp_v4_connect: call at 0: reference to missing symbol "PT_REGS_PARM1"
2022/01/10 19:34:00 exiting: program tcp_v4_connect: call at 0: reference to missing symbol "PT_REGS_PARM1"
exit status 1

lgadban · 2022-01-10T19:57:18Z

Ahh ok this makes sense; a krpobe that uses the arguments needs to be recompiled for arm, as it uses different registers.
I'm surprised recompiling didn't work, as it should pick up the right arch. @lgadban , maybe we are not passing the right ARCH defines?

Yeah we are currently hardcoding to x86 -- we definitely need to make the build more generic

yuval-k · 2022-01-10T20:09:42Z

@nickschuetz try with -D__TARGET_ARCH_arm64. this should work for you.
you can see the full list in https://github.com/torvalds/linux/blob/78d8b34751cf3c61b8dcd6ac40b0fc453de3c6a3/tools/lib/bpf/bpf_tracing.h#L6

nickschuetz · 2022-01-10T20:17:05Z

That did it @yuval-k. A clean build of tcpconnect on arm64 without warnings or errors.

It now runs as expected as well.

lgadban · 2022-01-10T20:33:48Z

tracking first-class arm64 support here: #55

sunfuze · 2023-02-17T05:59:12Z

Also need change vmlinux.h to arm64 version or building raise "error: incomplete definition of type 'struct user_pt_regs'".

nickschuetz changed the title ~~ERROR Failed to load OCI image~~ ERROR Loading BPF program and maps into Kernel Jan 7, 2022

yuval-k added the bug Something isn't working label Jan 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR Loading BPF program and maps into Kernel #44

ERROR Loading BPF program and maps into Kernel #44

nickschuetz commented Jan 7, 2022 •

edited

Loading

yuval-k commented Jan 7, 2022

lgadban commented Jan 8, 2022

nickschuetz commented Jan 10, 2022 •

edited

Loading

nickschuetz commented Jan 10, 2022

nickschuetz commented Jan 10, 2022

yuval-k commented Jan 10, 2022

yuval-k commented Jan 10, 2022

nickschuetz commented Jan 10, 2022 •

edited

Loading

lgadban commented Jan 10, 2022

yuval-k commented Jan 10, 2022 •

edited

Loading

nickschuetz commented Jan 10, 2022 •

edited

Loading

lgadban commented Jan 10, 2022

sunfuze commented Feb 17, 2023

ERROR Loading BPF program and maps into Kernel #44

ERROR Loading BPF program and maps into Kernel #44

Comments

nickschuetz commented Jan 7, 2022 • edited Loading

yuval-k commented Jan 7, 2022

lgadban commented Jan 8, 2022

nickschuetz commented Jan 10, 2022 • edited Loading

nickschuetz commented Jan 10, 2022

nickschuetz commented Jan 10, 2022

yuval-k commented Jan 10, 2022

yuval-k commented Jan 10, 2022

nickschuetz commented Jan 10, 2022 • edited Loading

lgadban commented Jan 10, 2022

yuval-k commented Jan 10, 2022 • edited Loading

nickschuetz commented Jan 10, 2022 • edited Loading

lgadban commented Jan 10, 2022

sunfuze commented Feb 17, 2023

nickschuetz commented Jan 7, 2022 •

edited

Loading

nickschuetz commented Jan 10, 2022 •

edited

Loading

nickschuetz commented Jan 10, 2022 •

edited

Loading

yuval-k commented Jan 10, 2022 •

edited

Loading

nickschuetz commented Jan 10, 2022 •

edited

Loading