[PATCH openEuler-22.09 000/379] BPF CO-RE support

Backport BPF CO-RE support patches for openEuler-22.09

Alan Maguire (5): libbpf: bpf__find_by_name[_kind] should use btf__get_nr_types() libbpf: BTF dumper support for typed data libbpf: Clarify/fix unaligned data issues for btf typed dump libbpf: Avoid use of __int128 in typed dump display libbpf: Propagate errors when retrieving enum value for typed data display

Alexei Starovoitov (47): bpf: Optimize program stats bpf: Run sleepable programs with migration disabled bpf: Compute program stats for sleepable programs bpf: Add per-program recursion prevention mechanism selftest/bpf: Add a recursion test bpf: Count the number of times recursion was prevented selftests/bpf: Improve recursion selftest bpf: Allows per-cpu maps and map-in-map in sleepable programs selftests/bpf: Add a test for map-in-map and per-cpu maps in sleepable progs bpf: Clear per_cpu pointers during bpf_prog_realloc bpf: Dont allow vmlinux BTF to be used in map_create and prog_load. libbpf: Remove unused field. bpf: Introduce bpf_sys_bpf() helper and program type. bpf: Introduce bpfptr_t user/kernel pointer. bpf: Prepare bpf syscall to be used from kernel and user space. libbpf: Support for syscall program type bpf: Make btf_load command to be bpfptr_t compatible. bpf: Introduce fd_idx bpf: Add bpf_btf_find_by_name_kind() helper. bpf: Add bpf_sys_close() helper. libbpf: Change the order of data and text relocations. libbpf: Add bpf_object pointer to kernel_supports(). libbpf: Preliminary support for fd_idx libbpf: Generate loader program out of BPF ELF file. libbpf: Cleanup temp FDs when intermediate sys_bpf fails. libbpf: Introduce bpf_map__initial_value(). bpftool: Use syscall/loader program in "prog load" and "gen skeleton" command. bpf: Add cmd alias BPF_PROG_RUN bpf: Prepare bpf_prog_put() to be called from irq context. bpf: Factor out bpf_spin_lock into helpers. libbpf: Cleanup the layering between CORE and bpf_program. libbpf: Split bpf_core_apply_relo() into bpf_program independent helper. libbpf: Move CO-RE types into relo_core.h. libbpf: Split CO-RE logic into relo_core.c. libbpf: Make gen_loader data aligned. libbpf: Replace btf__type_by_id() with btf_type_by_id(). bpf: Rename btf_member accessors. bpf: Prepare relo_core.c for kernel duty. bpf: Define enum bpf_core_relo_kind as uapi. bpf: Pass a set of bpf_core_relo-s to prog_load command. bpf: Add bpf_core_add_cands() and wire it into bpf_core_apply_relo_insn(). libbpf: Use CO-RE in the kernel in light skeleton. libbpf: Support init of inner maps in light skeleton. libbpf: Clean gen_loader's attach kind. libbpf: Reduce bpf_core_apply_relo_insn() stack usage. bpf: Silence purge_cand_cache build warning. libbpf: Fix gen_loader assumption on number of programs.

Andrei Matei (1): libbpf: Fail early when loading programs with unspecified type

Andrew Delgadillo (1): selftests/bpf: Drop the need for LLVM's llc

Andrii Nakryiko (149): libbpf: Factor out common operations in BTF writing APIs selftest/bpf: Relax btf_dedup test checks libbpf: Unify and speed up BTF string deduplication libbpf: Implement basic split BTF support selftests/bpf: Add split BTF basic test selftests/bpf: Add checking of raw type dump in BTF writer APIs selftests libbpf: Support BTF dedup of split BTFs libbpf: Accomodate DWARF/compiler bug with duplicated identical arrays selftests/bpf: Add split BTF dedup selftests tools/bpftool: Add bpftool support for split BTF bpf: Add in-kernel split BTF support bpf: Assign ID to vmlinux BTF and return extra info for BTF in GET_OBJ_INFO kbuild: Build kernel module BTFs if BTF is enabled and pahole supports it bpf: Load and verify kernel module BTFs tools/bpftool: Add support for in-kernel and named BTF in `btf show` bpf: Compile out btf_parse_module() if module BTF is not enabled bpf: Sanitize BTF data pointer after module is loaded tools/bpftool: Emit name <anon> for anonymous BTFs libbpf: Add base BTF accessor tools/bpftool: Auto-detect split BTFs in common cases bpf: Keep module's btf_data_size intact after load libbpf: Add internal helper to load BTF data by FD libbpf: Refactor CO-RE relocs to not assume a single BTF object libbpf: Add kernel module BTF support for CO-RE relocations selftests/bpf: Add bpf_testmod kernel module for testing selftests/bpf: Add support for marking sub-tests as skipped selftests/bpf: Add CO-RE relocs selftest relying on kernel module BTF bpf: Remove hard-coded btf_vmlinux assumption from BPF verifier bpf: Allow to specify kernel module BTFs when attaching BPF programs libbpf: Factor out low-level BPF program loading helper libbpf: Support attachment of BPF tracing programs to kernel modules selftests/bpf: Add tp_btf CO-RE reloc test for modules selftests/bpf: Add fentry/fexit/fmod_ret selftest for kernel module kbuild: Skip module BTF generation for out-of-tree external modules selftests/bpf: fix bpf_testmod.ko recompilation logic libbpf: Support modules in bpf_program__set_attach_target() API selftests/bpf: Work-around EBUSY errors from hashmap update/delete bpf: Allow empty module BTFs libbpf: Add user-space variants of BPF_CORE_READ() family of macros libbpf: Add non-CO-RE variants of BPF_CORE_READ() macro family selftests/bpf: Add tests for user- and non-CO-RE BPF_CORE_READ() variants libbpf: Clarify kernel type use with USER variants of CORE reading macros selftests/bpf: Sync RCU before unloading bpf_testmod bpf: Support BPF ksym variables in kernel modules libbpf: Support kernel module ksym externs selftests/bpf: Test kernel module ksym externs selftests/bpf: Don't exit on failed bpf_testmod unload libbpf: Stop using feature-detection Makefiles libbpf: provide NULL and KERNEL_VERSION macros in bpf_helpers.h libbpf: Expose btf_type_by_id() internally libbpf: Generalize BTF and BTF.ext type ID and strings iteration libbpf: Rename internal memory-management helpers libbpf: Extract internal set-of-strings datastructure APIs libbpf: Add generic BTF type shallow copy API libbpf: Add BPF static linker APIs libbpf: Add BPF static linker BTF and BTF.ext support bpftool: Add ability to specify custom skeleton object name bpftool: Add `gen object` command to perform BPF static linking selftests/bpf: Pass all BPF .o's through BPF static linker selftests/bpf: Add multi-file statically linked BPF object file test libbpf: Skip BTF fixup if object file has no BTF libbpf: Constify few bpf_program getters libbpf: Preserve empty DATASEC BTFs during static linking libbpf: Fix memory leak when emitting final btf_ext libbpf: Add bpf_map__inner_map API libbpf: Suppress compiler warning when using SEC() macro with externs libbpf: Mark BPF subprogs with hidden visibility as static for BPF verifier libbpf: Allow gaps in BPF program sections to support overriden weak functions libbpf: Refactor BTF map definition parsing libbpf: Factor out symtab and relos sanity checks libbpf: Make few internal helpers available outside of libbpf.c libbpf: Extend sanity checking ELF symbols with externs validation libbpf: Tighten BTF type ID rewriting with error checking libbpf: Add linker extern resolution support for functions and global variables libbpf: Support extern resolution for BTF-defined maps in .maps section libbpf: Support BTF_KIND_FLOAT during type compatibility checks in CO-RE bpftool: Strip const/volatile/restrict modifiers from .bss and .data vars libbpf: Add per-file linker opts selftests/bpf: Stop using static variables for passing data to/from user-space bpftool: Stop emitting static variables in BPF skeleton libbpf: Fix ELF symbol visibility update logic libbpf: Treat STV_INTERNAL same as STV_HIDDEN for functions libbpf: Reject static maps libbpf: Reject static entry-point BPF programs libbpf: Add libbpf_set_strict_mode() API to turn on libbpf 1.0 behaviors libbpf: Streamline error reporting for low-level APIs libbpf: Streamline error reporting for high-level APIs bpftool: Set errno on skeleton failures and propagate errors libbpf: Move few APIs from 0.4 to 0.5 version libbpf: Refactor header installation portions of Makefile libbpf: Install skel_internal.h header used from light skeletons selftests/bpf: Add remaining ASSERT_xxx() variants libbpf: Fix build with latest gcc/binutils with LTO libbpf: Make libbpf_version.h non-auto-generated selftests/bpf: Update selftests to always provide "struct_ops" SEC libbpf: Ensure BPF prog types are set before relocations libbpf: Simplify BPF program auto-attach code libbpf: Minimize explicit iterator of section definition array libbpf: Use pre-setup sec_def in libbpf_find_attach_btf_id() selftests/bpf: Stop using relaxed_core_relocs which has no effect libbpf: Deprecated bpf_object_open_opts.relaxed_core_relocs libbpf: Allow skipping attach_func_name in bpf_program__set_attach_target() libbpf: Schedule open_opts.attach_prog_fd deprecation since v0.7 libbpf: Constify all high-level program attach APIs selftests/bpf: Turn on libbpf 1.0 mode and fix all IS_ERR checks selftests/bpf: Switch fexit_bpf2bpf selftest to set_attach_target() API libbpf: Add "tc" SEC_DEF which is a better name for "classifier" libbpf: Add API that copies all BTF types from one BTF object to another libbpf: Deprecate btf__finalize_data() and move it into libbpf.c libbpf: Extract ELF processing state into separate struct libbpf: Refactor internal sec_def handling to enable pluggability libbpf: Reduce reliance of attach_fns on sec_def internals libbpf: Use Elf64-specific types explicitly for dealing with ELF libbpf: Remove assumptions about uniqueness of .rodata/.data/.bss maps bpftool: Support multiple .rodata/.data internal maps in skeleton bpftool: Improve skeleton generation for data maps without DATASEC type libbpf: Support multiple .rodata.* and .data.* BPF maps selftests/bpf: Demonstrate use of custom .rodata/.data sections libbpf: Simplify look up by name of internal maps selftests/bpf: Switch to ".bss"/".rodata"/".data" lookups for internal maps libbpf: Fix off-by-one bug in bpf_core_apply_relo() libbpf: Add ability to fetch bpf_program's underlying instructions libbpf: Deprecate multi-instance bpf_program APIs libbpf: Deprecate ambiguously-named bpf_program__size() API libbpf: Detect corrupted ELF symbols section libbpf: Improve sanity checking during BTF fix up libbpf: Validate that .BTF and .BTF.ext sections contain data libbpf: Fix section counting logic libbpf: Improve ELF relo sanitization libbpf: Deprecate bpf_program__load() API libbpf: Rename DECLARE_LIBBPF_OPTS into LIBBPF_OPTS selftests/bpf: Pass sanitizer flags to linker through LDFLAGS libbpf: Free up resources used by inner map definition selftests/bpf: Fix memory leaks in btf_type_c_dump() helper selftests/bpf: Free per-cpu values array in bpf_iter selftest selftests/bpf: Free inner strings index in btf selftest selftests/bpf: Avoid duplicate btf__parse() call libbpf: Load global data maps lazily on legacy kernels libbpf: Fix potential misaligned memory access in btf_ext__new() libbpf: Don't call libc APIs with NULL pointers libbpf: Fix glob_syms memory leak in bpf_linker libbpf: Fix using invalidated memory in bpf_linker selftests/bpf: Fix possible NULL passed to memcpy() with zero size selftests/bpf: Prevent misaligned memory access in get_stack_raw_tp test selftests/bpf: Fix misaligned memory access in queue_stack_map test selftests/bpf: Prevent out-of-bounds stack access in test_bpffs selftests/bpf: Fix misaligned accesses in xdp and xdp_bpf2bpf tests libbpf: Cleanup struct bpf_core_cand. libbpf: Fix non-C89 loop variable declaration in gen_loader.c

Arnaldo Carvalho de Melo (1): libbpf: Provide GELF_ST_VISIBILITY() define for older libelf

Brendan Jackman (15): tools/resolve_btfids: Fix some error messages bpf: Fix cold build of test_progs-no_alu32 bpf: Clarify return value of probe str helpers bpf: x86: Factor out emission of ModR/M for *(reg + off) bpf: x86: Factor out emission of REX byte bpf: x86: Factor out a lookup table for some ALU opcodes bpf: Rename BPF_XADD and prepare to encode other atomics in .imm bpf: Move BPF_STX reserved field check into BPF_STX verifier code bpf: Add BPF_FETCH field / create atomic_fetch_add instruction bpf: Add instructions for atomic_[cmp]xchg bpf: Pull out a macro for interpreting atomic ALU operations bpf: Add bitwise atomic instructions bpf: Add tests for new BPF atomic operations bpf: Document new atomic instructions bpf: Rename fixup_bpf_calls and add some comments

Cong Wang (1): bpf: Clear percpu pointers in bpf_prog_clone_free()

Daniel Xu (1): libbpf: Do not close un-owned FD 0 on errors

Dave Marchevsky (1): bpf: Add verified_insns to bpf_prog_info and fdinfo

Dmitrii Banshchikov (7): bpf: Rename bpf_reg_state variables bpf: Extract nullable reg type conversion into a helper function bpf: Support pointers in global func args selftests/bpf: Add unit tests for pointers in global functions bpf: Drop imprecise log message selftests/bpf: Fix a compiler warning in global func test bpf: Use MAX_BPF_FUNC_REG_ARGS macro

Florent Revest (7): bpf: Be less specific about socket cookies guarantees selftests/bpf: Fix the ASSERT_ERR_PTR macro bpf: Factorize bpf_trace_printk and bpf_seq_printf bpf: Add a ARG_PTR_TO_CONST_STR argument type bpf: Add a bpf_snprintf helper libbpf: Introduce a BPF_SNPRINTF helper macro libbpf: Move BPF_SEQ_PRINTF and BPF_SNPRINTF to bpf_helpers.h

Florian Lehner (2): selftests/bpf: Print reason when a tester could not run a program selftests/bpf: Avoid errno clobbering

Gary Lin (3): bpf,x64: Pad NOPs to make images converge more easily test_bpf: Remove EXPECTED_FAIL flag from bpf_fill_maxinsns11 selftests/bpf: Add verifier tests for x64 jit jump padding

Hao Luo (1): libbpf: Support weak typed ksyms.

Hengqi Chen (8): libbpf: Fix KERNEL_VERSION macro tools/resolve_btfids: Emit warnings and patch zero id for missing symbols libbpf: Add btf__load_vmlinux_btf/btf__load_module_btf libbpf: Support uniform BTF-defined key/value specification across all BPF maps libbpf: Deprecate bpf_object__unload() API since v0.6 libbpf: Deprecate bpf_{map,program}__{prev,next} APIs since v0.7 selftests/bpf: Switch to new bpf_object__next_{map,program} APIs libbpf: Support static initialization of BPF_MAP_TYPE_PROG_ARRAY

Ian Rogers (3): bpf, libbpf: Avoid unused function warning on bpf_tail_call_static tools/bpftool: Add -Wall when building BPF programs libbpf: Add NULL check to add_dummy_ksym_var

Ilya Leoshkevich (5): selftests/bpf: Copy extras in out-of-srctree builds bpf: Add BTF_KIND_FLOAT to uapi libbpf: Fix whitespace in btf_add_composite() comment libbpf: Add BTF_KIND_FLOAT support bpf: Generate BTF_KIND_FLOAT when linking vmlinux

Jason Wang (1): libbpf: Fix comment typo

Jean-Philippe Brucker (12): tools: Factor HOSTCC, HOSTLD, HOSTAR definitions tools/runqslower: Use Makefile.include tools/runqslower: Enable out-of-tree build tools/runqslower: Build bpftool using HOSTCC tools/bpftool: Fix build slowdown selftests/bpf: Enable cross-building selftests/bpf: Fix out-of-tree build selftests/bpf: Move generated test files to $(TEST_GEN_FILES) selftests/bpf: Fix installation of urandom_read selftests/bpf: Install btf_dump test cases tools/bpftool: Fix cross-build tools/runqslower: Fix cross-build

Jiri Olsa (3): tools/resolve_btfids: Warn when having multiple IDs for single type libbpf: Use string table index from index table if needed perf build: Move feature cleanup under tools/build

Joe Stringer (1): tools: Sync uapi bpf.h header with latest changes

Jonathan Edwards (1): libbpf: Add extra BPF_PROG_TYPE check to bpf_object__probe_loading

Kumar Kartikeya Dwivedi (14): libbpf: Add various netlink helpers libbpf: Add low level TC-BPF management API libbpf: Remove unneeded check for flags during tc detach libbpf: Set NLM_F_EXCL when creating qdisc libbpf: Fix segfault in static linker for objects without BTF libbpf: Fix segfault in light skeleton for objects without BTF libbpf: Update gen_loader to emit BTF_KIND_FUNC relocations bpf: Add bpf_kallsyms_lookup_name helper libbpf: Add typeless ksym support to gen_loader libbpf: Add weak ksym support to gen_loader libbpf: Perform map fd cleanup for gen_loader in case of error bpf: Change bpf_kallsyms_lookup_name size type to ARG_CONST_SIZE_OR_ZERO libbpf: Avoid double stores for success/failure case of ksym relocations libbpf: Avoid reload of imm for weak, unresolved, repeating ksym

Lorenz Bauer (2): bpf: Consolidate shared test timing code bpf: Add PROG_TEST_RUN support for sk_lookup programs

Martin KaFai Lau (9): bpf: Simplify freeing logic in linfo and jited_linfo bpf: Refactor btf_check_func_arg_match bpf: Support bpf program calling kernel function bpf: Support kernel function call in x86-32 libbpf: Refactor bpf_object__resolve_ksyms_btf_id libbpf: Refactor codes for finding btf id of a kernel symbol libbpf: Rename RELO_EXTERN to RELO_EXTERN_VAR libbpf: Record extern sym relocation first libbpf: Support extern kernel function

Martynas Pumputis (1): selftests/bpf: Check inner map deletion

Matt Smith (3): libbpf: Change bpf_object_skeleton data field to const pointer bpftool: Provide a helper method for accessing skeleton's embedded ELF data selftests/bpf: Add checks for X__elf_bytes() skeleton helper

Mauricio Vásquez (1): libbpf: Fix memory leak in btf__dedup()

Michal Suchanek (1): libbpf: Fix pr_warn type warnings on 32bit

Pedro Tammela (2): libbpf: Avoid inline hint definition from 'linux/stddef.h' libbpf: Clarify flags in ringbuf helpers

Quentin Monnet (17): libbpf: Return non-null error on failures in libbpf_find_prog_btf_id() libbpf: Rename btf__load() as btf__load_into_kernel() libbpf: Rename btf__get_from_id() as btf__load_from_kernel_by_id() tools: Free BTF objects at various locations tools: Replace btf__get_from_id() with btf__load_from_kernel_by_id() libbpf: Add split BTF support for btf__load_from_kernel_by_id() tools: bpftool: Support dumping split BTF by id libbpf: Add LIBBPF_DEPRECATED_SINCE macro for scheduling API deprecations libbpf: Skip re-installing headers file if source is older than target bpftool: Remove unused includes to <bpf/bpf_gen_internal.h> bpftool: Install libbpf headers instead of including the dir tools/resolve_btfids: Install libbpf headers when building tools/runqslower: Install libbpf headers when building bpf: preload: Install libbpf headers when building bpf: iterators: Install libbpf headers when building selftests/bpf: Better clean up for runqslower in test_bpftool_build.sh bpftool: Add install-bin target to install binary only

Rafael David Tinoco (1): libbpf: Add bpf object kern_version attribute setter

Sedat Dilek (1): tools: Factor Clang, LLC and LLVM utils definitions

Shuyi Cheng (2): libbpf: Introduce 'btf_custom_path' to 'bpf_obj_open_opts' libbpf: Add "bool skipped" to struct bpf_map

Song Liu (5): bpf: Use separate lockdep class for each hashtab bpf: Avoid hashtab deadlock with map_locked bpftool: Add Makefile target bootstrap perf build: Support build BPF skeletons with perf perf stat: Enable counting events for BPF programs

Stanislav Fomichev (2): libbpf: Cap retries in sys_bpf_prog_load libbpf: Skip bpf_object__probe_loading for light skeleton

Toke Høiland-Jørgensen (5): bpf: Return target info when a tracing bpf_link is queried libbpf: Restore errno return for functions that were already returning it libbpf: Don't crash on object files with no symbol tables libbpf: Ignore STT_SECTION symbols in 'maps' section libbpf: Properly ignore STT_SECTION symbols in legacy map definitions

Wang Hai (1): libbpf: Simplify the return expression of bpf_object__init_maps function

Wang Qing (1): bpf, btf: Remove the duplicate btf_ids.h include

Wedson Almeida Filho (1): bpf: Refactor check_cfg to use a structured loop.

Yauheni Kaliuta (7): selftests/bpf: test_progs/sockopt_sk: Remove version selftests/bpf: test_progs/sockopt_sk: Convert to use BPF skeleton selftests/bpf: Pass page size from userspace in sockopt_sk selftests/bpf: Pass page size from userspace in map_ptr selftests/bpf: mmap: Use runtime page size selftests/bpf: ringbuf: Use runtime page size selftests/bpf: ringbuf_multi: Use runtime page size

Yonghong Song (27): bpf: Permit cond_resched for some iterators bpf: Permit size-0 datasec bpf: Refactor BPF_PSEUDO_CALL checking as a helper function bpf: Factor out visit_func_call_insn() in check_cfg() bpf: Factor out verbose_invalid_scalar() bpf: Refactor check_func_call() to allow callback function bpf: Change return value of verifier function add_subprog() bpf: Add bpf_for_each_map_elem() helper libbpf: Move function is_ldimm64() earlier in libbpf.c libbpf: Support subprog address relocation selftests/bpf: Fix test_cpp compilation failure with clang bpftool: Fix a clang compilation warning libbpf: Add support for new llvm bpf relocations bpf: Emit better log message if bpf_iter ctx arg btf_id == 0 btf: Change BTF_KIND_* macros to enums bpf: Support for new btf kind BTF_KIND_TAG libbpf: Rename btf_{hash,equal}_int to btf_{hash,equal}_int_tag libbpf: Add support for BTF_KIND_TAG bpftool: Add support for BTF_KIND_TAG bpf: Add BTF_KIND_DECL_TAG typedef support docs/bpf: Update documentation for BTF_KIND_DECL_TAG typedef support bpf: Rename BTF_KIND_TAG to BTF_KIND_DECL_TAG bpf: Support BTF_KIND_TYPE_TAG for btf_type_tag attributes libbpf: Support BTF_KIND_TYPE_TAG bpftool: Support BTF_KIND_TYPE_TAG docs/bpf: Update documentation for BTF_KIND_TYPE_TAG support libbpf: Fix a couple of missed btf_type_tag handling in btf.c

Documentation/ABI/testing/sysfs-kernel-btf | 8 + Documentation/bpf/btf.rst | 40 +- Documentation/networking/filter.rst | 61 +- arch/arm/net/bpf_jit_32.c | 7 +- arch/arm64/net/bpf_jit_comp.c | 16 +- arch/mips/net/ebpf_jit.c | 11 +- arch/powerpc/net/bpf_jit_comp64.c | 25 +- arch/riscv/net/bpf_jit_comp32.c | 20 +- arch/riscv/net/bpf_jit_comp64.c | 16 +- arch/s390/net/bpf_jit_comp.c | 27 +- arch/sparc/net/bpf_jit_comp_64.c | 17 +- arch/x86/net/bpf_jit_comp.c | 408 +- arch/x86/net/bpf_jit_comp32.c | 204 +- drivers/net/ethernet/netronome/nfp/bpf/fw.h | 4 +- drivers/net/ethernet/netronome/nfp/bpf/jit.c | 14 +- drivers/net/ethernet/netronome/nfp/bpf/main.h | 4 +- .../net/ethernet/netronome/nfp/bpf/verifier.c | 15 +- include/linux/bpf.h | 150 +- include/linux/bpf_types.h | 2 + include/linux/bpf_verifier.h | 46 +- include/linux/bpfptr.h | 75 + include/linux/btf.h | 106 +- include/linux/filter.h | 45 +- include/linux/module.h | 4 + include/uapi/linux/bpf.h | 265 +- include/uapi/linux/btf.h | 57 +- kernel/bpf/Makefile | 4 + kernel/bpf/bpf_iter.c | 43 +- kernel/bpf/bpf_struct_ops.c | 6 +- kernel/bpf/btf.c | 1403 ++++- kernel/bpf/core.c | 153 +- kernel/bpf/disasm.c | 56 +- kernel/bpf/hashtab.c | 130 +- kernel/bpf/helpers.c | 326 +- kernel/bpf/preload/Makefile | 25 +- kernel/bpf/preload/iterators/Makefile | 38 +- kernel/bpf/preload/iterators/iterators.bpf.c | 1 - kernel/bpf/syscall.c | 363 +- kernel/bpf/sysfs_btf.c | 2 +- kernel/bpf/task_iter.c | 2 + kernel/bpf/trampoline.c | 77 +- kernel/bpf/verifier.c | 1508 ++++- kernel/module.c | 36 + kernel/trace/bpf_trace.c | 375 +- lib/Kconfig.debug | 9 + lib/test_bpf.c | 21 +- net/bpf/test_run.c | 290 +- net/core/filter.c | 1 + net/ipv4/bpf_tcp_ca.c | 7 +- samples/bpf/bpf_insn.h | 4 +- samples/bpf/cookie_uid_helper_example.c | 8 +- samples/bpf/sock_example.c | 2 +- samples/bpf/test_cgrp2_attach.c | 5 +- samples/bpf/xdp1_user.c | 2 +- samples/bpf/xdp_sample_pkts_user.c | 2 +- scripts/Makefile.modfinal | 25 +- scripts/link-vmlinux.sh | 7 +- .../bpf/bpftool/Documentation/bpftool-gen.rst | 78 +- tools/bpf/bpftool/Makefile | 55 +- tools/bpf/bpftool/bash-completion/bpftool | 17 +- tools/bpf/bpftool/btf.c | 80 +- tools/bpf/bpftool/btf_dumper.c | 6 +- tools/bpf/bpftool/gen.c | 651 +- tools/bpf/bpftool/iter.c | 2 +- tools/bpf/bpftool/main.c | 20 +- tools/bpf/bpftool/main.h | 2 + tools/bpf/bpftool/map.c | 14 +- tools/bpf/bpftool/net.c | 2 +- tools/bpf/bpftool/prog.c | 141 +- tools/bpf/bpftool/xlated_dumper.c | 3 + tools/bpf/resolve_btfids/Makefile | 19 +- tools/bpf/resolve_btfids/main.c | 38 +- tools/bpf/runqslower/Makefile | 71 +- tools/build/Makefile | 8 +- tools/build/Makefile.feature | 4 +- tools/build/feature/Makefile | 4 +- tools/include/linux/filter.h | 24 +- tools/include/uapi/linux/bpf.h | 977 ++- tools/include/uapi/linux/btf.h | 57 +- tools/lib/bpf/.gitignore | 2 - tools/lib/bpf/Build | 2 +- tools/lib/bpf/Makefile | 104 +- tools/lib/bpf/bpf.c | 272 +- tools/lib/bpf/bpf.h | 1 + tools/lib/bpf/bpf_core_read.h | 169 +- tools/lib/bpf/bpf_gen_internal.h | 65 + tools/lib/bpf/bpf_helpers.h | 108 +- tools/lib/bpf/bpf_prog_linfo.c | 18 +- tools/lib/bpf/bpf_tracing.h | 44 +- tools/lib/bpf/btf.c | 2085 ++++--- tools/lib/bpf/btf.h | 99 +- tools/lib/bpf/btf_dump.c | 910 ++- tools/lib/bpf/gen_loader.c | 1126 ++++ tools/lib/bpf/libbpf.c | 5436 +++++++++-------- tools/lib/bpf/libbpf.h | 212 +- tools/lib/bpf/libbpf.map | 54 + tools/lib/bpf/libbpf_common.h | 26 +- tools/lib/bpf/libbpf_errno.c | 7 +- tools/lib/bpf/libbpf_internal.h | 299 +- tools/lib/bpf/libbpf_legacy.h | 60 + tools/lib/bpf/libbpf_version.h | 9 + tools/lib/bpf/linker.c | 2901 +++++++++ tools/lib/bpf/netlink.c | 593 +- tools/lib/bpf/nlattr.h | 48 + tools/lib/bpf/relo_core.c | 1322 ++++ tools/lib/bpf/relo_core.h | 57 + tools/lib/bpf/ringbuf.c | 26 +- tools/lib/bpf/skel_internal.h | 123 + tools/lib/bpf/strset.c | 176 + tools/lib/bpf/strset.h | 21 + tools/lib/bpf/xsk.c | 4 +- tools/perf/Documentation/perf-stat.txt | 18 + tools/perf/Makefile.config | 9 + tools/perf/Makefile.perf | 58 +- tools/perf/builtin-stat.c | 82 +- tools/perf/util/Build | 1 + tools/perf/util/bpf-event.c | 11 +- tools/perf/util/bpf_counter.c | 320 + tools/perf/util/bpf_counter.h | 72 + tools/perf/util/bpf_skel/.gitignore | 3 + .../util/bpf_skel/bpf_prog_profiler.bpf.c | 93 + tools/perf/util/evsel.c | 5 + tools/perf/util/evsel.h | 5 + tools/perf/util/python.c | 21 + tools/perf/util/stat-display.c | 4 +- tools/perf/util/stat.c | 2 +- tools/perf/util/target.c | 34 +- tools/perf/util/target.h | 10 + tools/scripts/Makefile.include | 18 + tools/testing/selftests/bpf/.gitignore | 1 + tools/testing/selftests/bpf/Makefile | 183 +- tools/testing/selftests/bpf/bench.c | 1 + .../selftests/bpf/benchs/bench_rename.c | 2 +- .../selftests/bpf/benchs/bench_ringbufs.c | 6 +- .../selftests/bpf/benchs/bench_trigger.c | 2 +- .../selftests/bpf/bpf_testmod/.gitignore | 6 + .../selftests/bpf/bpf_testmod/Makefile | 20 + .../bpf/bpf_testmod/bpf_testmod-events.h | 36 + .../selftests/bpf/bpf_testmod/bpf_testmod.c | 55 + .../selftests/bpf/bpf_testmod/bpf_testmod.h | 14 + tools/testing/selftests/bpf/btf_helpers.c | 264 + tools/testing/selftests/bpf/btf_helpers.h | 19 + .../selftests/bpf/prog_tests/atomics.c | 246 + .../selftests/bpf/prog_tests/attach_probe.c | 12 +- .../selftests/bpf/prog_tests/bpf_iter.c | 34 +- .../selftests/bpf/prog_tests/bpf_tcp_ca.c | 8 +- tools/testing/selftests/bpf/prog_tests/btf.c | 166 +- .../bpf/prog_tests/btf_dedup_split.c | 325 + .../selftests/bpf/prog_tests/btf_dump.c | 10 +- .../selftests/bpf/prog_tests/btf_endian.c | 4 +- .../selftests/bpf/prog_tests/btf_map_in_map.c | 33 - .../selftests/bpf/prog_tests/btf_split.c | 99 + .../selftests/bpf/prog_tests/btf_write.c | 47 +- .../bpf/prog_tests/cg_storage_multi.c | 84 +- .../bpf/prog_tests/cgroup_attach_multi.c | 6 +- .../selftests/bpf/prog_tests/cgroup_link.c | 16 +- .../bpf/prog_tests/cgroup_skb_sk_lookup.c | 2 +- .../selftests/bpf/prog_tests/core_autosize.c | 2 +- .../bpf/prog_tests/core_read_macros.c | 64 + .../selftests/bpf/prog_tests/core_reloc.c | 105 +- .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 68 +- .../selftests/bpf/prog_tests/fexit_stress.c | 4 +- .../selftests/bpf/prog_tests/flow_dissector.c | 2 +- .../bpf/prog_tests/flow_dissector_reattach.c | 10 +- .../bpf/prog_tests/get_stack_raw_tp.c | 24 +- .../prog_tests/get_stackid_cannot_attach.c | 9 +- .../selftests/bpf/prog_tests/global_data.c | 11 +- .../bpf/prog_tests/global_data_init.c | 2 +- .../bpf/prog_tests/global_func_args.c | 60 + .../selftests/bpf/prog_tests/hashmap.c | 9 +- .../selftests/bpf/prog_tests/kfree_skb.c | 23 +- .../selftests/bpf/prog_tests/ksyms_btf.c | 34 +- .../selftests/bpf/prog_tests/ksyms_module.c | 31 + .../selftests/bpf/prog_tests/link_pinning.c | 7 +- .../selftests/bpf/prog_tests/map_ptr.c | 15 +- tools/testing/selftests/bpf/prog_tests/mmap.c | 24 +- .../selftests/bpf/prog_tests/module_attach.c | 53 + .../selftests/bpf/prog_tests/obj_name.c | 8 +- .../selftests/bpf/prog_tests/perf_branches.c | 4 +- .../selftests/bpf/prog_tests/perf_buffer.c | 2 +- .../bpf/prog_tests/perf_event_stackmap.c | 3 +- .../selftests/bpf/prog_tests/probe_user.c | 7 +- .../selftests/bpf/prog_tests/prog_run_xattr.c | 2 +- .../bpf/prog_tests/queue_stack_map.c | 12 +- .../bpf/prog_tests/raw_tp_test_run.c | 4 +- .../selftests/bpf/prog_tests/rdonly_maps.c | 9 +- .../selftests/bpf/prog_tests/recursion.c | 41 + .../bpf/prog_tests/reference_tracking.c | 2 +- .../selftests/bpf/prog_tests/resolve_btfids.c | 9 +- .../selftests/bpf/prog_tests/ringbuf.c | 17 +- .../selftests/bpf/prog_tests/ringbuf_multi.c | 23 +- .../bpf/prog_tests/select_reuseport.c | 55 +- .../selftests/bpf/prog_tests/send_signal.c | 5 +- .../selftests/bpf/prog_tests/sk_lookup.c | 2 +- .../selftests/bpf/prog_tests/skeleton.c | 41 +- .../selftests/bpf/prog_tests/snprintf_btf.c | 4 +- .../selftests/bpf/prog_tests/sock_fields.c | 14 +- .../selftests/bpf/prog_tests/sockmap_basic.c | 6 +- .../selftests/bpf/prog_tests/sockmap_ktls.c | 2 +- .../selftests/bpf/prog_tests/sockmap_listen.c | 10 +- .../selftests/bpf/prog_tests/sockopt_sk.c | 66 +- .../bpf/prog_tests/stacktrace_build_id_nmi.c | 3 +- .../selftests/bpf/prog_tests/stacktrace_map.c | 2 +- .../bpf/prog_tests/stacktrace_map_raw_tp.c | 5 +- .../selftests/bpf/prog_tests/static_linked.c | 35 + .../bpf/prog_tests/tcp_hdr_options.c | 15 +- .../selftests/bpf/prog_tests/tcp_rtt.c | 2 +- .../selftests/bpf/prog_tests/test_bpffs.c | 4 +- .../bpf/prog_tests/test_global_funcs.c | 8 + .../selftests/bpf/prog_tests/test_overhead.c | 12 +- .../bpf/prog_tests/trampoline_count.c | 18 +- .../selftests/bpf/prog_tests/udp_limit.c | 7 +- tools/testing/selftests/bpf/prog_tests/xdp.c | 11 +- .../selftests/bpf/prog_tests/xdp_bpf2bpf.c | 8 +- .../selftests/bpf/prog_tests/xdp_link.c | 8 +- tools/testing/selftests/bpf/progs/atomics.c | 154 + tools/testing/selftests/bpf/progs/bpf_cubic.c | 6 +- .../bpf/progs/bpf_iter_bpf_hash_map.c | 1 - .../selftests/bpf/progs/bpf_iter_bpf_map.c | 1 - .../selftests/bpf/progs/bpf_iter_ipv6_route.c | 1 - .../selftests/bpf/progs/bpf_iter_netlink.c | 1 - .../selftests/bpf/progs/bpf_iter_task.c | 1 - .../selftests/bpf/progs/bpf_iter_task_btf.c | 1 - .../selftests/bpf/progs/bpf_iter_task_file.c | 1 - .../selftests/bpf/progs/bpf_iter_task_stack.c | 1 - .../selftests/bpf/progs/bpf_iter_tcp4.c | 1 - .../selftests/bpf/progs/bpf_iter_tcp6.c | 1 - .../selftests/bpf/progs/bpf_iter_test_kern4.c | 4 +- .../selftests/bpf/progs/bpf_iter_udp4.c | 1 - .../selftests/bpf/progs/bpf_iter_udp6.c | 1 - .../selftests/bpf/progs/core_reloc_types.h | 17 + tools/testing/selftests/bpf/progs/kfree_skb.c | 4 +- tools/testing/selftests/bpf/progs/lsm.c | 69 + .../selftests/bpf/progs/map_ptr_kern.c | 4 +- tools/testing/selftests/bpf/progs/recursion.c | 46 + .../testing/selftests/bpf/progs/sockopt_sk.c | 11 +- tools/testing/selftests/bpf/progs/tailcall3.c | 2 +- tools/testing/selftests/bpf/progs/tailcall4.c | 2 +- tools/testing/selftests/bpf/progs/tailcall5.c | 2 +- .../selftests/bpf/progs/tailcall_bpf2bpf2.c | 2 +- .../selftests/bpf/progs/tailcall_bpf2bpf4.c | 2 +- .../selftests/bpf/progs/test_cls_redirect.c | 4 +- .../bpf/progs/test_core_read_macros.c | 50 + .../bpf/progs/test_core_reloc_module.c | 96 + .../selftests/bpf/progs/test_global_func10.c | 29 + .../selftests/bpf/progs/test_global_func11.c | 19 + .../selftests/bpf/progs/test_global_func12.c | 21 + .../selftests/bpf/progs/test_global_func13.c | 24 + .../selftests/bpf/progs/test_global_func14.c | 21 + .../selftests/bpf/progs/test_global_func15.c | 22 + .../selftests/bpf/progs/test_global_func16.c | 22 + .../selftests/bpf/progs/test_global_func9.c | 132 + .../bpf/progs/test_global_func_args.c | 91 + .../selftests/bpf/progs/test_ksyms_module.c | 26 + .../selftests/bpf/progs/test_ksyms_weak.c | 56 + .../bpf/progs/test_map_in_map_invalid.c | 26 + tools/testing/selftests/bpf/progs/test_mmap.c | 2 - .../selftests/bpf/progs/test_module_attach.c | 66 + .../selftests/bpf/progs/test_rdonly_maps.c | 6 +- .../selftests/bpf/progs/test_ringbuf.c | 1 - .../selftests/bpf/progs/test_ringbuf_multi.c | 1 - .../selftests/bpf/progs/test_skeleton.c | 20 +- .../selftests/bpf/progs/test_sockmap_listen.c | 2 +- .../selftests/bpf/progs/test_static_linked1.c | 30 + .../selftests/bpf/progs/test_static_linked2.c | 31 + .../selftests/bpf/progs/test_subprogs.c | 13 + .../selftests/bpf/test_bpftool_build.sh | 4 + tools/testing/selftests/bpf/test_btf.h | 3 + .../selftests/bpf/test_cgroup_storage.c | 2 +- tools/testing/selftests/bpf/test_maps.c | 279 +- tools/testing/selftests/bpf/test_progs.c | 79 +- tools/testing/selftests/bpf/test_progs.h | 72 +- .../selftests/bpf/test_tcpnotify_user.c | 7 +- tools/testing/selftests/bpf/test_verifier.c | 103 +- .../selftests/bpf/verifier/atomic_and.c | 77 + .../selftests/bpf/verifier/atomic_cmpxchg.c | 96 + .../selftests/bpf/verifier/atomic_fetch_add.c | 106 + .../selftests/bpf/verifier/atomic_or.c | 77 + .../selftests/bpf/verifier/atomic_xchg.c | 46 + .../selftests/bpf/verifier/atomic_xor.c | 77 + tools/testing/selftests/bpf/verifier/calls.c | 12 +- tools/testing/selftests/bpf/verifier/ctx.c | 7 +- .../selftests/bpf/verifier/dead_code.c | 10 +- .../bpf/verifier/direct_packet_access.c | 4 +- tools/testing/selftests/bpf/verifier/jit.c | 24 + .../testing/selftests/bpf/verifier/leak_ptr.c | 10 +- .../selftests/bpf/verifier/meta_access.c | 4 +- tools/testing/selftests/bpf/verifier/unpriv.c | 3 +- .../bpf/verifier/value_illegal_alu.c | 2 +- tools/testing/selftests/bpf/verifier/xadd.c | 18 +- tools/testing/selftests/bpf/xdping.c | 2 +- tools/testing/selftests/tc-testing/Makefile | 3 +- 292 files changed, 24645 insertions(+), 6370 deletions(-) create mode 100644 include/linux/bpfptr.h create mode 100644 tools/lib/bpf/bpf_gen_internal.h create mode 100644 tools/lib/bpf/gen_loader.c create mode 100644 tools/lib/bpf/libbpf_legacy.h create mode 100644 tools/lib/bpf/libbpf_version.h create mode 100644 tools/lib/bpf/linker.c create mode 100644 tools/lib/bpf/relo_core.c create mode 100644 tools/lib/bpf/relo_core.h create mode 100644 tools/lib/bpf/skel_internal.h create mode 100644 tools/lib/bpf/strset.c create mode 100644 tools/lib/bpf/strset.h create mode 100644 tools/perf/util/bpf_counter.c create mode 100644 tools/perf/util/bpf_counter.h create mode 100644 tools/perf/util/bpf_skel/.gitignore create mode 100644 tools/perf/util/bpf_skel/bpf_prog_profiler.bpf.c create mode 100644 tools/testing/selftests/bpf/bpf_testmod/.gitignore create mode 100644 tools/testing/selftests/bpf/bpf_testmod/Makefile create mode 100644 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod-events.h create mode 100644 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c create mode 100644 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h create mode 100644 tools/testing/selftests/bpf/btf_helpers.c create mode 100644 tools/testing/selftests/bpf/btf_helpers.h create mode 100644 tools/testing/selftests/bpf/prog_tests/atomics.c create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_split.c create mode 100644 tools/testing/selftests/bpf/prog_tests/core_read_macros.c create mode 100644 tools/testing/selftests/bpf/prog_tests/global_func_args.c create mode 100644 tools/testing/selftests/bpf/prog_tests/ksyms_module.c create mode 100644 tools/testing/selftests/bpf/prog_tests/module_attach.c create mode 100644 tools/testing/selftests/bpf/prog_tests/recursion.c create mode 100644 tools/testing/selftests/bpf/prog_tests/static_linked.c create mode 100644 tools/testing/selftests/bpf/progs/atomics.c create mode 100644 tools/testing/selftests/bpf/progs/recursion.c create mode 100644 tools/testing/selftests/bpf/progs/test_core_read_macros.c create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_module.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func10.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func11.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func12.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func13.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func14.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func15.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func16.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func9.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_func_args.c create mode 100644 tools/testing/selftests/bpf/progs/test_ksyms_module.c create mode 100644 tools/testing/selftests/bpf/progs/test_ksyms_weak.c create mode 100644 tools/testing/selftests/bpf/progs/test_map_in_map_invalid.c create mode 100644 tools/testing/selftests/bpf/progs/test_module_attach.c create mode 100644 tools/testing/selftests/bpf/progs/test_static_linked1.c create mode 100644 tools/testing/selftests/bpf/progs/test_static_linked2.c create mode 100644 tools/testing/selftests/bpf/verifier/atomic_and.c create mode 100644 tools/testing/selftests/bpf/verifier/atomic_cmpxchg.c create mode 100644 tools/testing/selftests/bpf/verifier/atomic_fetch_add.c create mode 100644 tools/testing/selftests/bpf/verifier/atomic_or.c create mode 100644 tools/testing/selftests/bpf/verifier/atomic_xchg.c create mode 100644 tools/testing/selftests/bpf/verifier/atomic_xor.c