This patchset implements initial support of kernel text and rodata replication for AArch64. openEuler 25.03 is used as a baseline. Current implementation supports next functionality: 1. Replicated kernel text and rodata per-NUMA node 2. Vmalloc is able to work with replicated areas, so kernel modules text and rodata also replicated during modules loading stage. 3. BPF handlers are not replicated by default, but this can be easily done by using existent APIs. 4. KPROBES, KGDB and all functionality that depends on kernel text patching work without any limitation. 5. KPTI, KASLR, and KASAN fully supported. 6. Replicates parts of translation table related to replicated text and rodata. 7. 4K and 64K pages are supported. Translation tables synchronization is necessary only in several special cases: 1. Kernel boot 2. Modules deployment 3. Any allocation in user space that require new PUD/P4D In current design mutable kernel data modifications don't require synchronization between translation tables due to on 64-bit platforms all physical memory already mapped in kernel space and this mapping is persistent. In user space the translation tables synchronizations are quite rare due to the only case is new PUD/P4D allocation. At the current stage only the PGD layer is replicated for the user space. Please refer to the next pics. TT overview: NODE 0 NODE 1 USER KERNEL USER KERNEL --------------------- --------------------- PGD | | | | | | | | |*| | | | | | | | | |*| --------------------- --------------------- | | ------------------- ------------------- | | --------------------- --------------------- PUD | | | | | | | |*|*| | | | | | | | |*|*| --------------------- --------------------- | | ------------------- ------------------- | | --------------------- --------------------- PMD |READ-ONLY|MUTABLE | |READ-ONLY|MUTABLE | --------------------- --------------------- | | | | | -------------------------- | | | -------- ------- -------- PHYS | | | | | | MEM -------- ------- -------- <------> <------> NODE 0 Shared NODE 1 between nodes * - entries unique in each table TT synchronization: NODE 0 NODE 1 USER KERNEL USER KERNEL --------------------- --------------------- PGD | | |0| | | | | | | | | |0| | | | | | | --------------------- --------------------- | | | | | PUD_ALLOC / P4D_ALLOC | | IN USERSPACE | \/ --------------------- --------------------- PGD | | |p| | | | | | | | | |p| | | | | | | --------------------- --------------------- | | | | --------------------------- | --------------------- PUD/P4D | | | | | | | | | | --------------------- Known problems: 1. Not all paging modes are supported, for example, the 16K 4-level table is not supported right now. 2. Replication support in vmalloc, possibly, can be optimized in future. 3. Module APIs currently have lack of memory policies support. This part will be fixed in future. Nikita Panov (20): mm: arm64 add Kconfig option for kernel replication arm64: align kernel text and rodata mm: allow per-NUMA node local P4D/PUD/PMD/PTE allocation arm64: add arch callbacks for kernel replication mm: per-NUMA node replication core infrastructure mm: add apply_to_page_range() for replicated memory mm: add support of memory protection for NUMA replicas arm64: add support of memory protection for NUMA replicas mm: set memory permissions for BPF handlers replicas arm64: bpf arch text poke support mm: add replicas allocation support for vmalloc arm64: enable per-NUMA node kernel text and rodata replication mm: enable per-NUMA node kernel text and rodata replication arm64: make power management aware about kernel replication arm64: make KASAN aware about kernel replication arm64: make kernel text patching aware about replicas arm64: add support of NUMA replication for efi page tables arm64: add support of NUMA replication for ptdump arm64: add kernel modules text and rodata replication support mm: init kernel modules with replication support arch/arm64/include/asm/efi.h | 18 +- arch/arm64/include/asm/mmu_context.h | 4 + arch/arm64/include/asm/numa_replication.h | 54 ++ arch/arm64/include/asm/pgtable.h | 13 + arch/arm64/kernel/alternative.c | 27 +- arch/arm64/kernel/cpufeature.c | 5 + arch/arm64/kernel/hibernate.c | 5 + arch/arm64/kernel/module.c | 36 +- arch/arm64/kernel/patching.c | 59 ++ arch/arm64/kernel/sleep.S | 8 + arch/arm64/kernel/smp.c | 8 + arch/arm64/kernel/suspend.c | 6 + arch/arm64/kernel/vmlinux.lds.S | 22 + arch/arm64/mm/context.c | 3 +- arch/arm64/mm/init.c | 49 ++ arch/arm64/mm/kasan_init.c | 68 +- arch/arm64/mm/mmu.c | 41 +- arch/arm64/mm/pageattr.c | 62 ++ arch/arm64/mm/pgd.c | 77 +++ arch/arm64/mm/ptdump.c | 17 +- arch/arm64/net/bpf_jit_comp.c | 4 +- drivers/firmware/efi/arm-runtime.c | 17 +- include/asm-generic/pgalloc.h | 78 +++ include/asm-generic/pgtable-nop4d.h | 5 + include/asm-generic/pgtable-nopmd.h | 5 + include/asm-generic/pgtable-nopud.h | 5 + include/asm-generic/set_memory.h | 12 + include/linux/gfp.h | 5 + include/linux/mm.h | 93 ++- include/linux/mm_types.h | 4 + include/linux/moduleloader.h | 6 + include/linux/numa_kernel_replication.h | 184 ++++++ include/linux/set_memory.h | 10 + include/linux/vmalloc.h | 19 + init/main.c | 17 + kernel/bpf/bpf_struct_ops.c | 8 +- kernel/bpf/core.c | 4 +- kernel/bpf/trampoline.c | 6 +- kernel/module/main.c | 32 + kernel/module/strict_rwx.c | 14 +- mm/Kconfig | 10 + mm/Makefile | 1 + mm/memory.c | 295 ++++++++- mm/mm_init.c | 1 + mm/numa_kernel_replication.c | 759 ++++++++++++++++++++++ mm/page_alloc.c | 20 + mm/vmalloc.c | 469 ++++++++++--- net/bpf/bpf_dummy_struct_ops.c | 2 +- 48 files changed, 2520 insertions(+), 147 deletions(-) create mode 100644 arch/arm64/include/asm/numa_replication.h create mode 100644 include/linux/numa_kernel_replication.h create mode 100644 mm/numa_kernel_replication.c -- 2.34.1