Support xcall prefetch. Changes in v9: - Add static key for xcall enable to reduce the overhead of checking xcall startup parameters, which also fix the system_supports_xcall() check stall bug when kernel start. - cleanup. Changes in v8: - Support xcall prefetch on and off runtime. - Refactor the xcall proc interface code, move to /fs/proc/proc_xcall.c. - Fix xint irqnr_xint_map array out of bounds access problem in __handle_domain_irq() by introducing is_hwirq_xint() helper. - Introduce system_supports_xint() and system_supports_xcall() to reduce the overhead, and remove fast_syscall_enabeld(), remove extern is_xint_support. - Limit the task xinfo alloc and xcall prefetch init by system_supports_xcall() to reduce the overhead. Changes in v7: - Remove "echo @xxx > /proc/$PID/xcall" usage, replace with "echo 1 > /proc/$PID/prefetch". Now the xcall enable and prefetch can be enabled separately. Changes in v6: - Only use one reserved kabi in task_struct for xcall. - Free prefetch item after eventpoll_release_file() to avoid possible competition between ep_prefetch_item_enqueue() and pfi free. - Fix "echo > /proc/xcall/cpu_list" show problem. Changes in v5: - Fix dentry free before free_prefetch_item() and then UAF in prefetch work by cancel_work_sync() before free pfi and move free_prefetch_item() to the first of __fput(). - Fix __queue_work() oops bug. - /proc/xcall/mask_list -> /proc/xcall/cpu_list - Fix xcall_prefetch print data type. - Update prefetch tracepoint, move xcall points definition to separete header file. - Make xcall_prefetch depends on EPOLL. - Adjust xcall prefetch buf to 16KB by default. - Add xcall_prefetch_init() function. - Only use write lock in free_prefetch_item(). - Simplify xcall_read() and add some comment. - Update commit message. Changes in v4: - Enable FAST_SYSCALL/IRQ and XCALL_PREFETCH by default. - Fix xint sleeping function called from invalid context bug. - free_prefetch_item() in file_free() instead of filp_close(), which fix the alloc-queue and free competition. - Fix kernel_read() warning for FMODE_READ not set. - Add sock_from_file() limit for prefetch. - Check NULL for rc_work. - Simplfy the cpumask interface code, and rename to "/proc/xcall/mask_list". - Remove the xcall_cache_pages_order interface. - tracepoint update: fd -> file. - Simplify the xcall_read() function again. - Handle copy_to_user() return value. - Remove unused XCALL_CACHE_QUEUED. - Update the commit message. Changes in v3: - Add XCALL_PREFETCH config to isolate feature code. - Split the /proc/xxx interface code out to independent patches, which will make it clear. - Update the cpumask interface to "/proc/xcall/numa_mask", and it can set the numa mask of all numas one time. - Add xcall select count to make xcall_cache_pages_order adjust safe. - Introduce xcall_read_start/end() to make it clear. - Simplify the xcall_read() function. - Use cpumask_next() instead of get_nth_cpu_in_cpumask() function. - Use independent cpu select policy function. - Remove some unnecessary pr_err(). - Update the commit message. Changes in v2: - Upadte the xcall prefetch state machine, remove the queued state and add prefetch, cancel states. - Remove the pfi lock and use atomic variables. - Change the 'xcall select' semantics and simplify the code a lot. - Remove keep_running, remove unrelated code. - Remove the count in struct read_cache_entry, and use percpu hit/miss count. - Remove sync mode, so remove struct read_cache_entry in struct task_struct. - Use hash table to find prefetch item for a file, which will not change the file struct KABI. - Use rwlock instead of spinlock for hash table. - Use alloc_page() instead kmalloc() to align 4KB. - Update the commit message. Jinjie Ruan (10): arm64: Introduce Xint software solution arm64: Add debugfs dir for xint arm64: Refactor the xcall proc code arm64: Reserve a kabi in task_struct exclusively for xcall arm64: Introduce xcall prefetch enable interface eventpoll: Support xcall async prefetch eventpoll: Add /proc/xcall/prefetch dir for performance tuning eventpoll: Add /proc/xcall/cpu_list for performance tuning xcall: Add tracepoints for different prefetching stages config: Enable FAST_SYSCALL/IRQ and XCALL_PREFETCH by default Liao Chen (1): arm64: revert kpti bypass Yipeng Zou (2): arm64: Introduce xcall a faster svc exception handling arm64: Faster SVC exception handler with xcall arch/Kconfig | 81 +++++ arch/arm64/Kconfig | 2 + arch/arm64/configs/openeuler_defconfig | 7 + arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 12 + arch/arm64/include/asm/exception.h | 3 + arch/arm64/include/asm/xcall.h | 21 ++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 + arch/arm64/kernel/cpufeature.c | 51 +++ arch/arm64/kernel/entry-common.c | 22 ++ arch/arm64/kernel/entry.S | 183 ++++++++++- arch/arm64/kernel/syscall.c | 57 ++++ arch/arm64/kernel/xcall.c | 38 +++ drivers/irqchip/irq-gic-v3.c | 145 +++++++++ fs/eventpoll.c | 412 +++++++++++++++++++++++++ fs/file_table.c | 1 + fs/proc/Makefile | 1 + fs/proc/base.c | 6 + fs/proc/internal.h | 7 + fs/proc/proc_xcall.c | 185 +++++++++++ fs/read_write.c | 10 +- include/linux/fs.h | 35 +++ include/linux/hardirq.h | 5 + include/linux/irqchip/arm-gic-v3.h | 12 + include/linux/sched.h | 5 + include/trace/events/xcall.h | 106 +++++++ kernel/fork.c | 19 ++ kernel/irq/debugfs.c | 36 +++ kernel/irq/internals.h | 16 + kernel/irq/irqdesc.c | 19 ++ kernel/irq/proc.c | 10 + kernel/softirq.c | 73 +++++ 33 files changed, 1583 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/xcall.h create mode 100644 arch/arm64/kernel/xcall.c create mode 100644 fs/proc/proc_xcall.c create mode 100644 include/trace/events/xcall.h -- 2.34.1