- Kernel - mailweb.openeuler.org

[PATCH OLK-6.6] timekeeping: Avoiding false sharing in field access of tk_core
by Yu Liao 21 Dec '23

21 Dec '23

From: Wang ShaoBo <bobo.shaobowang(a)huawei.com> hulk inclusion category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/I47W8L CVE: NA --------------------------- We detect a performance deterioration when using Unixbench, we use the dichotomy to locate the patch 7e66740ad725 ("MPAM / ACPI: Refactoring MPAM init process and set MPAM ACPI as entrance"), In comparing two commit df5defd901ff ("KVM: X86: MMU: Use the correct inherited permissions to get shadow page") and ac4dbb7554ef ("ACPI 6.x: Add definitions for MPAM table") we get following testing result: CMD: ./Run -c xx context1 RESULT: +-------------UnixBench context1-----------+ +---------+--------------+-----------------+ + + ac4dbb7554ef + df5defd901ff + +---------+--------------+---------+-------+ + Cores + Score + Score + +---------+--------------+-----------------+ + 1 + 522.8 + 535.7 + +---------+--------------+-----------------+ + 24 + 11231.5 + 12111.2 + +---------+--------------+-----------------+ + 48 + 8535.1 + 8745.1 + +---------+--------------+-----------------+ + 72 + 10821.9 + 10343.8 + +---------+--------------+-----------------+ + 96 + 15238.5 + 42947.8 + +---------+--------------+-----------------+ We found a irrefutable difference in latency sampling when using the perf tool: HEAD:ac4dbb7554ef HEAD:df5defd901ff 45.18% [kernel] [k] ktime_get_coarse_real_ts64 -> 1.78% [kernel] [k] ktime_get_coarse_real_ts64 ... 65.87 │ dmb ishld //smp_rmb() Through ftrace we get the calltrace and and detected the number of visits of ktime_get_coarse_real_ts64, which frequently visits tk_core->seq and tk_core->timekeeper->tkr_mono: - 48.86% [kernel] [k] ktime_get_coarse_real_ts64 - 5.76% ktime_get_coarse_real_ts64 #about 111437657 times per 10 seconds - 14.70% __audit_syscall_entry syscall_trace_enter el0_svc_common el0_svc_handler + el0_svc - 2.85% current_time So this may be performance degradation caused by interference when happened different fields access, We compare .bss and .data section of this two version: HEAD:ac4dbb7554ef `-> ffff00000962e680 l O .bss 0000000000000110 tk_core ffff000009355680 l O .data 0000000000000078 tk_fast_mono ffff0000093557a0 l O .data 0000000000000090 dummy_clock ffff000009355700 l O .data 0000000000000078 tk_fast_raw ffff000009355778 l O .data 0000000000000028 timekeeping_syscore_ops ffff00000962e640 l O .bss 0000000000000008 cycles_at_suspend HEAD:df5defd901ff `-> ffff00000957dbc0 l O .bss 0000000000000110 tk_core ffff0000092b4e80 l O .data 0000000000000078 tk_fast_mono ffff0000092b4fa0 l O .data 0000000000000090 dummy_clock ffff0000092b4f00 l O .data 0000000000000078 tk_fast_raw ffff0000092b4f78 l O .data 0000000000000028 timekeeping_syscore_ops ffff00000957db80 l O .bss 0000000000000008 cycles_at_suspend By comparing this two version tk_core's address: ffff00000962e680 is 128Byte aligned but latter df5defd901ff is 64Byte aligned, the memory storage layout of tk_core has undergone subtle changes: HEAD:ac4dbb7554ef `-> |<--------formmer 64Bytes---------->|<------------latter 64Byte------------->| 0xffff00000957dbc0_>|<-seq 8Bytes->|<-tkr_mono 56Bytes->|<-thr_raw 56Bytes->|<-xtime_sec 8Bytes->| 0xffff00000957dc00_>... HEAD:df5defd901ff `-> |<------formmer 64Bytes---->|<------------latter 64Byte-------->| 0xffff00000962e680_>|<-Other variables 64Bytes->|<-seq 8Bytes->|<-tkr_mono 56Bytes->| 0xffff00000962e6c0_>.. We testified thr_raw,xtime_sec fields interfere strongly with seq,tkr_mono field because of frequent load/store operation, this will cause as known false sharing. We add a 64Bytes padding field in tk_core for reservation of any after usefull usage and keep tk_core 128Byte aligned, this can avoid changes in the way tk_core's layout is stored, In this solution, layout of tk_core always like this: crash> struct -o tk_core_t struct tk_core_t { [0] u64 padding[8]; [64] seqcount_t seq; [72] struct timekeeper timekeeper; } SIZE: 336 crash> struct -o timekeeper struct timekeeper { [0] struct tk_read_base tkr_mono; [56] struct tk_read_base tkr_raw; [112] u64 xtime_sec; [120] unsigned long ktime_sec; ... } SIZE: 264 After appling our own solution: +---------+--------------+ + + Our solution + +---------+--------------+ + Cores + Score + +---------+--------------+ + 1 + 548.9 + +---------+--------------+ + 24 + 11018.3 + +---------+--------------+ + 48 + 8938.2 + +---------+--------------+ + 72 + 14610.7 + +---------+--------------+ + 96 + 40811.7 + +---------+--------------+ Signed-off-by: Wang ShaoBo <bobo.shaobowang(a)huawei.com> Signed-off-by: Yu Liao <liaoyu15(a)huawei.com> --- arch/arm64/Kconfig | 9 +++++++++ arch/arm64/include/asm/cache.h | 6 ++++++ kernel/time/timekeeping.c | 7 +++++++ 3 files changed, 22 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 2aca373a7038..f5559e38b243 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1493,6 +1493,15 @@ config HW_PERF_EVENTS def_bool y depends on ARM_PMU +config ARCH_LLC_128_LINE_SIZE + bool "Force 128 bytes alignment for fitting LLC cacheline" + depends on ARM64 + default y + help + As specific machine's LLC cacheline size may be up to + 128 bytes, gaining performance improvement from fitting + 128 Bytes LLC cache aligned. + # Supported by clang >= 7.0 or GCC >= 12.0.0 config CC_HAVE_SHADOW_CALL_STACK def_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index 1613779be63a..8455df351ef8 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -8,6 +8,12 @@ #define L1_CACHE_SHIFT (6) #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) +#ifdef CONFIG_ARCH_LLC_128_LINE_SIZE +#ifndef ____cacheline_aligned_128 +#define ____cacheline_aligned_128 __attribute__((__aligned__(128))) +#endif +#endif + #define CLIDR_LOUU_SHIFT 27 #define CLIDR_LOC_SHIFT 24 #define CLIDR_LOUIS_SHIFT 21 diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 266d02809dbb..35b366addf07 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -48,9 +48,16 @@ DEFINE_RAW_SPINLOCK(timekeeper_lock); * cache line. */ static struct { +#ifdef CONFIG_ARCH_LLC_128_LINE_SIZE + u64 padding[8]; +#endif seqcount_raw_spinlock_t seq; struct timekeeper timekeeper; +#ifdef CONFIG_ARCH_LLC_128_LINE_SIZE +} tk_core ____cacheline_aligned_128 = { +#else } tk_core ____cacheline_aligned = { +#endif .seq = SEQCNT_RAW_SPINLOCK_ZERO(tk_core.seq, &timekeeper_lock), }; -- 2.33.0

2 1

[PATCH OLK-6.6 0/2] support CLOCKSOURCE_VALIDATE_LAST_CYCLE on
by Yu Liao 21 Dec '23

21 Dec '23

Yu Liao (2): timekeeping: Make CLOCKSOURCE_VALIDATE_LAST_CYCLE configurable config: make CLOCKSOURCE_VALIDATE_LAST_CYCLE not set by default arch/arm64/configs/openeuler_defconfig | 1 + kernel/time/Kconfig | 13 ++++++++----- 2 files changed, 9 insertions(+), 5 deletions(-) -- 2.33.0

2 3

[PATCH OLK-6.6 v4 0/1] stop_machine: mask pseudo nmi before running the callback
by Yuntao Liu 21 Dec '23

21 Dec '23

*** BLURB HERE *** Wei Li (1): stop_machine: mask pseudo nmi before running the callback arch/arm64/include/asm/arch_gicv3.h | 12 ++++++++++++ kernel/stop_machine.c | 3 +++ 2 files changed, 15 insertions(+) -- 2.34.1

2 2

[PATCH OLK-6.6 0/3] Introduce CPU inspect feature
by Yu Liao 21 Dec '23

21 Dec '23

This patches series introduce CPU-inspect feature. CPU-inspect is designed to provide a framework for early detection of SDC by proactively executing CPU inspection test cases. Silent Data Corruption (SDC), sometimes referred to as Silent Data Error (SDE), is an industry-wide issue impacting not only long-protected memory, storage, and networking, but also computer CPUs. As with software issues, hardware-induced SDC can contribute to data loss and corruption. An SDC occurs when an impacted CPU inadvertently causes errors in the data it processes. For example, an impacted CPU might miscalculate data (i.e., 1+1=3). There may be no indication of these computational errors unless the software systematically checks for errors [1]. SDC issues have been around for many years, but as chips have become more advanced and compact in size, the transistors and lines have become so tiny that small electrical fluctuations can cause errors. Most of these errors are caused by defects during manufacturing and are screened out by the vendors; others are caught by hardware error detection or correction. However, some errors go undetected by hardware; therefore only detection software can protect against such errors [1]. [1] https://support.google.com/cloud/answer/10759085 Yu Liao (3): cpuinspect: add CPU-inspect infrastructure cpuinspect: add ATF inspector openeuler_defconfig: enable CPU inspect for arm64 by default arch/arm64/configs/openeuler_defconfig | 7 + drivers/Kconfig | 2 + drivers/Makefile | 1 + drivers/cpuinspect/Kconfig | 24 +++ drivers/cpuinspect/Makefile | 7 + drivers/cpuinspect/cpuinspect.c | 170 ++++++++++++++++ drivers/cpuinspect/cpuinspect.h | 46 +++++ drivers/cpuinspect/inspector-atf.c | 81 ++++++++ drivers/cpuinspect/inspector.c | 124 ++++++++++++ drivers/cpuinspect/sysfs.c | 258 +++++++++++++++++++++++++ include/linux/cpuinspect.h | 40 ++++ 11 files changed, 760 insertions(+) create mode 100644 drivers/cpuinspect/Kconfig create mode 100644 drivers/cpuinspect/Makefile create mode 100644 drivers/cpuinspect/cpuinspect.c create mode 100644 drivers/cpuinspect/cpuinspect.h create mode 100644 drivers/cpuinspect/inspector-atf.c create mode 100644 drivers/cpuinspect/inspector.c create mode 100644 drivers/cpuinspect/sysfs.c create mode 100644 include/linux/cpuinspect.h -- 2.33.0

2 4

[PATCH OLK-6.6 0/2] Introduce clear freelist feature
by Yu Liao 21 Dec '23

21 Dec '23

Yu Liao (2): mm: Add sysctl to clear free list pages config: make CONFIG_CLEAR_FREELIST_PAGE not set by default .../admin-guide/kernel-parameters.txt | 3 + Documentation/admin-guide/sysctl/vm.rst | 13 ++ arch/arm64/configs/openeuler_defconfig | 1 + mm/Kconfig | 12 ++ mm/Makefile | 2 + mm/clear_freelist_page.c | 178 ++++++++++++++++++ 6 files changed, 209 insertions(+) create mode 100644 mm/clear_freelist_page.c -- 2.33.0

2 3

[PATCH OLK-6.6 v3 0/1] stop_machine: mask pseudo nmi before running the callback
by Yuntao Liu 21 Dec '23

21 Dec '23

From: Jinjie Ruan <ruanjinjie(a)huawei.com> *** BLURB HERE *** Wei Li (1): stop_machine: mask pseudo nmi before running the callback arch/arm64/include/asm/arch_gicv3.h | 12 ++++++++++++ kernel/stop_machine.c | 3 +++ 2 files changed, 15 insertions(+) -- 2.34.1

2 2

[PATCH OLK-6.6 0/2] Introduce clear freelist feature
by Yu Liao 21 Dec '23

21 Dec '23

Yu Liao (2): mm: Add sysctl to clear free list pages config: make CONFIG_CLEAR_FREELIST_PAGE not set by default .../admin-guide/kernel-parameters.txt | 3 + Documentation/admin-guide/sysctl/vm.rst | 13 ++ arch/arm64/configs/openeuler_defconfig | 1 + mm/Kconfig | 13 ++ mm/Makefile | 2 + mm/clear_freelist_page.c | 178 ++++++++++++++++++ 6 files changed, 210 insertions(+) create mode 100644 mm/clear_freelist_page.c -- 2.33.0

2 3

[PATCH OLK-6.6 v2 0/1] stop_machine: mask pseudo nmi before running the callback
by Yuntao Liu 21 Dec '23

21 Dec '23

*** BLURB HERE *** Wei Li (1): stop_machine: mask pseudo nmi before running the callback arch/arm64/include/asm/arch_gicv3.h | 12 ++++++++++++ kernel/stop_machine.c | 3 +++ 2 files changed, 15 insertions(+) -- 2.34.1

2 2

[PATCH OLK-6.6 0/3] cgroup v1 writeback
by chenridong 21 Dec '23

21 Dec '23

Lu Jialin (2): cgroup: Factor out __cgroup_get_from_id() for cgroup v1 openeuler_defconfig: enable CONFIG_CGROUP_V1_WRITEBACK in openeuler_defconfig for x86 and arm64 chenridong (1): cgroup: support cgroup writeback on cgroupv1 arch/arm64/configs/openeuler_defconfig | 1 + arch/x86/configs/openeuler_defconfig | 1 + block/blk-cgroup.c | 3 + block/blk-cgroup.h | 3 + include/linux/backing-dev.h | 29 ++++++- include/linux/cgroup-defs.h | 2 + include/linux/cgroup.h | 1 + include/linux/memcontrol.h | 5 ++ init/Kconfig | 5 ++ kernel/cgroup/cgroup.c | 31 +++++-- mm/backing-dev.c | 116 ++++++++++++++++++++++++- mm/memcontrol.c | 83 +++++++++++++++++- 12 files changed, 267 insertions(+), 13 deletions(-) -- 2.34.1

2 4

[PATCH OLK-6.6 0/3] Revert "compiler: remove CONFIG_OPTIMIZE_INLINING entirely"
by Yuntao Liu 21 Dec '23

21 Dec '23

*** BLURB HERE *** Guo Xuenan (3): Revert "compiler: remove CONFIG_OPTIMIZE_INLINING entirely" make OPTIMIZE_INLINING config editable disable OPTIMIZE_INLINING by default arch/arm64/kvm/sys_regs.h | 5 +++++ arch/x86/configs/i386_defconfig | 1 + arch/x86/configs/x86_64_defconfig | 1 + drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h | 5 +++++ .../pci/hive_isp_css_include/print_support.h | 4 ++++ include/linux/compiler_types.h | 8 ++++++++ include/trace/trace_events.h | 2 +- kernel/configs/tiny.config | 1 + lib/Kconfig.debug | 13 +++++++++++++ 9 files changed, 39 insertions(+), 1 deletion(-) -- 2.34.1

2 4