[PATCH v6 OLK-6.6 0/4] arm64: Allow CPU0 to be nohz_full
fix kabi Xiongfeng Wang (4): isolation: Do not check whether housekeeping CPUs are present isolation: Check whether there exists a housekeeping CPU online arm64: Kconfig: select ARCH_SUSPEND_NONZERO_CPU by default arm64: defconfig: Enable CONFIG_PM_SLEEP_SMP_NONZERO_CPU and CONFIG_ARCH_SUSPEND_NONZERO_CPU arch/arm64/Kconfig | 3 ++ arch/arm64/configs/openeuler_defconfig | 2 + arch/arm64/kernel/process.c | 27 ++++++++++++- include/linux/cpu.h | 5 ++- include/linux/sched/isolation.h | 2 + include/linux/tick.h | 4 ++ kernel/cpu.c | 14 +++++++ kernel/sched/isolation.c | 56 ++++++++++++++++++++++---- kernel/smp.c | 4 ++ kernel/time/tick-common.c | 9 +++++ kernel/time/tick-sched.c | 11 ++++- 11 files changed, 126 insertions(+), 11 deletions(-) -- 2.34.1
From: Xiongfeng Wang <wangxiongfeng2@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9856 -------------------------------- If we set isolcpus including CPU0, it will print the following error: [ 0.000000] Housekeeping: must include one present CPU, using boot CPU:0 It is because when string 'isolcpus=xxx' is parsed, only CPU0 is set as present for arch arm64. Unlike X86 and PowerPC, the present_cpu_mask is set rather late in smp_prepare_cpus() for arm64. In CPU hotplug situation, some possible CPUs are not marked as present, only CPUs with its GICC marked as ACPI_MADT_ENABLED are marked as present. So we cannot set preset_cpu_mask for all possible CPUs at early stage. Only check cpu_present_mask cannot assure that the CPUs will be broughtup online, as described in the below Link1. So do not check whether housekeeping CPUs are present when parsing 'isolcpus'. A following commit will add check after CPU online process is finished, as suggested in below Link2. With the boot CPU allowed to be nohz_full, housekeeping_any_cpu() can return an invalid CPU number at boot time before start_secondary() brings the 1st housekeeping CPU up, which crashes the kernel when the value is used as a per-CPU index by get_nohz_timer_target()'s callers (timer.c/hrtimer.c get_target_base()). This is the code part of the upstream fix 5097cbcb38e6 ("sched/isolation: Prevent boot crash when the boot CPU is nohz_full"), which was stripped from the LTS backport already merged into this tree (only the documentation part was taken). Include it here since this series makes 'nohz_full=' masks including the boot CPU reachable on arm64. Link: https://lore.kernel.org/all/20190504002733.GB19076@lenoir/ Link: https://lore.kernel.org/all/1557186148.ocs72ssdjc.astroid@bobo.none/ Signed-off-by: Xiongfeng Wang <wangxiongfeng2@huawei.com> Signed-off-by: Xia Fukun <xiafukun@huawei.com> --- include/linux/tick.h | 2 ++ kernel/sched/isolation.c | 37 +++++++++++++++++++++++++++++-------- kernel/time/tick-sched.c | 9 +++++++++ 3 files changed, 40 insertions(+), 8 deletions(-) diff --git a/include/linux/tick.h b/include/linux/tick.h index 39f01f0bef76..5c929c429007 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -42,8 +42,10 @@ extern void tick_irq_enter(void); # ifndef arch_needs_cpu # define arch_needs_cpu() (0) # endif +extern bool support_cpu0_nohz_full; # else static inline void tick_irq_enter(void) { } +#define support_cpu0_nohz_full 0 #endif #if defined(CONFIG_GENERIC_CLOCKEVENTS_BROADCAST) && defined(CONFIG_TICK_ONESHOT) diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 1179dc21873e..e2d6a88611dc 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -46,7 +46,16 @@ int housekeeping_any_cpu(enum hk_type type) if (cpu < nr_cpu_ids) return cpu; - return cpumask_any_and(housekeeping.cpumasks[type], cpu_online_mask); + cpu = cpumask_any_and(housekeeping.cpumasks[type], cpu_online_mask); + if (likely(cpu < nr_cpu_ids)) + return cpu; + /* + * Unless we have another problem this can only happen + * at boot time before start_secondary() brings the 1st + * housekeeping CPU up. + */ + WARN_ON_ONCE(system_state == SYSTEM_RUNNING || + type != HK_TYPE_TIMER); } } return smp_processor_id(); @@ -130,13 +139,25 @@ static int __init housekeeping_setup(char *str, unsigned long flags) cpumask_andnot(housekeeping_staging, cpu_possible_mask, non_housekeeping_mask); - first_cpu = cpumask_first_and(cpu_present_mask, housekeeping_staging); - if (first_cpu >= nr_cpu_ids || first_cpu >= setup_max_cpus) { - __cpumask_set_cpu(smp_processor_id(), housekeeping_staging); - __cpumask_clear_cpu(smp_processor_id(), non_housekeeping_mask); - if (!housekeeping.flags) { - pr_warn("Housekeeping: must include one present CPU, " - "using boot CPU:%d\n", smp_processor_id()); + if (support_cpu0_nohz_full) { + if (cpumask_empty(housekeeping_staging)) { + pr_warn("Housekeeping cpumask is NULL, using boot CPU\n"); + __cpumask_set_cpu(smp_processor_id(), housekeeping_staging); + /* Update non_housekeeping_mask to reflect this change */ + cpumask_andnot(non_housekeeping_mask, + cpu_possible_mask, housekeeping_staging); + } + /* No need to check against cpu_present_mask here */ + first_cpu = smp_processor_id(); + } else { + first_cpu = cpumask_first_and(cpu_present_mask, housekeeping_staging); + if (first_cpu >= nr_cpu_ids || first_cpu >= setup_max_cpus) { + __cpumask_set_cpu(smp_processor_id(), housekeeping_staging); + __cpumask_clear_cpu(smp_processor_id(), non_housekeeping_mask); + if (!housekeeping.flags) { + pr_warn("Housekeeping: must include one present CPU, " + "using boot CPU:%d\n", smp_processor_id()); + } } } diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 55cbc49f70d1..33a471692def 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -38,6 +38,8 @@ */ static DEFINE_PER_CPU(struct tick_sched, tick_cpu_sched); +bool support_cpu0_nohz_full; + struct tick_sched *tick_get_tick_sched(int cpu) { return &per_cpu(tick_cpu_sched, cpu); @@ -1615,3 +1617,10 @@ int tick_check_oneshot_change(int allow_nohz) tick_nohz_switch_to_nohz(); return 0; } + +static int __init support_cpu0_nohz_full_setup(char *str) +{ + support_cpu0_nohz_full = true; + return 0; +} +early_param("support_cpu0_nohz_full", support_cpu0_nohz_full_setup); -- 2.34.1
From: Xiongfeng Wang <wangxiongfeng2@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9856 -------------------------------- We need to make sure that the housekeeper, one of the housekeeping CPUs, is online, as describe in the below Link. We add this check after secondary CPUs online is finished. Link: https://lore.kernel.org/all/20190504002733.GB19076@lenoir/ Signed-off-by: Xiongfeng Wang <wangxiongfeng2@huawei.com> Signed-off-by: Xia Fukun <xiafukun@huawei.com> --- include/linux/sched/isolation.h | 2 ++ kernel/sched/isolation.c | 19 +++++++++++++++++++ kernel/smp.c | 4 ++++ 3 files changed, 25 insertions(+) diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index 3894e74e8dc5..7addf79326df 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -27,6 +27,7 @@ extern bool housekeeping_enabled(enum hk_type type); extern void housekeeping_affine(struct task_struct *t, enum hk_type type); extern bool housekeeping_test_cpu(int cpu, enum hk_type type); extern void __init housekeeping_init(void); +extern void check_housekeeping_cpu_online(void); #else @@ -56,6 +57,7 @@ static inline bool housekeeping_test_cpu(int cpu, enum hk_type type) } static inline void housekeeping_init(void) { } +static inline void check_housekeeping_cpu_online(void) { } #endif /* CONFIG_CPU_ISOLATION */ static inline bool housekeeping_cpu(int cpu, enum hk_type type) diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index e2d6a88611dc..52f25e78ef45 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -273,3 +273,22 @@ static int __init enhanced_isolcpus_setup(char *str) return 0; } __setup("enhanced_isolcpus", enhanced_isolcpus_setup); + +void check_housekeeping_cpu_online(void) +{ + enum hk_type type; + + if (!support_cpu0_nohz_full) + return; + if (!housekeeping.flags) + return; + for_each_set_bit(type, &housekeeping.flags, HK_TYPE_MAX) { + if (!cpumask_intersects(housekeeping.cpumasks[type], + cpu_online_mask)) { + pr_err("All housekeeping CPUs for type %d are offline!\n", type); + pr_err("please modify the kernel parameter !\n"); + /* BUG() here, otherwise we may BUG() in sched_init_smp(). */ + BUG(); + } + } +} diff --git a/kernel/smp.c b/kernel/smp.c index 13b376c7296a..551702122fcb 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -20,6 +20,7 @@ #include <linux/cpu.h> #include <linux/sched.h> #include <linux/sched/idle.h> +#include <linux/sched/isolation.h> #include <linux/hypervisor.h> #include <linux/sched/clock.h> #include <linux/nmi.h> @@ -986,6 +987,9 @@ void __init smp_init(void) /* Any cleanup work */ smp_cpus_done(setup_max_cpus); + + /* Check whether there exists a housekeeping CPU online */ + check_housekeeping_cpu_online(); } /* -- 2.34.1
From: Xiongfeng Wang <wangxiongfeng2@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9856 -------------------------------- Commit 08ae95f4fd3b ("nohz_full: Allow the boot CPU to be nohz_full") allows the boot CPU to be a nohz_full CPU, handing over its housekeeping duty. But it is only supported when CONFIG_PM_SLEEP_SMP is not configured, or when it is configured and the arch allows suspend on non-zero CPUs. We cannot disable CONFIG_PM_SLEEP_SMP because freeze_secondary_cpus(), which is used in kexec, is included in the config. For arm64, there is no restriction in kernel to prevent nonzero cpu entering suspend using PSCI interface. when ARCH_SUSPEND_NONZERO_CPU is enabled, PM_SLEEP_SMP_NONZERO_CPU will be also enabled. The only influence is that suspend_disable_secondary_cpus() will use a housekeeping CPU to enter suspend. If pm sleep/hibernate is not involved, only kexec will use this function, which is tested on an arm64 server, and the vmcore can be created successfully. We add 'support_cpu0_nohz_full' in the CONFIG_PM_SLEEP_SMP_NONZERO_CPU check branch. When 'support_cpu0_nohz_full' is zero. There is no functional change. Besides, the normal kexec path (kexec -e, not the crash path) calls machine_shutdown() -> smp_shutdown_nonboot_cpus(reboot_cpu), with reboot_cpu being 0. After this patch makes 'nohz_full=' masks including CPU0 possible, tick_do_timer_cpu is handed over to a non-zero housekeeping CPU, and the offline of that CPU is rejected by tick_nohz_cpu_down() with -EBUSY. smp_shutdown_nonboot_cpus() then hits its unconditional BUG_ON(num_online_cpus() > 1) and kexec fails. The vmcore test above does not cover this path (the crash path goes through machine_crash_shutdown() instead). Fix it by extending smp_shutdown_nonboot_cpus() to accept primary_cpu == -1, which selects the CPU owning the do_timer duty (exposed by the new tick_get_do_timer_cpu() helper) as the reboot CPU, falling back to the first online CPU when the duty holder is a sentinel value or offline, and make arm64's machine_shutdown() pass -1 when 'support_cpu0_nohz_full' is given and CPU0 is in the nohz_full mask. Anchoring the reboot CPU to the do_timer owner matters on multi-NUMA systems: the alternative of picking a housekeeping TIMER CPU via housekeeping_any_cpu() would return the housekeeping CPU closest to the current node, while the reboot task itself may have been migrated by set_cpus_allowed_ptr() to a ~random CPU of its allowed mask. The selected housekeeping CPU then may differ from the do_timer owner, whose offline is still rejected by tick_nohz_cpu_down(), and the BUG_ON(num_online_cpus() > 1) is hit again. Since the reboot task was pinned to reboot_cpu (CPU0) by migrate_to_reboot_cpu() and CPU0 is now offlined by the shutdown loop, also move it to the do_timer owner (or, on a sentinel value, to the housekeeping TIMER cpumask) before calling smp_shutdown_nonboot_cpus(), so the task which executes machine_kexec() ends up on the surviving reboot CPU, as it did before. Without the boot parameter, reboot_cpu keeps being used and the behavior is unchanged. Signed-off-by: Xiongfeng Wang <wangxiongfeng2@huawei.com> Signed-off-by: Xia Fukun <xiafukun@huawei.com> --- arch/arm64/Kconfig | 3 +++ arch/arm64/kernel/process.c | 27 ++++++++++++++++++++++++++- include/linux/cpu.h | 5 ++++- include/linux/tick.h | 2 ++ kernel/cpu.c | 14 ++++++++++++++ kernel/time/tick-common.c | 9 +++++++++ kernel/time/tick-sched.c | 2 +- 7 files changed, 59 insertions(+), 3 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 182a6a06b299..2f6f5931f718 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2870,6 +2870,9 @@ config ARCH_HIBERNATION_HEADER config ARCH_SUSPEND_POSSIBLE def_bool y +config ARCH_SUSPEND_NONZERO_CPU + def_bool y + config ARCH_CPUIDLE_HALTPOLL bool "Enable selection of the cpuidle-haltpoll driver" help diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index e8e0e11b1c4d..0807a181415e 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -14,6 +14,9 @@ #include <linux/sched/debug.h> #include <linux/sched/task.h> #include <linux/sched/task_stack.h> +#ifndef __GENKSYMS__ +#include <linux/sched/isolation.h> +#endif #include <linux/kernel.h> #include <linux/mman.h> #include <linux/mm.h> @@ -87,7 +90,29 @@ void __noreturn arch_cpu_idle_dead(void) */ void machine_shutdown(void) { - smp_shutdown_nonboot_cpus(reboot_cpu); + unsigned int primary_cpu = reboot_cpu; + + if (support_cpu0_nohz_full && tick_nohz_full_cpu(0)) { + unsigned int tick_cpu = tick_get_do_timer_cpu(); + + primary_cpu = -1; + /* + * The reboot task was pinned to reboot_cpu (CPU0) by + * migrate_to_reboot_cpu(), but CPU0 will be offlined by + * smp_shutdown_nonboot_cpus() below. Move it to the CPU + * which owns the do_timer duty (that CPU survives the + * offline loop), so the task which executes + * machine_kexec() stays on the reboot CPU, as it did + * before. + */ + if (tick_cpu >= 0 && tick_cpu < nr_cpu_ids) + set_cpus_allowed_ptr(current, cpumask_of(tick_cpu)); + else + set_cpus_allowed_ptr(current, + housekeeping_cpumask(HK_TYPE_TIMER)); + } + + smp_shutdown_nonboot_cpus(primary_cpu); } /* diff --git a/include/linux/cpu.h b/include/linux/cpu.h index 98e06cf4b49d..e7ec449d9134 100644 --- a/include/linux/cpu.h +++ b/include/linux/cpu.h @@ -20,6 +20,9 @@ #include <linux/cpuhotplug.h> #include <linux/cpuhplock.h> #include <linux/cpu_smt.h> +#ifndef __GENKSYMS__ +#include <linux/tick.h> +#endif struct device; struct device_node; @@ -143,7 +146,7 @@ static inline int suspend_disable_secondary_cpus(void) { int cpu = 0; - if (IS_ENABLED(CONFIG_PM_SLEEP_SMP_NONZERO_CPU)) + if (support_cpu0_nohz_full && IS_ENABLED(CONFIG_PM_SLEEP_SMP_NONZERO_CPU)) cpu = -1; return freeze_secondary_cpus(cpu); diff --git a/include/linux/tick.h b/include/linux/tick.h index 5c929c429007..2d2ee1956b84 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -21,12 +21,14 @@ extern void tick_suspend_local(void); extern void tick_resume_local(void); extern void tick_handover_do_timer(void); extern void tick_cleanup_dead_cpu(int cpu); +extern int tick_get_do_timer_cpu(void); #else /* CONFIG_GENERIC_CLOCKEVENTS */ static inline void tick_init(void) { } static inline void tick_suspend_local(void) { } static inline void tick_resume_local(void) { } static inline void tick_handover_do_timer(void) { } static inline void tick_cleanup_dead_cpu(int cpu) { } +static inline int tick_get_do_timer_cpu(void) { return -1; } #endif /* !CONFIG_GENERIC_CLOCKEVENTS */ #if defined(CONFIG_GENERIC_CLOCKEVENTS) && defined(CONFIG_SUSPEND) diff --git a/kernel/cpu.c b/kernel/cpu.c index 8a0955256578..4795b0929262 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1530,6 +1530,20 @@ void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) cpu_maps_update_begin(); + if (primary_cpu == -1) { + /* + * Prefer the CPU which owns the do_timer duty: the offline + * of that CPU is rejected by tick_nohz_cpu_down() when + * nohz_full is running, so picking it as the reboot CPU + * keeps the offline loop below away from that rejection. + */ + primary_cpu = tick_get_do_timer_cpu(); + if (primary_cpu < 0 || !cpu_online(primary_cpu)) + primary_cpu = cpumask_first(cpu_online_mask); + if (!housekeeping_cpu(primary_cpu, HK_TYPE_TIMER)) + primary_cpu = housekeeping_any_cpu(HK_TYPE_TIMER); + } + /* * Make certain the cpu I'm about to reboot on is online. * diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c index ecdb8c2b2cab..6d47cc03a54d 100644 --- a/kernel/time/tick-common.c +++ b/kernel/time/tick-common.c @@ -396,6 +396,15 @@ void tick_handover_do_timer(void) tick_do_timer_cpu = cpumask_first(cpu_online_mask); } +/* + * Return the CPU which currently owns the do_timer duty. The value + * is either an online CPU or one of the TICK_DO_TIMER_* sentinels. + */ +int tick_get_do_timer_cpu(void) +{ + return READ_ONCE(tick_do_timer_cpu); +} + /* * Shutdown an event device on a given cpu: * diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 33a471692def..549160ecfd88 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -582,7 +582,7 @@ void __init tick_nohz_init(void) } if (IS_ENABLED(CONFIG_PM_SLEEP_SMP) && - !IS_ENABLED(CONFIG_PM_SLEEP_SMP_NONZERO_CPU)) { + (!support_cpu0_nohz_full || !IS_ENABLED(CONFIG_PM_SLEEP_SMP_NONZERO_CPU))) { cpu = smp_processor_id(); if (cpumask_test_cpu(cpu, tick_nohz_full_mask)) { -- 2.34.1
From: Xiongfeng Wang <wangxiongfeng2@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9856 -------------------------------- Enable CONFIG_PM_SLEEP_SMP_NONZERO_CPU and CONFIG_ARCH_SUSPEND_NONZERO_CPU. Signed-off-by: Xiongfeng Wang <wangxiongfeng2@huawei.com> Signed-off-by: Xia Fukun <xiafukun@huawei.com> --- arch/arm64/configs/openeuler_defconfig | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index 007158aed18e..19e44f0b84cc 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -661,6 +661,7 @@ CONFIG_HIBERNATION_SNAPSHOT_DEV=y CONFIG_PM_STD_PARTITION="" CONFIG_PM_SLEEP=y CONFIG_PM_SLEEP_SMP=y +CONFIG_PM_SLEEP_SMP_NONZERO_CPU=y # CONFIG_PM_AUTOSLEEP is not set # CONFIG_PM_USERSPACE_AUTOSLEEP is not set # CONFIG_PM_WAKELOCKS is not set @@ -680,6 +681,7 @@ CONFIG_CPU_PM=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y CONFIG_ARCH_HIBERNATION_HEADER=y CONFIG_ARCH_SUSPEND_POSSIBLE=y +CONFIG_ARCH_SUSPEND_NONZERO_CPU=y CONFIG_ARCH_CPUIDLE_HALTPOLL=y # end of Power management options -- 2.34.1
反馈： 您发送到kernel@openeuler.org的补丁/补丁集，已成功转换为PR！ PR链接地址： https://atomgit.com/openeuler/kernel/merge_requests/30264 邮件列表地址：https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/Z6E... FeedBack: The patch(es) which you have sent to kernel@openeuler.org mailing list has been converted to a pull request successfully! Pull request link: https://atomgit.com/openeuler/kernel/merge_requests/30264 Mailing list address: https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/Z6E...
participants (2)
-
patchwork bot -
Xia Fukun