- Kernel - mailweb.openeuler.org

[PATCH openEuler-26.09 0/3] Fix device memory leaks in XSched
by Liu Kai 11 Mar '26

11 Mar '26

Fix device memory leaks in XSched Liu Kai (3): xsched/dmem: ensure XSched Context creation before device memory allocation xsched/dmem: add size field to xsched_dmem_pool for dmem region tracking xsched/dmem: ensure dmem region release on AI task completion to prevent memory leaks include/linux/xsched.h | 2 ++ include/uapi/linux/xcu_vstream.h | 2 +- kernel/xsched/dmem.c | 38 +++++++++++++++++++++++++++----- kernel/xsched/vstream.c | 22 +++++++++++++++--- 4 files changed, 55 insertions(+), 9 deletions(-) -- 2.34.1

2 4

[PATCH OLK-6.6 0/2] optimize lightweight load balancing
by Chen Jinghuang 11 Mar '26

11 Mar '26

*** BLURB HERE *** Chen Jinghuang (1): sched: Fix kabi breakage of struct sched_domain for newidle_stats Peter Zijlstra (Intel) (1): sched/fair: Proportional newidle balance include/linux/sched/topology.h | 4 +++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 48 +++++++++++++++++++++++++++++++--- kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 7 +++++ kernel/sched/topology.c | 6 +++++ 6 files changed, 69 insertions(+), 4 deletions(-) -- 2.34.1

2 3

[PATCH OLK-6.6 0/2] Fix TLBI broadcast optimization loss and incorrect dvmbm handling
by Tian Zheng 11 Mar '26

11 Mar '26

Fix 1 In the TLBI broadcast-optimization feature, the SYS_LSUDVM_CTRL_EL2 register controls whether the feature is enabled. When LPI low-power mode is enabled, this register is cleared as the pCPU powers down into LPI mode, causing the TLBI broadcast optimization to stop working. The fix saves and restores this control register in the callbacks for entering and exiting LPI mode. Fix 2 The current TLBI broadcast-optimization logic only supports normal VMs. If the feature is enabled globally, VMs that are not yet adapted for it, such as CCA will use an incorrect TLBI broadcast bitmap, which can cause a panic in CCA VMs. The fix moves the actual enable/disable of the dvmbm functionality into the vcpu load and vcpu put functions. Tian Zheng (2): KVM: arm64: Fix TLBI optimization broken in LPI mode KVM: arm64: Fix CCA guest panic when dvmbm is enabled arch/arm64/kvm/arm.c | 6 ++++++ arch/arm64/kvm/hisilicon/hisi_virt.c | 9 ++++----- arch/arm64/kvm/hisilicon/hisi_virt.h | 19 +++++++++++++++++++ 3 files changed, 29 insertions(+), 5 deletions(-) -- 2.33.0

1 2

[PATCH OLK-6.6 v4 0/4] Fix CVE-2025-21696.
by Ze Zuo 10 Mar '26

10 Mar '26

Fix CVE-2025-21696. Change since v3: -- internalize struct to reduce kABI exposure. David Hildenbrand (1): mm/mremap: fix WARN with uffd that has remap events disabled Lokesh Gidra (1): userfaultfd: move userfaultfd_ctx struct to header file Ryan Roberts (1): mm: clear uffd-wp PTE/PMD state on mremap() Ze Zuo (1): userfaultfd: fix kabi breakage due to struct userfaultfd_ctx fs/userfaultfd.c | 40 +---------------- include/linux/userfaultfd_k_ext.h | 62 ++++++++++++++++++++++++++ mm/huge_memory.c | 13 ++++++ mm/hugetlb.c | 15 ++++++- mm/mremap.c | 74 +++++++++++++++++++++++-------- 5 files changed, 145 insertions(+), 59 deletions(-) create mode 100644 include/linux/userfaultfd_k_ext.h -- 2.25.1

2 5

[PATCH OLK-6.6 0/2] optimize lightweight load balancing
by Chen Jinghuang 10 Mar '26

10 Mar '26

*** BLURB HERE *** Chen Jinghuang (1): sched: Fix kabi breakage of struct sched_domain for newidle_stats Peter Zijlstra (Intel) (1): sched/fair: Proportional newidle balance include/linux/sched/topology.h | 6 +++++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 44 ++++++++++++++++++++++++++++++---- kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 7 ++++++ kernel/sched/topology.c | 6 +++++ 6 files changed, 67 insertions(+), 4 deletions(-) -- 2.34.1

2 3

[PATCH OLK-6.6 0/2] optimize lightweight load balancing
by Chen Jinghuang 10 Mar '26

10 Mar '26

*** BLURB HERE *** Chen Jinghuang (1): sched: Fix kabi breakage pf struct sched_domain for newidle_stats Peter Zijlstra (Intel) (1): sched/fair: Proportional newidle balance include/linux/sched/topology.h | 12 ++++++ kernel/sched/core.c | 3 ++ kernel/sched/fair.c | 78 ++++++++++++++++++++++++++++++++-- kernel/sched/features.h | 5 +++ kernel/sched/sched.h | 7 +++ kernel/sched/topology.c | 11 +++++ 6 files changed, 112 insertions(+), 4 deletions(-) -- 2.34.1

2 3

[PATCH OLK-6.6 1/2] sched/fair: Proportional newidle balance
by Chen Jinghuang 10 Mar '26

10 Mar '26

From: "Peter Zijlstra (Intel)" <peterz(a)infradead.org> stable inclusion from stable-v6.6.120 commit 51445190c10a36d292e70db085d0fb6cc3bec94f category: perf bugzilla: https://atomgit.com/openeuler/kernel/issues/8555 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id… -------------------------------- commit 33cf66d88306663d16e4759e9d24766b0aaa2e17 upstream. Add a randomized algorithm that runs newidle balancing proportional to its success rate. This improves schbench significantly: 6.18-rc4: 2.22 Mrps/s 6.18-rc4+revert: 2.04 Mrps/s 6.18-rc4+revert+random: 2.18 Mrps/S Conversely, per Adam Li this affects SpecJBB slightly, reducing it by 1%: 6.17: -6% 6.17+revert: 0% 6.17+revert+random: -1% Signed-off-by: Peter Zijlstra (Intel) <peterz(a)infradead.org> Reviewed-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Chris Mason <clm(a)meta.com> Link: https://lkml.kernel.org/r/6825c50d-7fa7-45d8-9b81-c6e7e25738e2@meta.com Link: https://patch.msgid.link/20251107161739.770122091@infradead.org [ Ajay: Modified to apply on v6.6 ] Signed-off-by: Ajay Kaher <ajay.kaher(a)broadcom.com> Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Conflicts: kernel/sched/core.c kernel/sched/fair.c [context conflicts] Signed-off-by: Chen Jinghuang <chenjinghuang2(a)huawei.com> --- include/linux/sched/topology.h | 3 +++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 44 ++++++++++++++++++++++++++++++---- kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 7 ++++++ kernel/sched/topology.c | 6 +++++ 6 files changed, 64 insertions(+), 4 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 7eee852aa384..95e5d7772800 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -130,6 +130,9 @@ struct sched_domain { unsigned int nr_balance_failed; /* initialise to 0 */ /* idle_balance() stats */ + unsigned int newidle_call; + unsigned int newidle_success; + unsigned int newidle_ratio; u64 max_newidle_lb_cost; unsigned long last_decay_max_lb_cost; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 69d10fdb84d8..cf45ea94bb05 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -119,6 +119,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_se_tp); EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DEFINE_PER_CPU(struct rnd_state, sched_rnd_state); #ifdef CONFIG_QOS_SCHED static void sched_change_qos_group(struct task_struct *tsk, struct task_group *tg); @@ -9976,6 +9977,8 @@ void __init sched_init_smp(void) sched_init_numa(NUMA_NO_NODE); set_sched_cluster(); + prandom_init_once(&sched_rnd_state); + /* * There's no userspace yet to cause hotplug operations; hence all the * CPU masks are stable and all blatant races in the below code cannot diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a4592d35da8b..d16c65d5fa34 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13651,8 +13651,24 @@ void update_max_interval(void) max_load_balance_interval = HZ*num_online_cpus()/10; } -static inline bool update_newidle_cost(struct sched_domain *sd, u64 cost) +static inline void update_newidle_stats(struct sched_domain *sd, unsigned int success) { + sd->newidle_call++; + sd->newidle_success += success; + + if (sd->newidle_call >= 1024) { + sd->newidle_ratio = sd->newidle_success; + sd->newidle_call /= 2; + sd->newidle_success /= 2; + } +} + +static inline bool +update_newidle_cost(struct sched_domain *sd, u64 cost, unsigned int success) +{ + if (cost) + update_newidle_stats(sd, success); + if (cost > sd->max_newidle_lb_cost) { /* * Track max cost of a domain to make sure to not delay the @@ -13700,7 +13716,7 @@ static void rebalance_domains(struct rq *rq, enum cpu_idle_type idle) * Decay the newidle max times here because this is a regular * visit to all the domains. */ - need_decay = update_newidle_cost(sd, 0); + need_decay = update_newidle_cost(sd, 0, 0); max_cost += sd->max_newidle_lb_cost; /* @@ -14336,6 +14352,22 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) break; if (sd->flags & SD_BALANCE_NEWIDLE) { + unsigned int weight = 1; + + if (sched_feat(NI_RANDOM)) { + /* + * Throw a 1k sided dice; and only run + * newidle_balance according to the success + * rate. + */ + u32 d1k = sched_rng() % 1024; + weight = 1 + sd->newidle_ratio; + if (d1k > weight) { + update_newidle_stats(sd, 0); + continue; + } + weight = (1024 + weight/2) / weight; + } pulled_task = load_balance(this_cpu, this_rq, sd, CPU_NEWLY_IDLE, @@ -14343,10 +14375,14 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) t1 = sched_clock_cpu(this_cpu); domain_cost = t1 - t0; - update_newidle_cost(sd, domain_cost); - curr_cost += domain_cost; t0 = t1; + + /* + * Track max cost of a domain to make sure to not delay the + * next wakeup on the CPU. + */ + update_newidle_cost(sd, domain_cost, weight * !!pulled_task); } /* diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 52ea0097c513..24a0c853a8a0 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -106,6 +106,11 @@ SCHED_FEAT(UTIL_EST_FASTUP, true) SCHED_FEAT(LATENCY_WARN, false) +/* + * Do newidle balancing proportional to its success rate using randomization. + */ +SCHED_FEAT(NI_RANDOM, false) + SCHED_FEAT(HZ_BW, true) SCHED_FEAT(IRQ_AVG, false) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index bb581cbdae8d..0c3abb44f365 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -5,6 +5,7 @@ #ifndef _KERNEL_SCHED_SCHED_H #define _KERNEL_SCHED_SCHED_H +#include <linux/prandom.h> #include <linux/sched/affinity.h> #include <linux/sched/autogroup.h> #include <linux/sched/cpufreq.h> @@ -1377,6 +1378,12 @@ static inline bool is_migration_disabled(struct task_struct *p) } DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DECLARE_PER_CPU(struct rnd_state, sched_rnd_state); + +static inline u32 sched_rng(void) +{ + return prandom_u32_state(this_cpu_ptr(&sched_rnd_state)); +} #define cpu_rq(cpu) (&per_cpu(runqueues, (cpu))) #define this_rq() this_cpu_ptr(&runqueues) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 3023e67da0fd..cf847fdf1063 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1644,6 +1644,12 @@ sd_init(struct sched_domain_topology_level *tl, .last_balance = jiffies, .balance_interval = sd_weight, + + /* 50% success rate */ + .newidle_call = 512, + .newidle_success = 256, + .newidle_ratio = 512, + .max_newidle_lb_cost = 0, .last_decay_max_lb_cost = jiffies, .child = child, -- 2.34.1

2 2

[PATCH OLK-6.6 1/2] [Backport] sched/fair: Proportional newidle balance
by Chen Jinghuang 10 Mar '26

10 Mar '26

From: "Peter Zijlstra (Intel)" <peterz(a)infradead.org> stable inclusion from stable-v6.6.120 commit 51445190c10a36d292e70db085d0fb6cc3bec94f category: perf bugzilla: https://atomgit.com/openeuler/kernel/issues/8555 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id… -------------------------------- commit 33cf66d88306663d16e4759e9d24766b0aaa2e17 upstream. Add a randomized algorithm that runs newidle balancing proportional to its success rate. This improves schbench significantly: 6.18-rc4: 2.22 Mrps/s 6.18-rc4+revert: 2.04 Mrps/s 6.18-rc4+revert+random: 2.18 Mrps/S Conversely, per Adam Li this affects SpecJBB slightly, reducing it by 1%: 6.17: -6% 6.17+revert: 0% 6.17+revert+random: -1% Signed-off-by: Peter Zijlstra (Intel) <peterz(a)infradead.org> Reviewed-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Chris Mason <clm(a)meta.com> Link: https://lkml.kernel.org/r/6825c50d-7fa7-45d8-9b81-c6e7e25738e2@meta.com Link: https://patch.msgid.link/20251107161739.770122091@infradead.org [ Ajay: Modified to apply on v6.6 ] Signed-off-by: Ajay Kaher <ajay.kaher(a)broadcom.com> Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Conflicts: kernel/sched/core.c kernel/sched/fair.c [context conflicts] Signed-off-by: Chen Jinghuang <chenjinghuang2(a)huawei.com> --- include/linux/sched/topology.h | 3 +++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 44 ++++++++++++++++++++++++++++++---- kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 7 ++++++ kernel/sched/topology.c | 6 +++++ 6 files changed, 64 insertions(+), 4 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 7eee852aa384..95e5d7772800 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -130,6 +130,9 @@ struct sched_domain { unsigned int nr_balance_failed; /* initialise to 0 */ /* idle_balance() stats */ + unsigned int newidle_call; + unsigned int newidle_success; + unsigned int newidle_ratio; u64 max_newidle_lb_cost; unsigned long last_decay_max_lb_cost; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 69d10fdb84d8..cf45ea94bb05 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -119,6 +119,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_se_tp); EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DEFINE_PER_CPU(struct rnd_state, sched_rnd_state); #ifdef CONFIG_QOS_SCHED static void sched_change_qos_group(struct task_struct *tsk, struct task_group *tg); @@ -9976,6 +9977,8 @@ void __init sched_init_smp(void) sched_init_numa(NUMA_NO_NODE); set_sched_cluster(); + prandom_init_once(&sched_rnd_state); + /* * There's no userspace yet to cause hotplug operations; hence all the * CPU masks are stable and all blatant races in the below code cannot diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a4592d35da8b..d16c65d5fa34 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13651,8 +13651,24 @@ void update_max_interval(void) max_load_balance_interval = HZ*num_online_cpus()/10; } -static inline bool update_newidle_cost(struct sched_domain *sd, u64 cost) +static inline void update_newidle_stats(struct sched_domain *sd, unsigned int success) { + sd->newidle_call++; + sd->newidle_success += success; + + if (sd->newidle_call >= 1024) { + sd->newidle_ratio = sd->newidle_success; + sd->newidle_call /= 2; + sd->newidle_success /= 2; + } +} + +static inline bool +update_newidle_cost(struct sched_domain *sd, u64 cost, unsigned int success) +{ + if (cost) + update_newidle_stats(sd, success); + if (cost > sd->max_newidle_lb_cost) { /* * Track max cost of a domain to make sure to not delay the @@ -13700,7 +13716,7 @@ static void rebalance_domains(struct rq *rq, enum cpu_idle_type idle) * Decay the newidle max times here because this is a regular * visit to all the domains. */ - need_decay = update_newidle_cost(sd, 0); + need_decay = update_newidle_cost(sd, 0, 0); max_cost += sd->max_newidle_lb_cost; /* @@ -14336,6 +14352,22 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) break; if (sd->flags & SD_BALANCE_NEWIDLE) { + unsigned int weight = 1; + + if (sched_feat(NI_RANDOM)) { + /* + * Throw a 1k sided dice; and only run + * newidle_balance according to the success + * rate. + */ + u32 d1k = sched_rng() % 1024; + weight = 1 + sd->newidle_ratio; + if (d1k > weight) { + update_newidle_stats(sd, 0); + continue; + } + weight = (1024 + weight/2) / weight; + } pulled_task = load_balance(this_cpu, this_rq, sd, CPU_NEWLY_IDLE, @@ -14343,10 +14375,14 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) t1 = sched_clock_cpu(this_cpu); domain_cost = t1 - t0; - update_newidle_cost(sd, domain_cost); - curr_cost += domain_cost; t0 = t1; + + /* + * Track max cost of a domain to make sure to not delay the + * next wakeup on the CPU. + */ + update_newidle_cost(sd, domain_cost, weight * !!pulled_task); } /* diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 52ea0097c513..24a0c853a8a0 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -106,6 +106,11 @@ SCHED_FEAT(UTIL_EST_FASTUP, true) SCHED_FEAT(LATENCY_WARN, false) +/* + * Do newidle balancing proportional to its success rate using randomization. + */ +SCHED_FEAT(NI_RANDOM, false) + SCHED_FEAT(HZ_BW, true) SCHED_FEAT(IRQ_AVG, false) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index bb581cbdae8d..0c3abb44f365 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -5,6 +5,7 @@ #ifndef _KERNEL_SCHED_SCHED_H #define _KERNEL_SCHED_SCHED_H +#include <linux/prandom.h> #include <linux/sched/affinity.h> #include <linux/sched/autogroup.h> #include <linux/sched/cpufreq.h> @@ -1377,6 +1378,12 @@ static inline bool is_migration_disabled(struct task_struct *p) } DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DECLARE_PER_CPU(struct rnd_state, sched_rnd_state); + +static inline u32 sched_rng(void) +{ + return prandom_u32_state(this_cpu_ptr(&sched_rnd_state)); +} #define cpu_rq(cpu) (&per_cpu(runqueues, (cpu))) #define this_rq() this_cpu_ptr(&runqueues) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 3023e67da0fd..cf847fdf1063 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1644,6 +1644,12 @@ sd_init(struct sched_domain_topology_level *tl, .last_balance = jiffies, .balance_interval = sd_weight, + + /* 50% success rate */ + .newidle_call = 512, + .newidle_success = 256, + .newidle_ratio = 512, + .max_newidle_lb_cost = 0, .last_decay_max_lb_cost = jiffies, .child = child, -- 2.34.1

2 3

[PATCH 1/2] [Backport] sched/fair: Proportional newidle balance
by Chen Jinghuang 10 Mar '26

10 Mar '26

From: "Peter Zijlstra (Intel)" <peterz(a)infradead.org> stable inclusion from stable-v6.6.120 commit 51445190c10a36d292e70db085d0fb6cc3bec94f category: perf bugzilla: https://atomgit.com/openeuler/kernel/issues/8555 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id… -------------------------------- commit 33cf66d88306663d16e4759e9d24766b0aaa2e17 upstream. Add a randomized algorithm that runs newidle balancing proportional to its success rate. This improves schbench significantly: 6.18-rc4: 2.22 Mrps/s 6.18-rc4+revert: 2.04 Mrps/s 6.18-rc4+revert+random: 2.18 Mrps/S Conversely, per Adam Li this affects SpecJBB slightly, reducing it by 1%: 6.17: -6% 6.17+revert: 0% 6.17+revert+random: -1% Signed-off-by: Peter Zijlstra (Intel) <peterz(a)infradead.org> Reviewed-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Chris Mason <clm(a)meta.com> Link: https://lkml.kernel.org/r/6825c50d-7fa7-45d8-9b81-c6e7e25738e2@meta.com Link: https://patch.msgid.link/20251107161739.770122091@infradead.org [ Ajay: Modified to apply on v6.6 ] Signed-off-by: Ajay Kaher <ajay.kaher(a)broadcom.com> Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Conflicts: kernel/sched/core.c kernel/sched/fair.c [context conflicts] Signed-off-by: Chen Jinghuang <chenjinghuang2(a)huawei.com> --- include/linux/sched/topology.h | 3 +++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 44 ++++++++++++++++++++++++++++++---- kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 7 ++++++ kernel/sched/topology.c | 6 +++++ 6 files changed, 64 insertions(+), 4 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 7eee852aa384..95e5d7772800 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -130,6 +130,9 @@ struct sched_domain { unsigned int nr_balance_failed; /* initialise to 0 */ /* idle_balance() stats */ + unsigned int newidle_call; + unsigned int newidle_success; + unsigned int newidle_ratio; u64 max_newidle_lb_cost; unsigned long last_decay_max_lb_cost; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 69d10fdb84d8..cf45ea94bb05 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -119,6 +119,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_se_tp); EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DEFINE_PER_CPU(struct rnd_state, sched_rnd_state); #ifdef CONFIG_QOS_SCHED static void sched_change_qos_group(struct task_struct *tsk, struct task_group *tg); @@ -9976,6 +9977,8 @@ void __init sched_init_smp(void) sched_init_numa(NUMA_NO_NODE); set_sched_cluster(); + prandom_init_once(&sched_rnd_state); + /* * There's no userspace yet to cause hotplug operations; hence all the * CPU masks are stable and all blatant races in the below code cannot diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a4592d35da8b..d16c65d5fa34 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13651,8 +13651,24 @@ void update_max_interval(void) max_load_balance_interval = HZ*num_online_cpus()/10; } -static inline bool update_newidle_cost(struct sched_domain *sd, u64 cost) +static inline void update_newidle_stats(struct sched_domain *sd, unsigned int success) { + sd->newidle_call++; + sd->newidle_success += success; + + if (sd->newidle_call >= 1024) { + sd->newidle_ratio = sd->newidle_success; + sd->newidle_call /= 2; + sd->newidle_success /= 2; + } +} + +static inline bool +update_newidle_cost(struct sched_domain *sd, u64 cost, unsigned int success) +{ + if (cost) + update_newidle_stats(sd, success); + if (cost > sd->max_newidle_lb_cost) { /* * Track max cost of a domain to make sure to not delay the @@ -13700,7 +13716,7 @@ static void rebalance_domains(struct rq *rq, enum cpu_idle_type idle) * Decay the newidle max times here because this is a regular * visit to all the domains. */ - need_decay = update_newidle_cost(sd, 0); + need_decay = update_newidle_cost(sd, 0, 0); max_cost += sd->max_newidle_lb_cost; /* @@ -14336,6 +14352,22 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) break; if (sd->flags & SD_BALANCE_NEWIDLE) { + unsigned int weight = 1; + + if (sched_feat(NI_RANDOM)) { + /* + * Throw a 1k sided dice; and only run + * newidle_balance according to the success + * rate. + */ + u32 d1k = sched_rng() % 1024; + weight = 1 + sd->newidle_ratio; + if (d1k > weight) { + update_newidle_stats(sd, 0); + continue; + } + weight = (1024 + weight/2) / weight; + } pulled_task = load_balance(this_cpu, this_rq, sd, CPU_NEWLY_IDLE, @@ -14343,10 +14375,14 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) t1 = sched_clock_cpu(this_cpu); domain_cost = t1 - t0; - update_newidle_cost(sd, domain_cost); - curr_cost += domain_cost; t0 = t1; + + /* + * Track max cost of a domain to make sure to not delay the + * next wakeup on the CPU. + */ + update_newidle_cost(sd, domain_cost, weight * !!pulled_task); } /* diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 52ea0097c513..24a0c853a8a0 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -106,6 +106,11 @@ SCHED_FEAT(UTIL_EST_FASTUP, true) SCHED_FEAT(LATENCY_WARN, false) +/* + * Do newidle balancing proportional to its success rate using randomization. + */ +SCHED_FEAT(NI_RANDOM, false) + SCHED_FEAT(HZ_BW, true) SCHED_FEAT(IRQ_AVG, false) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index bb581cbdae8d..0c3abb44f365 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -5,6 +5,7 @@ #ifndef _KERNEL_SCHED_SCHED_H #define _KERNEL_SCHED_SCHED_H +#include <linux/prandom.h> #include <linux/sched/affinity.h> #include <linux/sched/autogroup.h> #include <linux/sched/cpufreq.h> @@ -1377,6 +1378,12 @@ static inline bool is_migration_disabled(struct task_struct *p) } DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DECLARE_PER_CPU(struct rnd_state, sched_rnd_state); + +static inline u32 sched_rng(void) +{ + return prandom_u32_state(this_cpu_ptr(&sched_rnd_state)); +} #define cpu_rq(cpu) (&per_cpu(runqueues, (cpu))) #define this_rq() this_cpu_ptr(&runqueues) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 3023e67da0fd..cf847fdf1063 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1644,6 +1644,12 @@ sd_init(struct sched_domain_topology_level *tl, .last_balance = jiffies, .balance_interval = sd_weight, + + /* 50% success rate */ + .newidle_call = 512, + .newidle_success = 256, + .newidle_ratio = 512, + .max_newidle_lb_cost = 0, .last_decay_max_lb_cost = jiffies, .child = child, -- 2.34.1

1 1

[PATCH 1/2] [Backport] sched/fair: Proportional newidle balance
by Chen Jinghuang 10 Mar '26

10 Mar '26

From: "Peter Zijlstra (Intel)" <peterz(a)infradead.org> stable inclusion from stable-v6.6.120 commit 51445190c10a36d292e70db085d0fb6cc3bec94f category: perf bugzilla: https://atomgit.com/openeuler/kernel/issues/8555 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id… -------------------------------- commit 33cf66d88306663d16e4759e9d24766b0aaa2e17 upstream. Add a randomized algorithm that runs newidle balancing proportional to its success rate. This improves schbench significantly: 6.18-rc4: 2.22 Mrps/s 6.18-rc4+revert: 2.04 Mrps/s 6.18-rc4+revert+random: 2.18 Mrps/S Conversely, per Adam Li this affects SpecJBB slightly, reducing it by 1%: 6.17: -6% 6.17+revert: 0% 6.17+revert+random: -1% Signed-off-by: Peter Zijlstra (Intel) <peterz(a)infradead.org> Reviewed-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Dietmar Eggemann <dietmar.eggemann(a)arm.com> Tested-by: Chris Mason <clm(a)meta.com> Link: https://lkml.kernel.org/r/6825c50d-7fa7-45d8-9b81-c6e7e25738e2@meta.com Link: https://patch.msgid.link/20251107161739.770122091@infradead.org [ Ajay: Modified to apply on v6.6 ] Signed-off-by: Ajay Kaher <ajay.kaher(a)broadcom.com> Signed-off-by: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Conflicts: kernel/sched/core.c kernel/sched/fair.c [context conflicts] Signed-off-by: Chen Jinghuang <chenjinghuang2(a)huawei.com> --- include/linux/sched/topology.h | 3 +++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 44 ++++++++++++++++++++++++++++++---- kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 7 ++++++ kernel/sched/topology.c | 6 +++++ 6 files changed, 64 insertions(+), 4 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 7eee852aa384..95e5d7772800 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -130,6 +130,9 @@ struct sched_domain { unsigned int nr_balance_failed; /* initialise to 0 */ /* idle_balance() stats */ + unsigned int newidle_call; + unsigned int newidle_success; + unsigned int newidle_ratio; u64 max_newidle_lb_cost; unsigned long last_decay_max_lb_cost; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 69d10fdb84d8..cf45ea94bb05 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -119,6 +119,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_util_est_se_tp); EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DEFINE_PER_CPU(struct rnd_state, sched_rnd_state); #ifdef CONFIG_QOS_SCHED static void sched_change_qos_group(struct task_struct *tsk, struct task_group *tg); @@ -9976,6 +9977,8 @@ void __init sched_init_smp(void) sched_init_numa(NUMA_NO_NODE); set_sched_cluster(); + prandom_init_once(&sched_rnd_state); + /* * There's no userspace yet to cause hotplug operations; hence all the * CPU masks are stable and all blatant races in the below code cannot diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a4592d35da8b..d16c65d5fa34 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13651,8 +13651,24 @@ void update_max_interval(void) max_load_balance_interval = HZ*num_online_cpus()/10; } -static inline bool update_newidle_cost(struct sched_domain *sd, u64 cost) +static inline void update_newidle_stats(struct sched_domain *sd, unsigned int success) { + sd->newidle_call++; + sd->newidle_success += success; + + if (sd->newidle_call >= 1024) { + sd->newidle_ratio = sd->newidle_success; + sd->newidle_call /= 2; + sd->newidle_success /= 2; + } +} + +static inline bool +update_newidle_cost(struct sched_domain *sd, u64 cost, unsigned int success) +{ + if (cost) + update_newidle_stats(sd, success); + if (cost > sd->max_newidle_lb_cost) { /* * Track max cost of a domain to make sure to not delay the @@ -13700,7 +13716,7 @@ static void rebalance_domains(struct rq *rq, enum cpu_idle_type idle) * Decay the newidle max times here because this is a regular * visit to all the domains. */ - need_decay = update_newidle_cost(sd, 0); + need_decay = update_newidle_cost(sd, 0, 0); max_cost += sd->max_newidle_lb_cost; /* @@ -14336,6 +14352,22 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) break; if (sd->flags & SD_BALANCE_NEWIDLE) { + unsigned int weight = 1; + + if (sched_feat(NI_RANDOM)) { + /* + * Throw a 1k sided dice; and only run + * newidle_balance according to the success + * rate. + */ + u32 d1k = sched_rng() % 1024; + weight = 1 + sd->newidle_ratio; + if (d1k > weight) { + update_newidle_stats(sd, 0); + continue; + } + weight = (1024 + weight/2) / weight; + } pulled_task = load_balance(this_cpu, this_rq, sd, CPU_NEWLY_IDLE, @@ -14343,10 +14375,14 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) t1 = sched_clock_cpu(this_cpu); domain_cost = t1 - t0; - update_newidle_cost(sd, domain_cost); - curr_cost += domain_cost; t0 = t1; + + /* + * Track max cost of a domain to make sure to not delay the + * next wakeup on the CPU. + */ + update_newidle_cost(sd, domain_cost, weight * !!pulled_task); } /* diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 52ea0097c513..6cd6b5307430 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -106,6 +106,11 @@ SCHED_FEAT(UTIL_EST_FASTUP, true) SCHED_FEAT(LATENCY_WARN, false) +/* + * Do newidle balancing proportional to its success rate using randomization. + */ +SCHED_FEAT(NI_RANDOM, true) + SCHED_FEAT(HZ_BW, true) SCHED_FEAT(IRQ_AVG, false) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index bb581cbdae8d..0c3abb44f365 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -5,6 +5,7 @@ #ifndef _KERNEL_SCHED_SCHED_H #define _KERNEL_SCHED_SCHED_H +#include <linux/prandom.h> #include <linux/sched/affinity.h> #include <linux/sched/autogroup.h> #include <linux/sched/cpufreq.h> @@ -1377,6 +1378,12 @@ static inline bool is_migration_disabled(struct task_struct *p) } DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +DECLARE_PER_CPU(struct rnd_state, sched_rnd_state); + +static inline u32 sched_rng(void) +{ + return prandom_u32_state(this_cpu_ptr(&sched_rnd_state)); +} #define cpu_rq(cpu) (&per_cpu(runqueues, (cpu))) #define this_rq() this_cpu_ptr(&runqueues) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 3023e67da0fd..cf847fdf1063 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1644,6 +1644,12 @@ sd_init(struct sched_domain_topology_level *tl, .last_balance = jiffies, .balance_interval = sd_weight, + + /* 50% success rate */ + .newidle_call = 512, + .newidle_success = 256, + .newidle_ratio = 512, + .max_newidle_lb_cost = 0, .last_decay_max_lb_cost = jiffies, .child = child, -- 2.34.1

1 1