- Kernel - mailweb.openeuler.org

[PATCH OLK-6.6] net: bridge: Reject descending VLAN tunnel ranges
by JiangJieHua 29 Sep '26

29 Sep '26

From: Ruoyu Wang <ruoyuw560(a)gmail.com> stable inclusion from stable-v6.6.157 commit 4b0f5b25f114651f03058395b490626c4444875e category: bugfix bugzilla: https://atomgit.com/src-openeuler/kernel/issues/19404 CVE: CVE-2026-90114 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id… -------------------------------- [ Upstream commit b74a072d8fb71d3c9ffba4a17d5943e63266fb38 ] A pair of descending VLAN and tunnel IDs can pass the tunnel range span check. The VLAN subtraction produces a negative int, which is converted to unsigned when compared with the u32 tunnel ID subtraction. It can therefore equal the wrapped tunnel ID delta. The range loop then performs no iterations. Since the batched notification handling added a post-loop error check, this leaves err uninitialized and makes the request's return value unpredictable. Reject descending VLAN ranges before comparing the spans. Valid ascending and single-entry ranges remain unchanged, while malformed descending ranges consistently return -EINVAL. This issue was found by a static analysis checker and confirmed by manual source review. Fixes: 94339443686b ("net: bridge: notify on vlan tunnel changes done via the old api") Signed-off-by: Ruoyu Wang <ruoyuw560(a)gmail.com> Acked-by: Nikolay Aleksandrov <razor(a)blackwall.org> Link: https://patch.msgid.link/20260814134053.1387275-1-ruoyuw560@gmail.com Signed-off-by: David S. Miller <davem(a)davemloft.net> Signed-off-by: Jakub Kicinski <kuba(a)kernel.org> Signed-off-by: Sasha Levin <sashal(a)kernel.org> Signed-off-by: JiangJieHua <jiangjiehua1(a)huawei.com> --- net/bridge/br_netlink_tunnel.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/net/bridge/br_netlink_tunnel.c b/net/bridge/br_netlink_tunnel.c index b006d1127c549..3ee0e14be7279 100644 --- a/net/bridge/br_netlink_tunnel.c +++ b/net/bridge/br_netlink_tunnel.c @@ -302,7 +302,8 @@ int br_process_vlan_tunnel_info(const struct net_bridge *br, if (!(tinfo_last->flags & BRIDGE_VLAN_INFO_RANGE_BEGIN)) return -EINVAL; - if ((tinfo_curr->vid - tinfo_last->vid) != + if (tinfo_curr->vid < tinfo_last->vid || + (tinfo_curr->vid - tinfo_last->vid) != (tinfo_curr->tunid - tinfo_last->tunid)) return -EINVAL; t = tinfo_last->tunid; -- 2.33.8

2 1

[PATCH OLK-6.6] ip6mr: do not clone dst in ip6mr_cache_report()
by JiangJieHua 29 Sep '26

29 Sep '26

From: Eric Dumazet <edumazet(a)google.com> mainline inclusion from mainline-v7.3-rc1 commit 235b42b5860189eb8c27c36435ad932cae65a734 category: bugfix bugzilla: https://atomgit.com/src-openeuler/kernel/issues/19403 CVE: CVE-2026-90111 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… -------------------------------- IPv6 input attaches a non-refcounted (NOREF) dst to skbs under RCU. When an ingress multicast packet misses MFC lookup, ip6mr_cache_unresolved() places the skb onto the unresolved queue, escaping the receive-side RCU grace period. If the underlying route is deleted and freed, and the MFC queue is later resolved with a wrong parent interface, ip6_mr_forward() invokes ip6mr_cache_report(..., MRT6MSG_WRONGMIF), which executes dst_clone(skb_dst(pkt)) on the freed dst entry, triggering a slab use-after-free. Report packets queued to mroute6_sk (a raw socket) and netlink notifications do not require an attached dst entry. Fix this by: 1. Removing dst_clone() in ip6mr_cache_report() and ensuring report skbs do not hold a dst. 2. Dropping skb_dst before queuing unresolved skbs in ip6mr_cache_unresolved(), matching the fact that multicast forwarding resolves outgoing routes anew via ip6_route_output(). Fixes: 67f415dd2906 ("ipv6: convert rx data path to not take refcnt on dst") Reported-by: Zero Day Initiative <zdi-disclosures(a)trendmicro.com> Signed-off-by: Eric Dumazet <edumazet(a)google.com> Reviewed-by: Hangbin Liu <liuhangbin(a)kylinos.cn> Link: https://patch.msgid.link/20260818172755.4083692-1-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba(a)kernel.org> Conflicts: net/ipv6/ip6mr.c [Conflicts are caused by the following patches not being merged: ip6mr_cache_report() still looks up mroute6_sk after building the message and uses sock_queue_rcv_skb() instead of sock_queue_rcv_skb_reason()/sk_skb_reason_drop(): 8d59a7a188c0 ("ip6mr: plug drop_reason to ip6mr_cache_report()") ip6mr_cache_unresolved() still uses inline error returns and the if/else layout instead of the read_pnet()/check_net()/goto err refactor: 2c309a863dd6 ("ip6mr: Free mr_table after RCU grace period.")] Signed-off-by: JiangJieHua <jiangjiehua1(a)huawei.com> --- net/ipv6/ip6mr.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c index 18451dfeae682..7676bd1ff8dbe 100644 --- a/net/ipv6/ip6mr.c +++ b/net/ipv6/ip6mr.c @@ -1128,7 +1128,6 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt, msg->im6_src = ipv6_hdr(pkt)->saddr; msg->im6_dst = ipv6_hdr(pkt)->daddr; - skb_dst_set(skb, dst_clone(skb_dst(pkt))); skb->ip_summed = CHECKSUM_UNNECESSARY; } @@ -1138,6 +1137,7 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt, return -EINVAL; } + skb_dst_drop(skb); mrt6msg_netlink_event(mrt, skb); /* Deliver to user space multicast routing algorithms */ @@ -1217,6 +1217,7 @@ static int ip6mr_cache_unresolved(struct mr_table *mrt, mifi_t mifi, skb->dev = dev; skb->skb_iif = dev->ifindex; } + skb_dst_drop(skb); skb_queue_tail(&c->_c.mfc_un.unres.unresolved, skb); err = 0; } -- 2.33.8

2 1

[PATCH OLK-6.6] vxlan: fix reading neigh ha
by JiangJieHua 29 Sep '26

29 Sep '26

From: Nikolay Aleksandrov <razor(a)blackwall.org> mainline inclusion from mainline-v7.3-rc1 commit b824059a673b2283e78c7aae2c7d257aad7f0e1d category: bugfix bugzilla: https://atomgit.com/src-openeuler/kernel/issues/19398 CVE: CVE-2026-90105 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… -------------------------------- Currently arp/neigh_reduce read neigh ha directly which can lead to partial reads while the neigh is being updated. Use neigh_ha_snapshot to take a stable snapshot of the address similar to route_shortcircuit which already does the right thing. Fixes: e4f67addf158 ("add DOVE extensions for VXLAN") Fixes: f564f45c4518 ("vxlan: add ipv6 proxy support") Signed-off-by: Nikolay Aleksandrov <razor(a)blackwall.org> Reviewed-by: Petr Machata <petrm(a)nvidia.com> Reviewed-by: Ido Schimmel <idosch(a)nvidia.com> Link: https://patch.msgid.link/20260818150756.890025-3-razor@blackwall.org Signed-off-by: Jakub Kicinski <kuba(a)kernel.org> Conflicts: drivers/net/vxlan/vxlan_core.c [Conflicts are caused by the following patches not being merged: 5cde39ea3881 ("vxlan: Rename FDB Tx lookup function") 804b09be09f8 ("vxlan: Add RCU read-side critical sections in the Tx path")] Signed-off-by: JiangJieHua <jiangjiehua1(a)huawei.com> --- drivers/net/vxlan/vxlan_core.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c index 4ebeb450b47a9..9142af281188b 100644 --- a/drivers/net/vxlan/vxlan_core.c +++ b/drivers/net/vxlan/vxlan_core.c @@ -1888,6 +1888,7 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) if (n) { struct vxlan_rdst *rdst = NULL; + u8 ha[ETH_ALEN] __aligned(2); struct vxlan_fdb *f; struct sk_buff *reply; @@ -1896,7 +1897,9 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) goto out; } - f = vxlan_find_mac(vxlan, n->ha, vni); + neigh_ha_snapshot(ha, n, n->dev); + + f = vxlan_find_mac(vxlan, ha, vni); if (f) rdst = first_remote_rcu(f); if (rdst && vxlan_addr_any(&rdst->remote_ip)) { @@ -1906,7 +1909,7 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) } reply = arp_create(ARPOP_REPLY, ETH_P_ARP, sip, dev, tip, sha, - n->ha, sha); + ha, sha); neigh_release(n); @@ -1939,7 +1942,8 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) #if IS_ENABLED(CONFIG_IPV6) static struct sk_buff *vxlan_na_create(struct sk_buff *request, - struct neighbour *n, bool isrouter) + struct neighbour *n, u8 *ha, + bool isrouter) { struct net_device *dev = request->dev; struct sk_buff *reply; @@ -1985,7 +1989,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request, /* Ethernet header */ ether_addr_copy(eth_hdr(reply)->h_dest, daddr); - ether_addr_copy(eth_hdr(reply)->h_source, n->ha); + ether_addr_copy(eth_hdr(reply)->h_source, ha); eth_hdr(reply)->h_proto = htons(ETH_P_IPV6); reply->protocol = htons(ETH_P_IPV6); @@ -2014,7 +2018,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request, na->icmph.icmp6_override = 1; na->icmph.icmp6_solicited = 1; na->target = ns->target; - ether_addr_copy(&na->opt[2], n->ha); + ether_addr_copy(&na->opt[2], ha); na->opt[0] = ND_OPT_TARGET_LL_ADDR; na->opt[1] = na_olen >> 3; @@ -2057,6 +2061,7 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) if (n) { struct vxlan_rdst *rdst = NULL; + u8 ha[ETH_ALEN] __aligned(2); struct vxlan_fdb *f; struct sk_buff *reply; @@ -2065,7 +2070,8 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) goto out; } - f = vxlan_find_mac(vxlan, n->ha, vni); + neigh_ha_snapshot(ha, n, n->dev); + f = vxlan_find_mac(vxlan, ha, vni); if (f) rdst = first_remote_rcu(f); if (rdst && vxlan_addr_any(&rdst->remote_ip)) { @@ -2074,7 +2080,7 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) goto out; } - reply = vxlan_na_create(skb, n, + reply = vxlan_na_create(skb, n, ha, !!(f ? f->flags & NTF_ROUTER : 0)); neigh_release(n); -- 2.33.8

2 1

[PATCH OLK-6.6] tcp: fix corruption of urgent data on multi-segment retransmit
by JiangJieHua 29 Sep '26

29 Sep '26

From: Jiayuan Chen <jiayuan.chen(a)linux.dev> stable inclusion from stable-v6.6.157 commit 4fd2ee4ac154c204d95d8db72221446dac5af9b8 category: bugfix bugzilla: https://atomgit.com/src-openeuler/kernel/issues/19376 CVE: CVE-2026-90054 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id… -------------------------------- [ Upstream commit ce2b807f42ed5e55567b8864ab72963f90779270 ] On the normal xmit path, while in urgent mode we refuse to build a multi-segment TSO packet, so every segment gets its own urg_ptr: /* tcp_write_xmit() */ limit = mss_now; if (tso_segs > 1 && !tcp_urg_mode(tp)) limit = tcp_mss_split_point(...); The retransmit path has no such guard. __tcp_retransmit_skb() builds a segs > 1 skb and hands it to the GSO layer, which only advances th->seq per segment and copies urg_ptr verbatim: /* __tcp_retransmit_skb() */ len = cur_mss * segs; /* segs > 1, no urg_mode check */ ... /* tcp_gso_segment(): bumps seq only, urg_ptr is copied */ urg_ptr is an offset from the segment's own seq, so a copied value points at a different place on each segment. The receiver rebuilds the absolute urgent seq as seg.seq + urg_ptr, so it walks a moving urgent point instead of the one OOB byte: seg1 seq 1 urg_ptr 5001 -> urgent @ 5001 (ok) seg2 seq 1001 urg_ptr 5001 -> urgent @ 6001 (wrong, +MSS) seg3 seq 2001 urg_ptr 5001 -> urgent @ 7001 (wrong, +2*MSS) The real OOB byte is never pointed at, so the receiver stops splicing it out and delivers it as normal in-band data, corrupting the stream. Guard the retransmit length like the xmit path: keep segs = 1 while in urgent mode. Fixes: 10d3be569243 ("tcp-tso: do not split TSO packets at retransmit time") Signed-off-by: Jiayuan Chen <jiayuan.chen(a)linux.dev> Reviewed-by: Eric Dumazet <edumazet(a)google.com> Link: https://patch.msgid.link/20260826141145.67823-1-jiayuan.chen@linux.dev Signed-off-by: Jakub Kicinski <kuba(a)kernel.org> Signed-off-by: Sasha Levin <sashal(a)kernel.org> Signed-off-by: JiangJieHua <jiangjiehua1(a)huawei.com> --- net/ipv4/tcp_output.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 022c40469b764..5742ee246eea2 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3478,7 +3478,7 @@ int __tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb, int segs) avail_wnd = cur_mss; } - len = cur_mss * segs; + len = cur_mss * (tcp_urg_mode(tp) ? 1 : segs); if (len > avail_wnd) { len = rounddown(avail_wnd, cur_mss); if (!len) -- 2.33.8

2 1

[PATCH OLK-6.6 0/2] eventpoll: use spinlock under CONFIG_PREEMPT_RT, rwlock otherwise
by Ran Hongyun 29 Sep '26

29 Sep '26

Commit d26ee5945bbc ("eventpoll: Replace rwlock with spinlock") replaced ep->lock rwlock with spinlock to fix RT priority inversion. However, the rwlock read-side cannot be priority-boosted on PREEMPT_RT (rwbase_rt readers are not rt_mutex owners), while on non-RT kernels preempt_disable already guarantees bounded waiting. So the inversion is RT-only. Replacing rwlock with spinlock unconditionally causes performance regression on non-RT systems with many producers, since producers can no longer hold the read lock concurrently. This series reverts the unconditional replacement, then re-applies it under CONFIG_PREEMPT_RT via ep_lock_* wrapper macros: - RT: spinlock_t (PI chain complete, no inversion) - non-RT: rwlock_t (preserved for producer concurrency) The lockless helpers list_add_tail_lockless() and chain_epi_lockless() are compiled out under CONFIG_PREEMPT_RT. Ran Hongyun (2): Revert "eventpoll: Replace rwlock with spinlock" eventpoll: use spinlock under CONFIG_PREEMPT_RT, rwlock otherwise fs/eventpoll.c | 174 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 154 insertions(+), 20 deletions(-) -- 2.52.0

2 1

[PATCH OLK-6.6 0/2] eventpoll: use spinlock under CONFIG_PREEMPT_RT, rwlock otherwise
by Ran Hongyun 29 Sep '26

29 Sep '26

Commit d26ee5945bbc ("eventpoll: Replace rwlock with spinlock") replaced ep->lock rwlock with spinlock to fix RT priority inversion. However, the rwlock read-side cannot be priority-boosted on PREEMPT_RT (rwbase_rt readers are not rt_mutex owners), while on non-RT kernels preempt_disable already guarantees bounded waiting. So the inversion is RT-only. Replacing rwlock with spinlock unconditionally causes performance regression on non-RT systems with many producers, since producers can no longer hold the read lock concurrently. This series reverts the unconditional replacement, then re-applies it under CONFIG_PREEMPT_RT via ep_lock_* wrapper macros: - RT: spinlock_t (PI chain complete, no inversion) - non-RT: rwlock_t (preserved for producer concurrency) The lockless helpers list_add_tail_lockless() and chain_epi_lockless() are compiled out under CONFIG_PREEMPT_RT. Ran Hongyun (2): Revert "eventpoll: Replace rwlock with spinlock" eventpoll: use spinlock under CONFIG_PREEMPT_RT, rwlock otherwise fs/eventpoll.c | 174 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 154 insertions(+), 20 deletions(-) -- 2.52.0

2 1

[PATCH OLK-6.6 2/2] eventpoll: use spinlock under CONFIG_PREEMPT_RT, rwlock otherwise
by Ran Hongyun 29 Sep '26

29 Sep '26

hulk inclusion category: bugfix bugzilla: https://atomgit.com/openeuler/kernel/issues/10065 -------------------------------- Introduce ep_lock_* wrapper macros that select spinlock_t under CONFIG_PREEMPT_RT and rwlock_t otherwise. The rwlock read-side cannot be priority-boosted on RT (rwbase_rt readers are not rt_mutex owners), causing unbounded priority inversion; spinlock fixes this. On non-RT kernels the rwlock is preserved for producer concurrency. The lockless helpers list_add_tail_lockless() and chain_epi_lockless() are compiled out under CONFIG_PREEMPT_RT since the spinlock serializes all producers. Fixes: a218cc491420 ("epoll: use rwlock in order to reduce ep_poll_callback() contention") Signed-off-by: Ran Hongyun <ranhongyun1(a)huawei.com> --- fs/eventpoll.c | 95 +++++++++++++++++++++++++++++++++++++------------- 1 file changed, 71 insertions(+), 24 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ba719222a2da..523e69de1e32 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -39,13 +39,45 @@ #include <linux/rculist.h> #include <net/busy_poll.h> +/* + * Under CONFIG_PREEMPT_RT the ep->lock is a spinlock because the rwlock + * read-side cannot be priority-boosted (rwbase_rt readers are not + * rt_mutex owners), which causes unbounded priority inversion when a + * high-priority consumer (writer) waits on low-priority producers + * (readers). On non-RT kernels the rwlock is preserved so that multiple + * producers can run concurrently, which scales better with large event + * sources. + */ + +#ifdef CONFIG_PREEMPT_RT + +#define ep_lock_t spinlock_t +#define ep_lock_init(lock) spin_lock_init(lock) +#define ep_lock_irq(lock) spin_lock_irq(lock) +#define ep_unlock_irq(lock) spin_unlock_irq(lock) +#define ep_lock_irqsave(lock, flags) spin_lock_irqsave(lock, flags) +#define ep_unlock_irqrestore(lock, flags) \ + spin_unlock_irqrestore(lock, flags) + +#else /* !CONFIG_PREEMPT_RT */ + +#define ep_lock_t rwlock_t +#define ep_lock_init(lock) rwlock_init(lock) +#define ep_lock_irq(lock) write_lock_irq(lock) +#define ep_unlock_irq(lock) write_unlock_irq(lock) +#define ep_lock_irqsave(lock, flags) read_lock_irqsave(lock, flags) +#define ep_unlock_irqrestore(lock, flags) \ + read_unlock_irqrestore(lock, flags) + +#endif /* CONFIG_PREEMPT_RT */ + /* * LOCKING: * There are three level of locking required by epoll : * * 1) epnested_mutex (mutex) * 2) ep->mtx (mutex) - * 3) ep->lock (rwlock) + * 3) ep->lock (rwlock or spinlock, see above) * * The acquire order is the one listed above, from 1 to 3. * We need a rwlock (ep->lock) because we manipulate objects @@ -194,7 +226,7 @@ struct eventpoll { struct list_head rdllist; /* Lock which protects rdllist and ovflist */ - rwlock_t lock; + ep_lock_t lock; /* RB tree root used to store monitored fd structs */ struct rb_root_cached rbr; @@ -628,10 +660,10 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist) * in a lockless way. */ lockdep_assert_irqs_enabled(); - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); list_splice_init(&ep->rdllist, txlist); WRITE_ONCE(ep->ovflist, NULL); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } static void ep_done_scan(struct eventpoll *ep, @@ -639,7 +671,7 @@ static void ep_done_scan(struct eventpoll *ep, { struct epitem *epi, *nepi; - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* * During the time we spent inside the "sproc" callback, some * other events might have been queued by the poll callback. @@ -680,7 +712,7 @@ static void ep_done_scan(struct eventpoll *ep, wake_up(&ep->wq); } - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } static void epi_rcu_free(struct rcu_head *head) @@ -777,10 +809,10 @@ static bool ep_remove_epi(struct eventpoll *ep, struct epitem *epi) rb_erase_cached(&epi->rbn, &ep->rbr); - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); if (ep_is_linked(epi)) list_del_init(&epi->rdllink); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); wakeup_source_unregister(ep_wakeup_source(epi)); /* @@ -1040,7 +1072,7 @@ static int ep_alloc(struct eventpoll **pep) return -ENOMEM; mutex_init(&ep->mtx); - rwlock_init(&ep->lock); + ep_lock_init(&ep->lock); init_waitqueue_head(&ep->wq); init_waitqueue_head(&ep->poll_wait); INIT_LIST_HEAD(&ep->rdllist); @@ -1127,6 +1159,7 @@ struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, } #endif /* CONFIG_KCMP */ +#ifndef CONFIG_PREEMPT_RT /* * Adds a new entry to the tail of the list in a lockless way, i.e. * multiple CPUs are allowed to call this function concurrently. @@ -1204,16 +1237,17 @@ static inline bool chain_epi_lockless(struct epitem *epi) return true; } +#endif /* !CONFIG_PREEMPT_RT */ /* * This is the callback that is passed to the wait queue wakeup * mechanism. It is called by the stored file descriptors when they * have events to report. * - * This callback takes a read lock in order not to contend with concurrent - * events from another file descriptor, thus all modifications to ->rdllist - * or ->ovflist are lockless. Read lock is paired with the write lock from - * ep_start/done_scan(), which stops all list modifications and guarantees - * that lists state is seen correctly. + * Under !CONFIG_PREEMPT_RT this callback takes a read lock in order not + * to contend with concurrent events from another file descriptor, thus + * all modifications to ->rdllist or ->ovflist are lockless. Read lock is + * paired with the write lock from ep_start/done_scan(), which stops all + * list modifications and guarantees that lists state is seen correctly. * * Another thing worth to mention is that ep_poll_callback() can be called * concurrently for the same @epi from different CPUs if poll table was inited @@ -1231,7 +1265,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v unsigned long flags; int ewake = 0; - read_lock_irqsave(&ep->lock, flags); + ep_lock_irqsave(&ep->lock, flags); ep_set_busy_poll_napi_id(epi); @@ -1260,12 +1294,25 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v * chained in ep->ovflist and requeued later on. */ if (READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR) { +#ifdef CONFIG_PREEMPT_RT + if (epi->next == EP_UNACTIVE_PTR) { + epi->next = READ_ONCE(ep->ovflist); + WRITE_ONCE(ep->ovflist, epi); + ep_pm_stay_awake_rcu(epi); + } +#else if (chain_epi_lockless(epi)) ep_pm_stay_awake_rcu(epi); +#endif } else if (!ep_is_linked(epi)) { /* In the usual case, add event to ready list. */ +#ifdef CONFIG_PREEMPT_RT + list_add_tail(&epi->rdllink, &ep->rdllist); + ep_pm_stay_awake_rcu(epi); +#else if (list_add_tail_lockless(&epi->rdllink, &ep->rdllist)) ep_pm_stay_awake_rcu(epi); +#endif } /* @@ -1298,7 +1345,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v pwake++; out_unlock: - read_unlock_irqrestore(&ep->lock, flags); + ep_unlock_irqrestore(&ep->lock, flags); /* We have to call this outside the lock */ if (pwake) @@ -1633,7 +1680,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, } /* We have to drop the new item inside our item list to keep track of it */ - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* record NAPI ID of new item if present */ ep_set_busy_poll_napi_id(epi); @@ -1650,7 +1697,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, pwake++; } - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); /* We have to call this outside the lock */ if (pwake) @@ -1714,7 +1761,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, * list, push it inside. */ if (ep_item_poll(epi, &pt, 1)) { - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); if (!ep_is_linked(epi)) { list_add_tail(&epi->rdllink, &ep->rdllist); ep_pm_stay_awake(epi); @@ -1725,7 +1772,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, if (waitqueue_active(&ep->poll_wait)) pwake++; } - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } /* We have to call this outside the lock */ @@ -1958,7 +2005,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, init_wait(&wait); wait.func = ep_autoremove_wake_function; - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* * Barrierless variant, waitqueue_active() is called under * the same lock on wakeup ep_poll_callback() side, so it @@ -1977,7 +2024,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (!eavail) __add_wait_queue_exclusive(&ep->wq, &wait); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); if (!eavail) timed_out = !schedule_hrtimeout_range(to, slack, @@ -1992,7 +2039,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, eavail = 1; if (!list_empty_careful(&wait.entry)) { - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* * If the thread timed out and is not on the wait queue, * it means that the thread was woken up after its @@ -2003,7 +2050,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timed_out) eavail = list_empty(&wait.entry); __remove_wait_queue(&ep->wq, &wait); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } } } -- 2.52.0

1 0

[PATCH OLK-6.6 1/2] Revert "eventpoll: Replace rwlock with spinlock"
by Ran Hongyun 29 Sep '26

29 Sep '26

hulk inclusion category: bugfix bugzilla: https://atomgit.com/openeuler/kernel/issues/10065 -------------------------------- This reverts commit d26ee5945bbcd8be44bf38853afe5d0b4e2e1e31. Commit d26ee5945bbc ("eventpoll: Replace rwlock with spinlock") replaced rwlock with spinlock to fix priority inversion. However, in eventpoll, the issue only affects RT. To avoid the performance regression on non-RT systems, revert this change in preparation for adding macro isolation to keep using rwlock on non-RT systems. Fixes: d26ee5945bbc ("eventpoll: Replace rwlock with spinlock") Signed-off-by: Ran Hongyun <ranhongyun1(a)huawei.com> --- fs/eventpoll.c | 139 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 113 insertions(+), 26 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 2cfee7d7b50c..ba719222a2da 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -45,10 +45,10 @@ * * 1) epnested_mutex (mutex) * 2) ep->mtx (mutex) - * 3) ep->lock (spinlock) + * 3) ep->lock (rwlock) * * The acquire order is the one listed above, from 1 to 3. - * We need a spinlock (ep->lock) because we manipulate objects + * We need a rwlock (ep->lock) because we manipulate objects * from inside the poll callback, that might be triggered from * a wake_up() that in turn might be called from IRQ context. * So we can't sleep inside the poll callback and hence we need @@ -194,7 +194,7 @@ struct eventpoll { struct list_head rdllist; /* Lock which protects rdllist and ovflist */ - spinlock_t lock; + rwlock_t lock; /* RB tree root used to store monitored fd structs */ struct rb_root_cached rbr; @@ -628,10 +628,10 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist) * in a lockless way. */ lockdep_assert_irqs_enabled(); - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); list_splice_init(&ep->rdllist, txlist); WRITE_ONCE(ep->ovflist, NULL); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } static void ep_done_scan(struct eventpoll *ep, @@ -639,7 +639,7 @@ static void ep_done_scan(struct eventpoll *ep, { struct epitem *epi, *nepi; - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* * During the time we spent inside the "sproc" callback, some * other events might have been queued by the poll callback. @@ -680,7 +680,7 @@ static void ep_done_scan(struct eventpoll *ep, wake_up(&ep->wq); } - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } static void epi_rcu_free(struct rcu_head *head) @@ -777,10 +777,10 @@ static bool ep_remove_epi(struct eventpoll *ep, struct epitem *epi) rb_erase_cached(&epi->rbn, &ep->rbr); - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); if (ep_is_linked(epi)) list_del_init(&epi->rdllink); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); wakeup_source_unregister(ep_wakeup_source(epi)); /* @@ -1040,7 +1040,7 @@ static int ep_alloc(struct eventpoll **pep) return -ENOMEM; mutex_init(&ep->mtx); - spin_lock_init(&ep->lock); + rwlock_init(&ep->lock); init_waitqueue_head(&ep->wq); init_waitqueue_head(&ep->poll_wait); INIT_LIST_HEAD(&ep->rdllist); @@ -1127,10 +1127,100 @@ struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, } #endif /* CONFIG_KCMP */ +/* + * Adds a new entry to the tail of the list in a lockless way, i.e. + * multiple CPUs are allowed to call this function concurrently. + * + * Beware: it is necessary to prevent any other modifications of the + * existing list until all changes are completed, in other words + * concurrent list_add_tail_lockless() calls should be protected + * with a read lock, where write lock acts as a barrier which + * makes sure all list_add_tail_lockless() calls are fully + * completed. + * + * Also an element can be locklessly added to the list only in one + * direction i.e. either to the tail or to the head, otherwise + * concurrent access will corrupt the list. + * + * Return: %false if element has been already added to the list, %true + * otherwise. + */ +static inline bool list_add_tail_lockless(struct list_head *new, + struct list_head *head) +{ + struct list_head *prev; + + /* + * This is simple 'new->next = head' operation, but cmpxchg() + * is used in order to detect that same element has been just + * added to the list from another CPU: the winner observes + * new->next == new. + */ + if (!try_cmpxchg(&new->next, &new, head)) + return false; + + /* + * Initially ->next of a new element must be updated with the head + * (we are inserting to the tail) and only then pointers are atomically + * exchanged. XCHG guarantees memory ordering, thus ->next should be + * updated before pointers are actually swapped and pointers are + * swapped before prev->next is updated. + */ + + prev = xchg(&head->prev, new); + + /* + * It is safe to modify prev->next and new->prev, because a new element + * is added only to the tail and new->next is updated before XCHG. + */ + + prev->next = new; + new->prev = prev; + + return true; +} + +/* + * Chains a new epi entry to the tail of the ep->ovflist in a lockless way, + * i.e. multiple CPUs are allowed to call this function concurrently. + * + * Return: %false if epi element has been already chained, %true otherwise. + */ +static inline bool chain_epi_lockless(struct epitem *epi) +{ + struct eventpoll *ep = epi->ep; + + /* Fast preliminary check */ + if (epi->next != EP_UNACTIVE_PTR) + return false; + + /* Check that the same epi has not been just chained from another CPU */ + if (cmpxchg(&epi->next, EP_UNACTIVE_PTR, NULL) != EP_UNACTIVE_PTR) + return false; + + /* Atomically exchange tail */ + epi->next = xchg(&ep->ovflist, epi); + + return true; +} + /* * This is the callback that is passed to the wait queue wakeup * mechanism. It is called by the stored file descriptors when they * have events to report. + * + * This callback takes a read lock in order not to contend with concurrent + * events from another file descriptor, thus all modifications to ->rdllist + * or ->ovflist are lockless. Read lock is paired with the write lock from + * ep_start/done_scan(), which stops all list modifications and guarantees + * that lists state is seen correctly. + * + * Another thing worth to mention is that ep_poll_callback() can be called + * concurrently for the same @epi from different CPUs if poll table was inited + * with several wait queues entries. Plural wakeup from different CPUs of a + * single wait queue is serialized by wq.lock, but the case when multiple wait + * queues are used should be detected accordingly. This is detected using + * cmpxchg() operation. */ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, void *key) { @@ -1141,7 +1231,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v unsigned long flags; int ewake = 0; - spin_lock_irqsave(&ep->lock, flags); + read_lock_irqsave(&ep->lock, flags); ep_set_busy_poll_napi_id(epi); @@ -1170,15 +1260,12 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v * chained in ep->ovflist and requeued later on. */ if (READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR) { - if (epi->next == EP_UNACTIVE_PTR) { - epi->next = READ_ONCE(ep->ovflist); - WRITE_ONCE(ep->ovflist, epi); + if (chain_epi_lockless(epi)) ep_pm_stay_awake_rcu(epi); - } } else if (!ep_is_linked(epi)) { /* In the usual case, add event to ready list. */ - list_add_tail(&epi->rdllink, &ep->rdllist); - ep_pm_stay_awake_rcu(epi); + if (list_add_tail_lockless(&epi->rdllink, &ep->rdllist)) + ep_pm_stay_awake_rcu(epi); } /* @@ -1211,7 +1298,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v pwake++; out_unlock: - spin_unlock_irqrestore(&ep->lock, flags); + read_unlock_irqrestore(&ep->lock, flags); /* We have to call this outside the lock */ if (pwake) @@ -1546,7 +1633,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, } /* We have to drop the new item inside our item list to keep track of it */ - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* record NAPI ID of new item if present */ ep_set_busy_poll_napi_id(epi); @@ -1563,7 +1650,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, pwake++; } - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); /* We have to call this outside the lock */ if (pwake) @@ -1627,7 +1714,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, * list, push it inside. */ if (ep_item_poll(epi, &pt, 1)) { - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); if (!ep_is_linked(epi)) { list_add_tail(&epi->rdllink, &ep->rdllist); ep_pm_stay_awake(epi); @@ -1638,7 +1725,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, if (waitqueue_active(&ep->poll_wait)) pwake++; } - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } /* We have to call this outside the lock */ @@ -1871,7 +1958,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, init_wait(&wait); wait.func = ep_autoremove_wake_function; - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* * Barrierless variant, waitqueue_active() is called under * the same lock on wakeup ep_poll_callback() side, so it @@ -1890,7 +1977,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (!eavail) __add_wait_queue_exclusive(&ep->wq, &wait); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); if (!eavail) timed_out = !schedule_hrtimeout_range(to, slack, @@ -1905,7 +1992,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, eavail = 1; if (!list_empty_careful(&wait.entry)) { - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* * If the thread timed out and is not on the wait queue, * it means that the thread was woken up after its @@ -1916,7 +2003,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timed_out) eavail = list_empty(&wait.entry); __remove_wait_queue(&ep->wq, &wait); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } } } -- 2.52.0

1 0

[PATCH OLK-6.6 2/2] eventpoll: use spinlock under CONFIG_PREEMPT_RT, rwlock otherwise
by Ran Hongyun 29 Sep '26

29 Sep '26

hulk inclusion category: bugfix bugzilla: NA -------------------------------- Introduce ep_lock_* wrapper macros that select spinlock_t under CONFIG_PREEMPT_RT and rwlock_t otherwise. The rwlock read-side cannot be priority-boosted on RT (rwbase_rt readers are not rt_mutex owners), causing unbounded priority inversion; spinlock fixes this. On non-RT kernels the rwlock is preserved for producer concurrency. The lockless helpers list_add_tail_lockless() and chain_epi_lockless() are compiled out under CONFIG_PREEMPT_RT since the spinlock serializes all producers. Fixes: a218cc491420 ("epoll: use rwlock in order to reduce ep_poll_callback() contention") Signed-off-by: Ran Hongyun <ranhongyun1(a)huawei.com> --- fs/eventpoll.c | 95 +++++++++++++++++++++++++++++++++++++------------- 1 file changed, 71 insertions(+), 24 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ba719222a2da..523e69de1e32 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -39,13 +39,45 @@ #include <linux/rculist.h> #include <net/busy_poll.h> +/* + * Under CONFIG_PREEMPT_RT the ep->lock is a spinlock because the rwlock + * read-side cannot be priority-boosted (rwbase_rt readers are not + * rt_mutex owners), which causes unbounded priority inversion when a + * high-priority consumer (writer) waits on low-priority producers + * (readers). On non-RT kernels the rwlock is preserved so that multiple + * producers can run concurrently, which scales better with large event + * sources. + */ + +#ifdef CONFIG_PREEMPT_RT + +#define ep_lock_t spinlock_t +#define ep_lock_init(lock) spin_lock_init(lock) +#define ep_lock_irq(lock) spin_lock_irq(lock) +#define ep_unlock_irq(lock) spin_unlock_irq(lock) +#define ep_lock_irqsave(lock, flags) spin_lock_irqsave(lock, flags) +#define ep_unlock_irqrestore(lock, flags) \ + spin_unlock_irqrestore(lock, flags) + +#else /* !CONFIG_PREEMPT_RT */ + +#define ep_lock_t rwlock_t +#define ep_lock_init(lock) rwlock_init(lock) +#define ep_lock_irq(lock) write_lock_irq(lock) +#define ep_unlock_irq(lock) write_unlock_irq(lock) +#define ep_lock_irqsave(lock, flags) read_lock_irqsave(lock, flags) +#define ep_unlock_irqrestore(lock, flags) \ + read_unlock_irqrestore(lock, flags) + +#endif /* CONFIG_PREEMPT_RT */ + /* * LOCKING: * There are three level of locking required by epoll : * * 1) epnested_mutex (mutex) * 2) ep->mtx (mutex) - * 3) ep->lock (rwlock) + * 3) ep->lock (rwlock or spinlock, see above) * * The acquire order is the one listed above, from 1 to 3. * We need a rwlock (ep->lock) because we manipulate objects @@ -194,7 +226,7 @@ struct eventpoll { struct list_head rdllist; /* Lock which protects rdllist and ovflist */ - rwlock_t lock; + ep_lock_t lock; /* RB tree root used to store monitored fd structs */ struct rb_root_cached rbr; @@ -628,10 +660,10 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist) * in a lockless way. */ lockdep_assert_irqs_enabled(); - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); list_splice_init(&ep->rdllist, txlist); WRITE_ONCE(ep->ovflist, NULL); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } static void ep_done_scan(struct eventpoll *ep, @@ -639,7 +671,7 @@ static void ep_done_scan(struct eventpoll *ep, { struct epitem *epi, *nepi; - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* * During the time we spent inside the "sproc" callback, some * other events might have been queued by the poll callback. @@ -680,7 +712,7 @@ static void ep_done_scan(struct eventpoll *ep, wake_up(&ep->wq); } - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } static void epi_rcu_free(struct rcu_head *head) @@ -777,10 +809,10 @@ static bool ep_remove_epi(struct eventpoll *ep, struct epitem *epi) rb_erase_cached(&epi->rbn, &ep->rbr); - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); if (ep_is_linked(epi)) list_del_init(&epi->rdllink); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); wakeup_source_unregister(ep_wakeup_source(epi)); /* @@ -1040,7 +1072,7 @@ static int ep_alloc(struct eventpoll **pep) return -ENOMEM; mutex_init(&ep->mtx); - rwlock_init(&ep->lock); + ep_lock_init(&ep->lock); init_waitqueue_head(&ep->wq); init_waitqueue_head(&ep->poll_wait); INIT_LIST_HEAD(&ep->rdllist); @@ -1127,6 +1159,7 @@ struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, } #endif /* CONFIG_KCMP */ +#ifndef CONFIG_PREEMPT_RT /* * Adds a new entry to the tail of the list in a lockless way, i.e. * multiple CPUs are allowed to call this function concurrently. @@ -1204,16 +1237,17 @@ static inline bool chain_epi_lockless(struct epitem *epi) return true; } +#endif /* !CONFIG_PREEMPT_RT */ /* * This is the callback that is passed to the wait queue wakeup * mechanism. It is called by the stored file descriptors when they * have events to report. * - * This callback takes a read lock in order not to contend with concurrent - * events from another file descriptor, thus all modifications to ->rdllist - * or ->ovflist are lockless. Read lock is paired with the write lock from - * ep_start/done_scan(), which stops all list modifications and guarantees - * that lists state is seen correctly. + * Under !CONFIG_PREEMPT_RT this callback takes a read lock in order not + * to contend with concurrent events from another file descriptor, thus + * all modifications to ->rdllist or ->ovflist are lockless. Read lock is + * paired with the write lock from ep_start/done_scan(), which stops all + * list modifications and guarantees that lists state is seen correctly. * * Another thing worth to mention is that ep_poll_callback() can be called * concurrently for the same @epi from different CPUs if poll table was inited @@ -1231,7 +1265,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v unsigned long flags; int ewake = 0; - read_lock_irqsave(&ep->lock, flags); + ep_lock_irqsave(&ep->lock, flags); ep_set_busy_poll_napi_id(epi); @@ -1260,12 +1294,25 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v * chained in ep->ovflist and requeued later on. */ if (READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR) { +#ifdef CONFIG_PREEMPT_RT + if (epi->next == EP_UNACTIVE_PTR) { + epi->next = READ_ONCE(ep->ovflist); + WRITE_ONCE(ep->ovflist, epi); + ep_pm_stay_awake_rcu(epi); + } +#else if (chain_epi_lockless(epi)) ep_pm_stay_awake_rcu(epi); +#endif } else if (!ep_is_linked(epi)) { /* In the usual case, add event to ready list. */ +#ifdef CONFIG_PREEMPT_RT + list_add_tail(&epi->rdllink, &ep->rdllist); + ep_pm_stay_awake_rcu(epi); +#else if (list_add_tail_lockless(&epi->rdllink, &ep->rdllist)) ep_pm_stay_awake_rcu(epi); +#endif } /* @@ -1298,7 +1345,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v pwake++; out_unlock: - read_unlock_irqrestore(&ep->lock, flags); + ep_unlock_irqrestore(&ep->lock, flags); /* We have to call this outside the lock */ if (pwake) @@ -1633,7 +1680,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, } /* We have to drop the new item inside our item list to keep track of it */ - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* record NAPI ID of new item if present */ ep_set_busy_poll_napi_id(epi); @@ -1650,7 +1697,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, pwake++; } - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); /* We have to call this outside the lock */ if (pwake) @@ -1714,7 +1761,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, * list, push it inside. */ if (ep_item_poll(epi, &pt, 1)) { - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); if (!ep_is_linked(epi)) { list_add_tail(&epi->rdllink, &ep->rdllist); ep_pm_stay_awake(epi); @@ -1725,7 +1772,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, if (waitqueue_active(&ep->poll_wait)) pwake++; } - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } /* We have to call this outside the lock */ @@ -1958,7 +2005,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, init_wait(&wait); wait.func = ep_autoremove_wake_function; - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* * Barrierless variant, waitqueue_active() is called under * the same lock on wakeup ep_poll_callback() side, so it @@ -1977,7 +2024,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (!eavail) __add_wait_queue_exclusive(&ep->wq, &wait); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); if (!eavail) timed_out = !schedule_hrtimeout_range(to, slack, @@ -1992,7 +2039,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, eavail = 1; if (!list_empty_careful(&wait.entry)) { - write_lock_irq(&ep->lock); + ep_lock_irq(&ep->lock); /* * If the thread timed out and is not on the wait queue, * it means that the thread was woken up after its @@ -2003,7 +2050,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timed_out) eavail = list_empty(&wait.entry); __remove_wait_queue(&ep->wq, &wait); - write_unlock_irq(&ep->lock); + ep_unlock_irq(&ep->lock); } } } -- 2.52.0

1 0

[PATCH OLK-6.6 1/2] Revert "eventpoll: Replace rwlock with spinlock"
by Ran Hongyun 29 Sep '26

29 Sep '26

hulk inclusion category: bugfix bugzilla: NA -------------------------------- This reverts commit d26ee5945bbcd8be44bf38853afe5d0b4e2e1e31. Commit d26ee5945bbc ("eventpoll: Replace rwlock with spinlock") replaced rwlock with spinlock to fix priority inversion. However, in eventpoll, the issue only affects RT. To avoid the performance regression on non-RT systems, revert this change in preparation for adding macro isolation to keep using rwlock on non-RT systems. Fixes: d26ee5945bbc ("eventpoll: Replace rwlock with spinlock") Signed-off-by: Ran Hongyun <ranhongyun1(a)huawei.com> --- fs/eventpoll.c | 139 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 113 insertions(+), 26 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 2cfee7d7b50c..ba719222a2da 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -45,10 +45,10 @@ * * 1) epnested_mutex (mutex) * 2) ep->mtx (mutex) - * 3) ep->lock (spinlock) + * 3) ep->lock (rwlock) * * The acquire order is the one listed above, from 1 to 3. - * We need a spinlock (ep->lock) because we manipulate objects + * We need a rwlock (ep->lock) because we manipulate objects * from inside the poll callback, that might be triggered from * a wake_up() that in turn might be called from IRQ context. * So we can't sleep inside the poll callback and hence we need @@ -194,7 +194,7 @@ struct eventpoll { struct list_head rdllist; /* Lock which protects rdllist and ovflist */ - spinlock_t lock; + rwlock_t lock; /* RB tree root used to store monitored fd structs */ struct rb_root_cached rbr; @@ -628,10 +628,10 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist) * in a lockless way. */ lockdep_assert_irqs_enabled(); - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); list_splice_init(&ep->rdllist, txlist); WRITE_ONCE(ep->ovflist, NULL); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } static void ep_done_scan(struct eventpoll *ep, @@ -639,7 +639,7 @@ static void ep_done_scan(struct eventpoll *ep, { struct epitem *epi, *nepi; - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* * During the time we spent inside the "sproc" callback, some * other events might have been queued by the poll callback. @@ -680,7 +680,7 @@ static void ep_done_scan(struct eventpoll *ep, wake_up(&ep->wq); } - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } static void epi_rcu_free(struct rcu_head *head) @@ -777,10 +777,10 @@ static bool ep_remove_epi(struct eventpoll *ep, struct epitem *epi) rb_erase_cached(&epi->rbn, &ep->rbr); - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); if (ep_is_linked(epi)) list_del_init(&epi->rdllink); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); wakeup_source_unregister(ep_wakeup_source(epi)); /* @@ -1040,7 +1040,7 @@ static int ep_alloc(struct eventpoll **pep) return -ENOMEM; mutex_init(&ep->mtx); - spin_lock_init(&ep->lock); + rwlock_init(&ep->lock); init_waitqueue_head(&ep->wq); init_waitqueue_head(&ep->poll_wait); INIT_LIST_HEAD(&ep->rdllist); @@ -1127,10 +1127,100 @@ struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, } #endif /* CONFIG_KCMP */ +/* + * Adds a new entry to the tail of the list in a lockless way, i.e. + * multiple CPUs are allowed to call this function concurrently. + * + * Beware: it is necessary to prevent any other modifications of the + * existing list until all changes are completed, in other words + * concurrent list_add_tail_lockless() calls should be protected + * with a read lock, where write lock acts as a barrier which + * makes sure all list_add_tail_lockless() calls are fully + * completed. + * + * Also an element can be locklessly added to the list only in one + * direction i.e. either to the tail or to the head, otherwise + * concurrent access will corrupt the list. + * + * Return: %false if element has been already added to the list, %true + * otherwise. + */ +static inline bool list_add_tail_lockless(struct list_head *new, + struct list_head *head) +{ + struct list_head *prev; + + /* + * This is simple 'new->next = head' operation, but cmpxchg() + * is used in order to detect that same element has been just + * added to the list from another CPU: the winner observes + * new->next == new. + */ + if (!try_cmpxchg(&new->next, &new, head)) + return false; + + /* + * Initially ->next of a new element must be updated with the head + * (we are inserting to the tail) and only then pointers are atomically + * exchanged. XCHG guarantees memory ordering, thus ->next should be + * updated before pointers are actually swapped and pointers are + * swapped before prev->next is updated. + */ + + prev = xchg(&head->prev, new); + + /* + * It is safe to modify prev->next and new->prev, because a new element + * is added only to the tail and new->next is updated before XCHG. + */ + + prev->next = new; + new->prev = prev; + + return true; +} + +/* + * Chains a new epi entry to the tail of the ep->ovflist in a lockless way, + * i.e. multiple CPUs are allowed to call this function concurrently. + * + * Return: %false if epi element has been already chained, %true otherwise. + */ +static inline bool chain_epi_lockless(struct epitem *epi) +{ + struct eventpoll *ep = epi->ep; + + /* Fast preliminary check */ + if (epi->next != EP_UNACTIVE_PTR) + return false; + + /* Check that the same epi has not been just chained from another CPU */ + if (cmpxchg(&epi->next, EP_UNACTIVE_PTR, NULL) != EP_UNACTIVE_PTR) + return false; + + /* Atomically exchange tail */ + epi->next = xchg(&ep->ovflist, epi); + + return true; +} + /* * This is the callback that is passed to the wait queue wakeup * mechanism. It is called by the stored file descriptors when they * have events to report. + * + * This callback takes a read lock in order not to contend with concurrent + * events from another file descriptor, thus all modifications to ->rdllist + * or ->ovflist are lockless. Read lock is paired with the write lock from + * ep_start/done_scan(), which stops all list modifications and guarantees + * that lists state is seen correctly. + * + * Another thing worth to mention is that ep_poll_callback() can be called + * concurrently for the same @epi from different CPUs if poll table was inited + * with several wait queues entries. Plural wakeup from different CPUs of a + * single wait queue is serialized by wq.lock, but the case when multiple wait + * queues are used should be detected accordingly. This is detected using + * cmpxchg() operation. */ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, void *key) { @@ -1141,7 +1231,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v unsigned long flags; int ewake = 0; - spin_lock_irqsave(&ep->lock, flags); + read_lock_irqsave(&ep->lock, flags); ep_set_busy_poll_napi_id(epi); @@ -1170,15 +1260,12 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v * chained in ep->ovflist and requeued later on. */ if (READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR) { - if (epi->next == EP_UNACTIVE_PTR) { - epi->next = READ_ONCE(ep->ovflist); - WRITE_ONCE(ep->ovflist, epi); + if (chain_epi_lockless(epi)) ep_pm_stay_awake_rcu(epi); - } } else if (!ep_is_linked(epi)) { /* In the usual case, add event to ready list. */ - list_add_tail(&epi->rdllink, &ep->rdllist); - ep_pm_stay_awake_rcu(epi); + if (list_add_tail_lockless(&epi->rdllink, &ep->rdllist)) + ep_pm_stay_awake_rcu(epi); } /* @@ -1211,7 +1298,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v pwake++; out_unlock: - spin_unlock_irqrestore(&ep->lock, flags); + read_unlock_irqrestore(&ep->lock, flags); /* We have to call this outside the lock */ if (pwake) @@ -1546,7 +1633,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, } /* We have to drop the new item inside our item list to keep track of it */ - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* record NAPI ID of new item if present */ ep_set_busy_poll_napi_id(epi); @@ -1563,7 +1650,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, pwake++; } - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); /* We have to call this outside the lock */ if (pwake) @@ -1627,7 +1714,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, * list, push it inside. */ if (ep_item_poll(epi, &pt, 1)) { - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); if (!ep_is_linked(epi)) { list_add_tail(&epi->rdllink, &ep->rdllist); ep_pm_stay_awake(epi); @@ -1638,7 +1725,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, if (waitqueue_active(&ep->poll_wait)) pwake++; } - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } /* We have to call this outside the lock */ @@ -1871,7 +1958,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, init_wait(&wait); wait.func = ep_autoremove_wake_function; - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* * Barrierless variant, waitqueue_active() is called under * the same lock on wakeup ep_poll_callback() side, so it @@ -1890,7 +1977,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (!eavail) __add_wait_queue_exclusive(&ep->wq, &wait); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); if (!eavail) timed_out = !schedule_hrtimeout_range(to, slack, @@ -1905,7 +1992,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, eavail = 1; if (!list_empty_careful(&wait.entry)) { - spin_lock_irq(&ep->lock); + write_lock_irq(&ep->lock); /* * If the thread timed out and is not on the wait queue, * it means that the thread was woken up after its @@ -1916,7 +2003,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timed_out) eavail = list_empty(&wait.entry); __remove_wait_queue(&ep->wq, &wait); - spin_unlock_irq(&ep->lock); + write_unlock_irq(&ep->lock); } } } -- 2.52.0

1 0