From: Max Kellermann <max.kellermann@ionos.com> mainline inclusion from mainline-v6.15-rc1 commit 486ba4d84d62e92716cd395c4b1612b8ce70a257 category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IC6ES1 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... -------------------------------- Previously, the `hash` variable was initialized with `-1` and only updated by io_get_next_work() if the current work was hashed. Commit 60cf46ae6054 ("io-wq: hash dependent work") changed this to always call io_get_work_hash() even if the work was not hashed. This caused the `hash != -1U` check to always be true, adding some overhead for the `hash->wait` code. This patch fixes the regression by checking the `IO_WQ_WORK_HASHED` flag. Perf diff for a flood of `IORING_OP_NOP` with `IOSQE_ASYNC`: 38.55% -1.57% [kernel.kallsyms] [k] queued_spin_lock_slowpath 6.86% -0.72% [kernel.kallsyms] [k] io_worker_handle_work 0.10% +0.67% [kernel.kallsyms] [k] put_prev_entity 1.96% +0.59% [kernel.kallsyms] [k] io_nop_prep 3.31% -0.51% [kernel.kallsyms] [k] try_to_wake_up 7.18% -0.47% [kernel.kallsyms] [k] io_wq_free_work Fixes: 60cf46ae6054 ("io-wq: hash dependent work") Cc: Pavel Begunkov <asml.silence@gmail.com> Signed-off-by: Max Kellermann <max.kellermann@ionos.com> Link: https://lore.kernel.org/r/20250128133927.3989681-6-max.kellermann@ionos.com Signed-off-by: Jens Axboe <axboe@kernel.dk> Conflicts: io_uring/io-wq.c [Commit 6ee78354eaa6 ("io_uring/io-wq: cache work->flags in variable") replace io_get_work_hash with __io_get_work_hash.] Signed-off-by: Li Lingfeng <lilingfeng3@huawei.com> --- io_uring/io-wq.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 14d70520ed9e..b14af1b6f9dc 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -577,7 +577,9 @@ static void io_worker_handle_work(struct io_worker *worker) /* handle a whole dependent link */ do { struct io_wq_work *next_hashed, *linked; - unsigned int hash = io_get_work_hash(work); + unsigned int hash = io_wq_is_hashed(work) + ? io_get_work_hash(work) + : -1U; next_hashed = wq_next_work(work); -- 2.31.1