From: Yin Fengwei fengwei.yin@intel.com
mainline inclusion from mainline-v6.5-rc1 commit deedad80f660af8199ea3b3f70939f2d226b9154 category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/I9NU9F
Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
--------------------------------
free_transhuge_page() acquires split queue lock then check whether the THP was added to deferred list or not. It brings high deferred queue lock contention.
It's safe to check whether the THP is in deferred list or not without holding the deferred queue lock in free_transhuge_page() because when code hit free_transhuge_page(), there is no one tries to add the folio to _deferred_list.
Running page_fault1 of will-it-scale + order 2 folio for anonymous mapping with 96 processes on an Ice Lake 48C/96T test box, we could see the 61% split_queue_lock contention: - 63.02% 0.01% page_fault1_pro [kernel.kallsyms] [k] free_transhuge_page - 63.01% free_transhuge_page + 62.91% _raw_spin_lock_irqsave
With this patch applied, the split_queue_lock contention is less than 1%.
Link: https://lkml.kernel.org/r/20230429082759.1600796-2-fengwei.yin@intel.com Signed-off-by: Yin Fengwei fengwei.yin@intel.com Acked-by: Kirill A. Shutemov kirill.shutemov@linux.intel.com Reviewed-by: "Huang, Ying" ying.huang@intel.com Cc: Matthew Wilcox willy@infradead.org Cc: Ryan Roberts ryan.roberts@arm.com Cc: Yu Zhao yuzhao@google.com Signed-off-by: Andrew Morton akpm@linux-foundation.org
Conflicts: mm/huge_memory.c [Ma Wupeng: mainline use folio & refactor on ds_queue lead to conflicts] Signed-off-by: Ma Wupeng mawupeng1@huawei.com --- mm/huge_memory.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8b7086cfd1ed..b2e39b947126 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2907,13 +2907,20 @@ void free_transhuge_page(struct page *page) struct deferred_split ds_queue; unsigned long flags;
+ /* + * At this point, there is no one trying to add the folio to + * deferred_list. If folio is not in deferred_list, it's safe + * to check without acquiring the split_queue_lock. + */ get_deferred_split_queue(page, &ds_queue); - spin_lock_irqsave(ds_queue.split_queue_lock, flags); - if (!list_empty(page_deferred_list(page))) { - (*ds_queue.split_queue_len)--; - list_del(page_deferred_list(page)); + if (data_race(!list_empty(page_deferred_list(page)))) { + spin_lock_irqsave(ds_queue.split_queue_lock, flags); + if (!list_empty(page_deferred_list(page))) { + (*ds_queue.split_queue_len)--; + list_del(page_deferred_list(page)); + } + spin_unlock_irqrestore(ds_queue.split_queue_lock, flags); } - spin_unlock_irqrestore(ds_queue.split_queue_lock, flags); free_compound_page(page); }