From: Vlastimil Babka vbabka@suse.cz
mainline inclusion from mainline-v6.12-rc6 commit d4148aeab412432bf928f311eca8a2ba52bb05df category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IB1RRY
Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
--------------------------------
Since commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") a mmap() of anonymous memory without a specific address hint and of at least PMD_SIZE will be aligned to PMD so that it can benefit from a THP backing page.
However this change has been shown to regress some workloads significantly. [1] reports regressions in various spec benchmarks, with up to 600% slowdown of the cactusBSSN benchmark on some platforms. The benchmark seems to create many mappings of 4632kB, which would have merged to a large THP-backed area before commit efa7df3e3bb5 and now they are fragmented to multiple areas each aligned to PMD boundary with gaps between. The regression then seems to be caused mainly due to the benchmark's memory access pattern suffering from TLB or cache aliasing due to the aligned boundaries of the individual areas.
Another known regression bisected to commit efa7df3e3bb5 is darktable [2] [3] and early testing suggests this patch fixes the regression there as well.
To fix the regression but still try to benefit from THP-friendly anonymous mapping alignment, add a condition that the size of the mapping must be a multiple of PMD size instead of at least PMD size. In case of many odd-sized mapping like the cactusBSSN creates, those will stop being aligned and with gaps between, and instead naturally merge again.
Link: https://lkml.kernel.org/r/20241024151228.101841-2-vbabka@suse.cz Fixes: efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") Signed-off-by: Vlastimil Babka vbabka@suse.cz Reported-by: Michael Matz matz@suse.de Debugged-by: Gabriel Krisman Bertazi gabriel@krisman.be Closes: https://bugzilla.suse.com/show_bug.cgi?id=1229012 [1] Reported-by: Matthias Bodenbinder matthias@bodenbinder.de Closes: https://bugzilla.kernel.org/show_bug.cgi?id=219366 [2] Closes: https://lore.kernel.org/all/2050f0d4-57b0-481d-bab8-05e8d48fed0c@leemhuis.in... [3] Reviewed-by: Lorenzo Stoakes lorenzo.stoakes@oracle.com Reviewed-by: Yang Shi yang@os.amperecomputing.com Cc: Rik van Riel riel@surriel.com Cc: Jann Horn jannh@google.com Cc: Liam R. Howlett Liam.Howlett@Oracle.com Cc: Petr Tesarik ptesarik@suse.com Cc: Thorsten Leemhuis regressions@leemhuis.info Cc: stable@vger.kernel.org Signed-off-by: Andrew Morton akpm@linux-foundation.org
Conflicts: mm/mmap.c mm/huge_memory.c [Move check in thp_get_unmapped_area() to adapt with mthp. Check alignmenmt for 2M, 64K anon mapping and 64K file mapping.] Signed-off-by: Jinjiang Tu tujinjiang@huawei.com --- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c index eafe846739f4..599ecf6d265e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1115,7 +1115,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT;
- if (filp || thp_anon_mapping_align()) { + if (filp || (thp_anon_mapping_align() && IS_ALIGNED(len, PMD_SIZE))) { ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); if (ret) @@ -1125,7 +1125,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, if (mthp_mapping_align_enabled(filp)) { int order = arch_wants_exec_folio_order();
- if (order >= 0) { + if (order >= 0 && IS_ALIGNED(len, PAGE_SIZE << order)) { ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PAGE_SIZE << order); if (ret)