Jinjiang Tu (1): mm: fix kabi breakage due to struct per_cpu_pages
yangge (1): mm/page_alloc: Separate THP PCP into movable and non-movable categories
include/linux/mmzone.h | 16 +++++++++++----- mm/page_alloc.c | 9 +++++++-- 2 files changed, 18 insertions(+), 7 deletions(-)
From: yangge yangge1116@126.com
stable inclusion from stable-v6.6.37 commit c5978b996260534aca5ff1b6dcbef92c2bbea947 category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IAD6H2
Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=...
--------------------------------
commit bf14ed81f571f8dba31cd72ab2e50fbcc877cc31 upstream.
Since commit 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations") no longer differentiates the migration type of pages in THP-sized PCP list, it's possible that non-movable allocation requests may get a CMA page from the list, in some cases, it's not acceptable.
If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual machine with device passthrough will get stuck. During starting the virtual machine, it will call pin_user_pages_remote(..., FOLL_LONGTERM, ...) to pin memory. Normally if a page is present and in CMA area, pin_user_pages_remote() will migrate the page from CMA area to non-CMA area because of FOLL_LONGTERM flag. But if non-movable allocation requests return CMA memory, migrate_longterm_unpinnable_pages() will migrate a CMA page to another CMA page, which will fail to pass the check in check_and_migrate_movable_pages() and cause migration endless.
Call trace: pin_user_pages_remote --__gup_longterm_locked // endless loops in this function ----_get_user_pages_locked ----check_and_migrate_movable_pages ------migrate_longterm_unpinnable_pages --------alloc_migration_target
This problem will also have a negative impact on CMA itself. For example, when CMA is borrowed by THP, and we need to reclaim it through cma_alloc() or dma_alloc_coherent(), we must move those pages out to ensure CMA's users can retrieve that contigous memory. Currently, CMA's memory is occupied by non-movable pages, meaning we can't relocate them. As a result, cma_alloc() is more likely to fail.
To fix the problem above, we add one PCP list for THP, which will not introduce a new cacheline for struct per_cpu_pages. THP will have 2 PCP lists, one PCP list is used by MOVABLE allocation, and the other PCP list is used by UNMOVABLE allocation. MOVABLE allocation contains GPF_MOVABLE, and UNMOVABLE allocation contains GFP_UNMOVABLE and GFP_RECLAIMABLE.
Link: https://lkml.kernel.org/r/1718845190-4456-1-git-send-email-yangge1116@126.co... Fixes: 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations") Signed-off-by: yangge yangge1116@126.com Cc: Baolin Wang baolin.wang@linux.alibaba.com Cc: Barry Song 21cnbao@gmail.com Cc: Mel Gorman mgorman@techsingularity.net Cc: stable@vger.kernel.org Signed-off-by: Andrew Morton akpm@linux-foundation.org Signed-off-by: Greg Kroah-Hartman gregkh@linuxfoundation.org
Conflicts: mm/page_alloc.c include/linux/mmzone.h [Conflicts due to ab3141753341 ("mm: prepare more high-order pages to be stored on the per-cpu lists")] Signed-off-by: Jinjiang Tu tujinjiang@huawei.com --- include/linux/mmzone.h | 9 ++++----- mm/page_alloc.c | 9 +++++++-- 2 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index ec1d63cbc2e0..f76cc19b7fcd 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -678,13 +678,12 @@ enum zone_watermarks { };
/* - * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list - * for THP which will usually be GFP_MOVABLE. Even if it is another type, - * it should not contribute to serious fragmentation causing THP allocation - * failures. + * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists + * are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list + * is used by GFP_UNMOVABLE and GFP_RECLAIMABLE. */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_PCP_THP 1 +#define NR_PCP_THP 2 #define NR_PCP_ORDERS (PAGE_ALLOC_COSTLY_ORDER + 2) #else #define NR_PCP_THP 0 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c63081268df4..a88c51df703f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -528,10 +528,15 @@ static void bad_page(struct page *page, const char *reason)
static inline unsigned int order_to_pindex(int migratetype, int order) { + bool __maybe_unused movable; + #ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order > PAGE_ALLOC_COSTLY_ORDER + 1) { VM_BUG_ON(order != HPAGE_PMD_ORDER); - return NR_LOWORDER_PCP_LISTS; + + movable = migratetype == MIGRATE_MOVABLE; + + return NR_LOWORDER_PCP_LISTS + movable; } #else VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); @@ -545,7 +550,7 @@ static inline int pindex_to_order(unsigned int pindex) int order = pindex / MIGRATE_PCPTYPES;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (pindex == NR_LOWORDER_PCP_LISTS) + if (pindex >= NR_LOWORDER_PCP_LISTS) order = HPAGE_PMD_ORDER; #else VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
hulk inclusion category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IAD6H2
----------------------------------------
Since bf14ed81f571 ("mm/page_alloc: Separate THP PCP into movable and non-movable categories"), NR_PCP_LISTS increases by 1 when CONFIG_TRANSPARENT_HUGEPAGE is enabled and leads to kabi breakage of struct per_cpu_pages.
Since struct per_cpu_pages contains hole at the end, we can directly add one list without breaking KABI.
Fixes: bf14ed81f571 ("mm/page_alloc: Separate THP PCP into movable and non-movable categories") Signed-off-by: Jinjiang Tu tujinjiang@huawei.com --- include/linux/mmzone.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index f76cc19b7fcd..3cee238de7c8 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -683,14 +683,17 @@ enum zone_watermarks { * is used by GFP_UNMOVABLE and GFP_RECLAIMABLE. */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_PCP_THP 2 +#define NR_PCP_THP 1 +#define NR_PCP_THP_UNMOVABLE 1 #define NR_PCP_ORDERS (PAGE_ALLOC_COSTLY_ORDER + 2) #else #define NR_PCP_THP 0 +#define NR_PCP_THP_UNMOVABLE 0 #define NR_PCP_ORDERS (PAGE_ALLOC_COSTLY_ORDER + 1) #endif #define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * NR_PCP_ORDERS) -#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) +#define NR_PCP_OLD_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) +#define NR_PCP_LISTS (NR_PCP_OLD_LISTS + NR_PCP_THP_UNMOVABLE)
#define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost) #define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost) @@ -727,7 +730,11 @@ struct per_cpu_pages { short free_count; /* consecutive free count */
/* Lists of pages, one per migrate type stored on the pcp-lists */ +#ifndef __GENKSYMS__ struct list_head lists[NR_PCP_LISTS]; +#else + struct list_head lists[NR_PCP_OLD_LISTS]; +#endif } ____cacheline_aligned_in_smp;
struct per_cpu_zonestat {
反馈: 您发送到kernel@openeuler.org的补丁/补丁集,已成功转换为PR! PR链接地址: https://gitee.com/openeuler/kernel/pulls/14117 邮件列表地址:https://mailweb.openeuler.org/hyperkitty/list/kernel@openeuler.org/message/N...
FeedBack: The patch(es) which you have sent to kernel@openeuler.org mailing list has been converted to a pull request successfully! Pull request link: https://gitee.com/openeuler/kernel/pulls/14117 Mailing list address: https://mailweb.openeuler.org/hyperkitty/list/kernel@openeuler.org/message/N...