From: Rik van Riel <riel@surriel.com> mainline inclusion from mainline-v6.14-rc1 commit 1aa43598c03b71d945f8b12d6d579f0ad836f8f8 category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IC916E Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... ---------------------------------------------------------------------- There seem to be several categories of calls to lru_add_drain and lru_add_drain_all. The first are code paths that recently allocated, swapped in, or otherwise processed a batch of pages, and want them all on the LRU. These drain pages that were recently allocated, probably on the local CPU. A second category are code paths that are actively trying to reclaim, migrate, or offline memory. These often use lru_add_drain_all, to drain the caches on all CPUs. However, there also seem to be some other callers where we aren't really doing either. They are calling lru_add_drain(), despite operating on pages that may have been allocated long ago, and quite possibly on different CPUs. Those calls are not likely to be effective at anything but creating lock contention on the LRU locks. Remove the lru_add_drain calls in the latter category. For detailed reasoning, see [1] and [2]. Link: https://lkml.kernel.org/r/dca2824e8e88e826c6b260a831d79089b5b9c79d.camel@sur... [1] Link: https://lkml.kernel.org/r/xxfhcjaq2xxcl5adastz5omkytenq7izo2e5f4q7e3ns4z6lko... [2] Link: https://lkml.kernel.org/r/20241219153253.3da9e8aa@fangorn Signed-off-by: Rik van Riel <riel@surriel.com> Suggested-by: David Hildenbrand <david@redhat.com> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Acked-by: David Hildenbrand <david@redhat.com> Acked-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Cc: Chris Li <chrisl@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Conflicts: mm/mmap.c mm/vma.c [vma.c, unmap_region belongs to in the mainline, is not factored out. shift_arg_pages was replaced by relocate_vma_down. vms_clear_ptes was not introduced until 6.12.] Signed-off-by: Chen Ridong <chenridong@huawei.com> --- fs/exec.c | 1 - mm/memory.c | 1 - mm/mmap.c | 2 -- mm/swap_state.c | 1 - 4 files changed, 5 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index eaec57f79aa19..2ac3ef80628f0 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -714,7 +714,6 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) vma, new_start, length, false)) return -ENOMEM; - lru_add_drain(); tlb_gather_mmu(&tlb, mm); next = vma_next(&vmi); if (new_end > old_start) { diff --git a/mm/memory.c b/mm/memory.c index a25c60fdeab89..25c501c1aa713 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1929,7 +1929,6 @@ void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, struct mmu_notifier_range range; struct mmu_gather tlb; - lru_add_drain(); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, address, end); hugetlb_zap_begin(vma, &range.start, &range.end); diff --git a/mm/mmap.c b/mm/mmap.c index 4a36526d8caf8..59f410c03f2b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2407,7 +2407,6 @@ static void unmap_region(struct mm_struct *mm, struct ma_state *mas, struct mmu_gather tlb; unsigned long mt_start = mas->index; - lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); unmap_vmas(&tlb, mas, vma, start, end, tree_end, mm_wr_locked); @@ -3360,7 +3359,6 @@ void exit_mmap(struct mm_struct *mm) goto destroy; } - lru_add_drain(); flush_cache_mm(mm); tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ diff --git a/mm/swap_state.c b/mm/swap_state.c index c574a9ecc6dd5..52b3e5c4d1ade 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -314,7 +314,6 @@ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) struct folio_batch folios; unsigned int refs[PAGEVEC_SIZE]; - lru_add_drain(); folio_batch_init(&folios); for (int i = 0; i < nr; i++) { struct folio *folio = page_folio(encoded_page_ptr(pages[i])); -- 2.34.1