From: Alistair Popple <apopple@nvidia.com> mainline inclusion from mainline-v6.15-rc1 commit 82ba975e4c43d98afebced82d940ddb7aec42a9d category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10302 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... -------------------------------- Zone device pages are used to represent various type of device memory managed by device drivers. Currently compound zone device pages are not supported. This is because MEMORY_DEVICE_FS_DAX pages are the only user of higher order zone device pages and have their own page reference counting. A future change will unify FS DAX reference counting with normal page reference counting rules and remove the special FS DAX reference counting. Supporting that requires compound zone device pages. Supporting compound zone device pages requires compound_head() to distinguish between head and tail pages whilst still preserving the special struct page fields that are specific to zone device pages. A tail page is distinguished by having bit zero being set in page->compound_head, with the remaining bits pointing to the head page. For zone device pages page->compound_head is shared with page->pgmap. The page->pgmap field must be common to all pages within a folio, even if the folio spans memory sections. Therefore pgmap is the same for both head and tail pages and can be moved into the folio and we can use the standard scheme to find compound_head from a tail page. Link: https://lkml.kernel.org/r/67055d772e6102accf85161d0b57b0b3944292bf.174071340... Signed-off-by: Alistair Popple <apopple@nvidia.com> Signed-off-by: Balbir Singh <balbirs@nvidia.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Reviewed-by: Dan Williams <dan.j.williams@intel.com> Acked-by: David Hildenbrand <david@redhat.com> Tested-by: Alison Schofield <alison.schofield@intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Asahi Lina <lina@asahilina.net> Cc: Bjorn Helgaas <bhelgaas@google.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Chunyan Zhang <zhang.lyra@gmail.com> Cc: "Darrick J. Wong" <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Dave Jiang <dave.jiang@intel.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Ira Weiny <ira.weiny@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: John Hubbard <jhubbard@nvidia.com> Cc: linmiaohe <linmiaohe@huawei.com> Cc: Logan Gunthorpe <logang@deltatee.com> Cc: Matthew Wilcow (Oracle) <willy@infradead.org> Cc: Michael "Camp Drill Sergeant" Ellerman <mpe@ellerman.id.au> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Ted Ts'o <tytso@mit.edu> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vishal Verma <vishal.l.verma@intel.com> Cc: Vivek Goyal <vgoyal@redhat.com> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Conflicts: drivers/gpu/drm/nouveau/nouveau_dmem.c include/linux/migrate.h include/linux/mm_types.h include/linux/mmzone.h lib/test_hmm.c mm/hmm.c mm/memory.c mm/memremap.c mm/migrate_device.c mm/mlock.c mm/mm_init.c [OLK-6.6 keeps pgmap in struct page and has no folio->pgmap, so the struct page/folio layout change in mm_types.h is not backported and page_pgmap() is implemented with compound_head()->pgmap. OLK-6.6's dump_page() takes a non-const struct page *, so the VM_WARN_ON_ONCE_PAGE() argument in page_pgmap() is cast. The device-private users (nouveau, test_hmm, hmm, memory, migrate_device, migrate.h) are left unchanged because those pages are order-0 and page->pgmap remains valid. In mm/memremap.c only the zone_device_page_init() hunk is taken; free_zone_device_folio() and __put_devmap_managed_folio_refs() use folio->pgmap and do not apply. mm/mm_init.c keeps assigning page->pgmap. mm/mlock.c keeps the PMD path guard against ZONE_DEVICE folios: compound zone device folios must not enter the mlock LRU path, and the PTE path already skips them.] Signed-off-by: Zizhi Wo <wozizhi@huawei.com> --- drivers/pci/p2pdma.c | 6 +++--- include/linux/memremap.h | 6 +++--- include/linux/mmzone.h | 13 ++++++++++++- mm/memremap.c | 2 +- mm/mlock.c | 2 ++ 5 files changed, 21 insertions(+), 8 deletions(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index b807f3fa221d..5522377f671d 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -192,11 +192,11 @@ static const struct attribute_group p2pmem_group = { .name = "p2pmem", }; static void p2pdma_page_free(struct page *page) { - struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page->pgmap); + struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page_pgmap(page)); /* safe to dereference while a reference is held to the percpu ref */ struct pci_p2pdma *p2pdma = rcu_dereference_protected(pgmap->provider->p2pdma, 1); struct percpu_ref *ref; @@ -1024,12 +1024,12 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, */ enum pci_p2pdma_map_type pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *dev, struct scatterlist *sg) { - if (state->pgmap != sg_page(sg)->pgmap) { - state->pgmap = sg_page(sg)->pgmap; + if (state->pgmap != page_pgmap(sg_page(sg))) { + state->pgmap = page_pgmap(sg_page(sg)); state->map = pci_p2pdma_map_type(state->pgmap, dev); state->bus_off = to_p2p_pgmap(state->pgmap)->bus_offset; } if (state->map == PCI_P2PDMA_MAP_BUS_ADDR) { diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 816532f7ae92..237b04d9ab1b 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -167,11 +167,11 @@ static inline unsigned long pgmap_vmemmap_nr(struct dev_pagemap *pgmap) static inline bool is_device_private_page(const struct page *page) { return IS_ENABLED(CONFIG_DEVICE_PRIVATE) && is_zone_device_page(page) && - page->pgmap->type == MEMORY_DEVICE_PRIVATE; + page_pgmap(page)->type == MEMORY_DEVICE_PRIVATE; } static inline bool folio_is_device_private(const struct folio *folio) { return is_device_private_page(&folio->page); @@ -179,17 +179,17 @@ static inline bool folio_is_device_private(const struct folio *folio) static inline bool is_pci_p2pdma_page(const struct page *page) { return IS_ENABLED(CONFIG_PCI_P2PDMA) && is_zone_device_page(page) && - page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA; + page_pgmap(page)->type == MEMORY_DEVICE_PCI_P2PDMA; } static inline bool is_device_coherent_page(const struct page *page) { return is_zone_device_page(page) && - page->pgmap->type == MEMORY_DEVICE_COHERENT; + page_pgmap(page)->type == MEMORY_DEVICE_COHERENT; } static inline bool folio_is_device_coherent(const struct folio *folio) { return is_device_coherent_page(&folio->page); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a6ed47ed840b..107e80ddbd8a 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1162,10 +1162,16 @@ static inline enum zone_type folio_zonenum(const struct folio *folio) static inline bool is_zone_device_page(const struct page *page) { return page_zonenum(page) == ZONE_DEVICE; } +static inline struct dev_pagemap *page_pgmap(const struct page *page) +{ + VM_WARN_ON_ONCE_PAGE(!is_zone_device_page(page), (struct page *)page); + return compound_head(page)->pgmap; +} + /* * Consecutive zone device pages should not be merged into the same sgl * or bvec segment with other types of pages or if they belong to different * pgmaps. Otherwise getting the pgmap of a given segment is not possible * without scanning the entire segment. This helper returns true either if @@ -1177,11 +1183,11 @@ static inline bool zone_device_pages_have_same_pgmap(const struct page *a, { if (is_zone_device_page(a) != is_zone_device_page(b)) return false; if (!is_zone_device_page(a)) return true; - return a->pgmap == b->pgmap; + return page_pgmap(a) == page_pgmap(b); } extern void memmap_init_zone_device(struct zone *, unsigned long, unsigned long, struct dev_pagemap *); #else @@ -1192,10 +1198,15 @@ static inline bool is_zone_device_page(const struct page *page) static inline bool zone_device_pages_have_same_pgmap(const struct page *a, const struct page *b) { return true; } + +static inline struct dev_pagemap *page_pgmap(const struct page *page) +{ + return NULL; +} #endif static inline bool folio_is_zone_device(const struct folio *folio) { return is_zone_device_page(&folio->page); diff --git a/mm/memremap.c b/mm/memremap.c index 8c5ebe5b0fc7..04306f3568dd 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -515,11 +515,11 @@ void zone_device_page_init(struct page *page) { /* * Drivers shouldn't be allocating pages after calling * memunmap_pages(). */ - WARN_ON_ONCE(!percpu_ref_tryget_live(&page->pgmap->ref)); + WARN_ON_ONCE(!percpu_ref_tryget_live(&page_pgmap(page)->ref)); set_page_count(page, 1); lock_page(page); } EXPORT_SYMBOL_GPL(zone_device_page_init); diff --git a/mm/mlock.c b/mm/mlock.c index 3e6e477741a9..5e76b064cd57 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -377,10 +377,12 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, if (!pmd_present(*pmd)) goto out; if (is_huge_zero_pmd(*pmd)) goto out; folio = pmd_folio(*pmd); + if (folio_is_zone_device(folio)) + goto out; if (vma->vm_flags & VM_LOCKED) mlock_folio(folio); else munlock_folio(folio); goto out; -- 2.52.0