[PATCH OLK-6.6 00/20] multi_pool_zram: support multi pool zram driver
This series supports the multi_pool_zram driver. Based on the zram driver, it supports multiple zspools, each bound to a different memcg, with the zspool memory allocated from the dpool of the corresponding memcg. Jinjiang Tu (14): multi_pool_zram: copy from zram and delete unused code multi_pool_zram: use zram_slot_lock in zram_slot_free_notify multi_pool_zram: prepare to support multiple pool multi_pool_zram: add sysfs file to support binding memcg for zpool multi_pool_zram: make write IO from memcg use zspool bounded to the memcg multi_pool_zram: add mm_stat interface multi_pool_zram: add mm_stat interface for each zspool multi_pool_zram: remove swapcache pages in unbinding memcg for zspool zram: fix zram usage charge in memcg openeuler_defconfig: enable CONFIG_MULTI_POOL_ZRAM by default multi_pool_zram: fix return value of comp_algorithm_store multi_pool_zram: fix race between disksize_store and zspool_mm_stat_show multi_pool_zram: fix zspool_stats not cleared on device reset multi_pool_zram: fix use-after-free in mem_cgroup_show Liu Shixin (6): mm/dynamic_pool: support to allocate page by specifying memcg mm/swap: support to allocate page from dpool for swapcache zram: set memcg for zram device zram: charge zram usage in memory cgroup zram: support to allocate page from dpool for zram openeuler_defconfig: enable memcg_zram for x86_64 and arm64 arch/arm64/configs/openeuler_defconfig | 2 + arch/x86/configs/openeuler_defconfig | 2 + drivers/block/Kconfig | 2 + drivers/block/Makefile | 1 + drivers/block/multi_pool_zram/Kconfig | 12 + drivers/block/multi_pool_zram/Makefile | 4 + .../multi_pool_zram/multi_pool_zram_drv.c | 1559 +++++++++++++++++ .../multi_pool_zram/multi_pool_zram_drv.h | 122 ++ drivers/block/multi_pool_zram/zcomp.c | 233 +++ drivers/block/multi_pool_zram/zcomp.h | 43 + include/linux/dynamic_pool.h | 13 +- include/linux/memcontrol.h | 21 + include/linux/swap.h | 10 + include/linux/swap_slots.h | 4 + include/linux/zsmalloc.h | 4 + init/Kconfig | 8 + mm/dynamic_pool.c | 23 +- mm/memcontrol.c | 147 ++ mm/page_alloc.c | 4 +- mm/swap_slots.c | 9 + mm/swap_state.c | 3 +- mm/swapfile.c | 65 + mm/zsmalloc.c | 81 +- 23 files changed, 2355 insertions(+), 17 deletions(-) create mode 100644 drivers/block/multi_pool_zram/Kconfig create mode 100644 drivers/block/multi_pool_zram/Makefile create mode 100644 drivers/block/multi_pool_zram/multi_pool_zram_drv.c create mode 100644 drivers/block/multi_pool_zram/multi_pool_zram_drv.h create mode 100644 drivers/block/multi_pool_zram/zcomp.c create mode 100644 drivers/block/multi_pool_zram/zcomp.h -- 2.43.0
From: Liu Shixin <liushixin2@huawei.com> hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH -------------------------------- Next patch will support to allocate page from a dpool for zram and zspool if they are bounded with a memcg, and the memcg is bounded with the dpool. As a preparation, support to allocate page by specifying memcg. Signed-off-by: Liu Shixin <liushixin2@huawei.com> --- include/linux/dynamic_pool.h | 13 ++++++++----- mm/dynamic_pool.c | 22 +++++++++++++++------- mm/page_alloc.c | 4 ++-- 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/include/linux/dynamic_pool.h b/include/linux/dynamic_pool.h index 9f53ad2d8064..0d64c5a02bd4 100644 --- a/include/linux/dynamic_pool.h +++ b/include/linux/dynamic_pool.h @@ -97,14 +97,15 @@ static inline bool page_from_or_in_dynamic_pool(struct page *page) } int dynamic_pool_can_attach(struct task_struct *tsk, struct mem_cgroup *memcg); -struct page *dynamic_pool_alloc_page(gfp_t gfp, unsigned int order, - unsigned int alloc_flags); +struct page *dynamic_pool_alloc_page(struct mem_cgroup *memcg, gfp_t gfp, + unsigned int order, unsigned int alloc_flags); void dynamic_pool_free_page(struct page *page); void dynamic_pool_bind_file(struct hugetlbfs_inode_info *p, struct hstate *h); void dynamic_pool_unbind_file(struct hugetlbfs_inode_info *p); int dynamic_pool_hugetlb_acct_memory(struct hstate *h, long delta, struct hugetlbfs_inode_info *p); -bool dynamic_pool_should_alloc(gfp_t gfp_mask, unsigned int order); +bool dynamic_pool_should_alloc(gfp_t gfp_mask, unsigned int order, + struct mem_cgroup *memcg); struct folio *dynamic_pool_alloc_hugepage(struct hugetlbfs_inode_info *p, struct hstate *h, bool reserved); void dynamic_pool_free_hugepage(struct folio *folio, bool restore_reserve); @@ -147,7 +148,8 @@ static inline int dynamic_pool_can_attach(struct task_struct *tsk, return 0; } -static inline struct page *dynamic_pool_alloc_page(gfp_t gfp, unsigned int order, +static inline struct page *dynamic_pool_alloc_page(struct mem_cgroup *memcg, + gfp_t gfp, unsigned int order, unsigned int alloc_flags) { return NULL; @@ -190,7 +192,8 @@ static inline void dynamic_pool_free_hugepage(struct folio *folio, } #endif -static inline bool dynamic_pool_should_alloc(gfp_t gfp_mask, unsigned int order) +static inline bool dynamic_pool_should_alloc(gfp_t gfp_mask, unsigned int order, + struct mem_cgroup *memcg) { return false; } diff --git a/mm/dynamic_pool.c b/mm/dynamic_pool.c index c5a45b1d3484..4dc9e80c2cb7 100644 --- a/mm/dynamic_pool.c +++ b/mm/dynamic_pool.c @@ -702,14 +702,19 @@ int dynamic_pool_can_attach(struct task_struct *tsk, struct mem_cgroup *memcg) return ret; } -bool dynamic_pool_should_alloc(gfp_t gfp_mask, unsigned int order) +bool dynamic_pool_should_alloc(gfp_t gfp_mask, unsigned int order, + struct mem_cgroup *memcg) { gfp_t gfp = gfp_mask & GFP_HIGHUSER_MOVABLE; - if (current->flags & PF_KTHREAD) + if (order != 0) return false; - if (order != 0) + /* Ignore the following restrictions if memcg is specfied */ + if (memcg) + return true; + + if (current->flags & PF_KTHREAD) return false; /* @@ -728,8 +733,8 @@ bool dynamic_pool_should_alloc(gfp_t gfp_mask, unsigned int order) return true; } -struct page *dynamic_pool_alloc_page(gfp_t gfp, unsigned int order, - unsigned int alloc_flags) +struct page *dynamic_pool_alloc_page(struct mem_cgroup *memcg, gfp_t gfp, + unsigned int order, unsigned int alloc_flags) { struct dynamic_pool *dpool; struct pages_pool *pool; @@ -739,10 +744,13 @@ struct page *dynamic_pool_alloc_page(gfp_t gfp, unsigned int order, if (!dpool_enabled) return NULL; - if (!dynamic_pool_should_alloc(gfp, order)) + if (!dynamic_pool_should_alloc(gfp, order, memcg)) return NULL; - dpool = dpool_get_from_task(current); + if (memcg) + dpool = dpool_get_from_memcg(memcg); + else + dpool = dpool_get_from_task(current); if (!dpool) return NULL; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6ec10acc9106..fcf29801c490 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4347,7 +4347,7 @@ static inline bool mem_reliable_fallback_dpool(gfp_t gfp_mask, unsigned int orde if (!(gfp_mask & GFP_RELIABLE)) return false; - return dynamic_pool_should_alloc(gfp_mask & ~GFP_RELIABLE, order); + return dynamic_pool_should_alloc(gfp_mask & ~GFP_RELIABLE, order, NULL); } #else static inline struct zone *mem_reliable_fallback_zone(gfp_t gfp_mask, @@ -5036,7 +5036,7 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, /* Before alloc from buddy system, alloc from dpool firstly */ if (dpool_enabled) { - page = dynamic_pool_alloc_page(alloc_gfp, order, alloc_flags); + page = dynamic_pool_alloc_page(NULL, alloc_gfp, order, alloc_flags); if (page) goto out; } -- 2.43.0
From: Liu Shixin <liushixin2@huawei.com> hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH -------------------------------- When swapin in async mode, if the swap entry is belong to a memcg and the memcg is bound with a dpool, then we should try to allocate page from dpool firstly. If the swapin is triggered in page fault, then we don't need to allocate from dpool proactively, since the task is belong to the memcg, the page will allocate from dpool naturally. Signed-off-by: Liu Shixin <liushixin2@huawei.com> --- include/linux/memcontrol.h | 10 ++++++++ mm/memcontrol.c | 48 ++++++++++++++++++++++++++++++++++++++ mm/swap_state.c | 3 +-- 3 files changed, 59 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 25e6580705ef..254741e81085 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -814,6 +814,9 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp, long nr_pages); +struct folio *memcg_vma_alloc_folio(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx); + int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry); @@ -1422,6 +1425,13 @@ static inline int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, return 0; } +static inline struct folio *memcg_vma_alloc_folio(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx) +{ + return (struct folio *)alloc_pages_mpol(gfp_mask, 0, + mpol, ilx, numa_node_id()); +} + static inline int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4ed54affdfa9..f4a948b49624 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -8479,6 +8479,54 @@ int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp, return 0; } +#ifdef CONFIG_DYNAMIC_POOL +static struct mem_cgroup *get_mem_cgroup_from_swap(swp_entry_t entry) +{ + struct mem_cgroup *memcg; + unsigned short id; + + if (mem_cgroup_disabled()) + return NULL; + + id = lookup_swap_cgroup_id(entry); + + rcu_read_lock(); + memcg = mem_cgroup_from_id(id); + if (memcg && !css_tryget_online(&memcg->css)) + memcg = NULL; + rcu_read_unlock(); + + return memcg; +} + +struct folio *memcg_vma_alloc_folio(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx) +{ + if (dpool_enabled) { + struct mem_cgroup *memcg; + struct page *page; + + memcg = get_mem_cgroup_from_swap(entry); + if (memcg) { + page = dynamic_pool_alloc_page(memcg, gfp_mask, 0, 0); + css_put(&memcg->css); + if (page) + return page_rmappable_folio(page); + } + } + + return (struct folio *)alloc_pages_mpol(gfp_mask, 0, + mpol, ilx, numa_node_id()); +} +#else +struct folio *memcg_vma_alloc_folio(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx) +{ + return (struct folio *)alloc_pages_mpol(gfp_mask, 0, + mpol, ilx, numa_node_id()); +} +#endif + /** * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin. * @folio: folio to charge. diff --git a/mm/swap_state.c b/mm/swap_state.c index 840e0f93bf7d..ae108ae1e21f 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -467,8 +467,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * before marking swap_map SWAP_HAS_CACHE, when -EEXIST will * cause any racers to loop around until we add it to cache. */ - folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, - mpol, ilx, numa_node_id()); + folio = memcg_vma_alloc_folio(entry, gfp_mask, mpol, ilx); if (!folio) goto fail_put_swap; -- 2.43.0
From: Liu Shixin <liushixin2@huawei.com> hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH -------------------------------- Add a new zram file mem_cgroup to set memcg for zram device. The memcg will be record in zs_pool too when create zs_pool. Signed-off-by: Liu Shixin <liushixin2@huawei.com> --- include/linux/memcontrol.h | 3 +++ include/linux/zsmalloc.h | 3 +++ init/Kconfig | 8 ++++++++ mm/memcontrol.c | 27 +++++++++++++++++++++++++++ mm/zsmalloc.c | 27 +++++++++++++++++++++++++++ 5 files changed, 68 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 254741e81085..cd5e626d8ba4 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1318,6 +1318,9 @@ static inline bool memcg_has_children(struct mem_cgroup *memcg) int mem_cgroup_force_empty(struct mem_cgroup *memcg); +#ifdef CONFIG_MEMCG_ZRAM +struct mem_cgroup *memcg_get_from_path(char *path, size_t buflen); +#endif #else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0 diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..2c09676f9178 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -41,6 +41,9 @@ struct zs_pool_stats { struct zs_pool; struct zs_pool *zs_create_pool(const char *name); +#ifdef CONFIG_MEMCG_ZRAM +struct zs_pool *zs_create_pool_with_memcg(const char *name, void *memcg); +#endif void zs_destroy_pool(struct zs_pool *pool); unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t flags); diff --git a/init/Kconfig b/init/Kconfig index cdcfe16d5a4d..09e59d22b556 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1036,6 +1036,14 @@ config MEMCG_SWAP_QOS memcg swap control include memory force swapin, swapfile control and swap limit. +config MEMCG_ZRAM + bool "Enable Memory Cgroup charge of zram usage" + depends on MEMCG_SWAP_QOS && ZRAM + depends on X86 || ARM64 + default n + help + Support to charge zram usage in memory cgroup. + config MEMCG_KMEM bool depends on MEMCG diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f4a948b49624..f6ca96b654d5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4141,6 +4141,33 @@ static int mem_cgroup_dummy_seq_show(__always_unused struct seq_file *m, return -EINVAL; } +#ifdef CONFIG_MEMCG_ZRAM +struct mem_cgroup *memcg_get_from_path(char *path, size_t buflen) +{ + struct mem_cgroup *memcg; + char *memcg_path; + + if (mem_cgroup_disabled()) + return NULL; + + memcg_path = kzalloc(buflen, GFP_KERNEL); + if (!memcg_path) + return NULL; + + for_each_mem_cgroup(memcg) { + cgroup_path(memcg->css.cgroup, memcg_path, buflen); + if (!strcmp(path, memcg_path) && css_tryget_online(&memcg->css)) { + mem_cgroup_iter_break(NULL, memcg); + break; + } + } + + kfree(memcg_path); + return memcg; +} +EXPORT_SYMBOL(memcg_get_from_path); +#endif + #ifdef CONFIG_MEMCG_KMEM static int memcg_online_kmem(struct mem_cgroup *memcg) { diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 5568ecb4d910..d873ce10e893 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -63,6 +63,7 @@ #include <linux/pagemap.h> #include <linux/fs.h> #include <linux/local_lock.h> +#include <linux/memcontrol.h> #define ZSPAGE_MAGIC 0x58 @@ -239,6 +240,9 @@ struct zs_pool { /* protect page/zspage migration */ rwlock_t migrate_lock; atomic_t compaction_in_progress; +#ifdef CONFIG_MEMCG_ZRAM + struct mem_cgroup *memcg; +#endif KABI_EXTEND(struct shrinker_v2 *shrinker) }; @@ -2224,10 +2228,33 @@ struct zs_pool *zs_create_pool(const char *name) } EXPORT_SYMBOL_GPL(zs_create_pool); +#ifdef CONFIG_MEMCG_ZRAM +static inline void zs_set_memcg(struct zs_pool *pool, void *memcg) +{ + if (pool) + pool->memcg = memcg; +} + +struct zs_pool *zs_create_pool_with_memcg(const char *name, void *memcg) +{ + struct zs_pool *pool = zs_create_pool(name); + + zs_set_memcg(pool, memcg); + + return pool; +} +EXPORT_SYMBOL_GPL(zs_create_pool_with_memcg); +#else +static inline void zs_set_memcg(struct zs_pool *pool, void *memcg) +{ +} +#endif + void zs_destroy_pool(struct zs_pool *pool) { int i; + zs_set_memcg(pool, NULL); zs_unregister_shrinker(pool); zs_flush_migration(pool); zs_pool_stat_destroy(pool); -- 2.43.0
From: Liu Shixin <liushixin2@huawei.com> hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH -------------------------------- In the previous patch, the zram and zspool can be bounded with memcg. Now support to charge the usage of zspage and zram->table in memory cgroup. This can be help limiting memory usage in container environment and prevent zram resource contention between containers. Add a new file memory.zram_usage_in_bytes to show zram usage. At the same time, the zram usage will also be added to memory.usage_in_bytes. Now the stats are as follows: memory.zram_usage_in_bytes = zram_usage memory.usage_in_bytes = program_usage + zram_usage memory.memsw.usage_in_bytes = program_usage + swap_usage Signed-off-by: Liu Shixin <liushixin2@huawei.com> --- include/linux/memcontrol.h | 6 +++++ mm/memcontrol.c | 52 ++++++++++++++++++++++++++++++++++++++ mm/zsmalloc.c | 31 +++++++++++++++++++++++ 3 files changed, 89 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index cd5e626d8ba4..c0e30821c94b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -418,7 +418,11 @@ struct mem_cgroup { #else KABI_RESERVE(1) #endif +#ifdef CONFIG_MEMCG_ZRAM + KABI_USE(2, atomic64_t zram_usage) +#else KABI_RESERVE(2) +#endif KABI_RESERVE(3) KABI_RESERVE(4) KABI_RESERVE(5) @@ -1320,6 +1324,8 @@ int mem_cgroup_force_empty(struct mem_cgroup *memcg); #ifdef CONFIG_MEMCG_ZRAM struct mem_cgroup *memcg_get_from_path(char *path, size_t buflen); +void memcg_charge_zram(struct mem_cgroup *memcg, unsigned int nr_pages); +void memcg_uncharge_zram(struct mem_cgroup *memcg, unsigned int nr_pages); #endif #else /* CONFIG_MEMCG */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f6ca96b654d5..960028758d92 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4166,6 +4166,49 @@ struct mem_cgroup *memcg_get_from_path(char *path, size_t buflen) return memcg; } EXPORT_SYMBOL(memcg_get_from_path); + +static inline void memcg_zram_usage_init(struct mem_cgroup *memcg) +{ + atomic64_set(&memcg->zram_usage, 0); +} + +void memcg_charge_zram(struct mem_cgroup *memcg, unsigned int nr_pages) +{ + if (mem_cgroup_disabled() || !memcg) + return; + + if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + page_counter_charge(&memcg->memory, nr_pages); + atomic_long_add(nr_pages, &memcg->zram_usage); +} +EXPORT_SYMBOL_GPL(memcg_charge_zram); + +void memcg_uncharge_zram(struct mem_cgroup *memcg, unsigned int nr_pages) +{ + if (mem_cgroup_disabled() || !memcg) + return; + + if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + page_counter_uncharge(&memcg->memory, nr_pages); + atomic_long_sub(nr_pages, &memcg->zram_usage); +} +EXPORT_SYMBOL_GPL(memcg_uncharge_zram); + +static u64 mem_cgroup_zram_usage(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + struct mem_cgroup *memcg = mem_cgroup_from_css(css); + + return (u64)atomic64_read(&memcg->zram_usage) * PAGE_SIZE; +} +#else +static inline void memcg_zram_usage_init(struct mem_cgroup *memcg) +{ +} #endif #ifdef CONFIG_MEMCG_KMEM @@ -4699,6 +4742,8 @@ static void memcg_swap_device_init(struct mem_cgroup *memcg, WRITE_ONCE(memcg->swap_dev->type, READ_ONCE(parent->swap_dev->type)); } + + memcg_zram_usage_init(memcg); } static u64 memcg_swapmax_read(struct cgroup_subsys_state *css, struct cftype *cft) @@ -6509,6 +6554,13 @@ static struct cftype mem_cgroup_legacy_files[] = { .write = memcg_swapfile_write, .seq_show = memcg_swapfile_read, }, +#ifdef CONFIG_MEMCG_ZRAM + { + .name = "zram_usage_in_bytes", + .flags = CFTYPE_NOT_ON_ROOT, + .read_u64 = mem_cgroup_zram_usage, + }, +#endif #endif #ifdef CONFIG_DYNAMIC_POOL { diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d873ce10e893..907a51cc28b1 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -826,6 +826,35 @@ static int trylock_zspage(struct zspage *zspage) return 0; } +#ifdef CONFIG_MEMCG_ZRAM +static inline void zs_charge_memory(struct zs_pool *pool, + unsigned long nr_pages) +{ + /* + * Since only zram configures memcg for zs_pool, + * charge the memory in zram usage. + */ + memcg_charge_zram(pool->memcg, nr_pages); +} + +static inline void zs_uncharge_memory(struct zs_pool *pool, + unsigned long nr_pages) +{ + /* See zs_charge_memory() for detail */ + memcg_uncharge_zram(pool->memcg, nr_pages); +} +#else +static inline void zs_charge_memory(struct zs_pool *pool, + unsigned long nr_pages) +{ +} + +static inline void zs_uncharge_memory(struct zs_pool *pool, + unsigned long nr_pages) +{ +} +#endif + static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { @@ -851,6 +880,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, class_stat_dec(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated); + zs_uncharge_memory(pool, class->pages_per_zspage); } static void free_zspage(struct zs_pool *pool, struct size_class *class, @@ -1383,6 +1413,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) insert_zspage(class, zspage, newfg); record_obj(handle, obj); atomic_long_add(class->pages_per_zspage, &pool->pages_allocated); + zs_charge_memory(pool, class->pages_per_zspage); class_stat_inc(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); class_stat_inc(class, ZS_OBJS_INUSE, 1); -- 2.43.0
From: Liu Shixin <liushixin2@huawei.com> hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH -------------------------------- After bound the zram with a memcg which has a dpool as memory provider. The memory usage of zram should prefer to allocate from dpool. zspool is allocated by alloc_page() before, replace it with zs_alloc_page() to achieve the effect. zram->table is allocated by vzalloc(), and vzalloc() is calling alloc_pages_node() to allocate memory. Add vzalloc_with_memcg() and __vmalloc_alloc_pages() to replace vzalloc() and alloc_pages_node() respectively to achieve the effect. Signed-off-by: Liu Shixin <liushixin2@huawei.com> --- mm/dynamic_pool.c | 1 + mm/zsmalloc.c | 22 +++++++++++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/mm/dynamic_pool.c b/mm/dynamic_pool.c index 4dc9e80c2cb7..0924105671ec 100644 --- a/mm/dynamic_pool.c +++ b/mm/dynamic_pool.c @@ -799,6 +799,7 @@ struct page *dynamic_pool_alloc_page(struct mem_cgroup *memcg, gfp_t gfp, return page; } +EXPORT_SYMBOL_GPL(dynamic_pool_alloc_page); void dynamic_pool_free_page(struct page *page) { diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 907a51cc28b1..1fb34d1dd563 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -64,6 +64,7 @@ #include <linux/fs.h> #include <linux/local_lock.h> #include <linux/memcontrol.h> +#include <linux/dynamic_pool.h> #define ZSPAGE_MAGIC 0x58 @@ -843,6 +844,20 @@ static inline void zs_uncharge_memory(struct zs_pool *pool, /* See zs_charge_memory() for detail */ memcg_uncharge_zram(pool->memcg, nr_pages); } + +static inline struct page *zs_alloc_page(struct zs_pool *pool, gfp_t gfp) +{ + struct mem_cgroup *memcg = pool->memcg; + struct page *page; + + if (dpool_enabled && memcg) { + page = dynamic_pool_alloc_page(memcg, gfp, 0, 0); + if (page) + return page; + } + + return alloc_page(gfp); +} #else static inline void zs_charge_memory(struct zs_pool *pool, unsigned long nr_pages) @@ -853,6 +868,11 @@ static inline void zs_uncharge_memory(struct zs_pool *pool, unsigned long nr_pages) { } + +static inline struct page *zs_alloc_page(struct zs_pool *pool, gfp_t gfp) +{ + return alloc_page(gfp); +} #endif static void __free_zspage(struct zs_pool *pool, struct size_class *class, @@ -1004,7 +1024,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, for (i = 0; i < class->pages_per_zspage; i++) { struct page *page; - page = alloc_page(gfp); + page = zs_alloc_page(pool, gfp); if (!page) { while (--i >= 0) { dec_zone_page_state(pages[i], NR_ZSPAGES); -- 2.43.0
From: Liu Shixin <liushixin2@huawei.com> hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH -------------------------------- Enable CONFIG_MEMCG_ZRAM to support memcg_zram feature. Signed-off-by: Liu Shixin <liushixin2@huawei.com> --- arch/arm64/configs/openeuler_defconfig | 1 + arch/x86/configs/openeuler_defconfig | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index edf2ed90efb8..100af4496175 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -173,6 +173,7 @@ CONFIG_MEMCG_QOS=y CONFIG_MEMCG_MEMFS_INFO=y CONFIG_MEMCG_OOM_PRIORITY=y CONFIG_MEMCG_SWAP_QOS=y +CONFIG_MEMCG_ZRAM=y CONFIG_MEMCG_KMEM=y CONFIG_MEMCG_KMEM_STOCK=y CONFIG_BLK_CGROUP=y diff --git a/arch/x86/configs/openeuler_defconfig b/arch/x86/configs/openeuler_defconfig index 5bc6af6a1c4b..4beff1733c18 100644 --- a/arch/x86/configs/openeuler_defconfig +++ b/arch/x86/configs/openeuler_defconfig @@ -192,6 +192,7 @@ CONFIG_MEMCG_QOS=y CONFIG_MEMCG_MEMFS_INFO=y CONFIG_MEMCG_OOM_PRIORITY=y CONFIG_MEMCG_SWAP_QOS=y +CONFIG_MEMCG_ZRAM=y CONFIG_MEMCG_KMEM=y CONFIG_MEMCG_KMEM_STOCK=y CONFIG_BLK_CGROUP=y -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Initialize multiple pool zram driver by copying from zram and delete unused code. The following code is deleted: * CONFIG_ZRAM_WRITEBACK related code * idle related code * CONFIG_ZRAM_MEMORY_TRACKING and ZRAM_TRACK_ENTRY_ACTIME related code * mem_limit and max_used_pages related code * CONFIG_MEMCG_ZRAM related code * max_comp_streams interface * CONFIG_ZRAM_MULTI_COMP related code * compact interface The following interfaces need to be adapted to multiple pools, so delete them in this patch and add them back in the later patch. * mm_stat interface Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- drivers/block/Kconfig | 2 + drivers/block/Makefile | 1 + drivers/block/multi_pool_zram/Kconfig | 12 + drivers/block/multi_pool_zram/Makefile | 4 + .../multi_pool_zram/multi_pool_zram_drv.c | 1128 +++++++++++++++++ .../multi_pool_zram/multi_pool_zram_drv.h | 89 ++ drivers/block/multi_pool_zram/zcomp.c | 233 ++++ drivers/block/multi_pool_zram/zcomp.h | 43 + 8 files changed, 1512 insertions(+) create mode 100644 drivers/block/multi_pool_zram/Kconfig create mode 100644 drivers/block/multi_pool_zram/Makefile create mode 100644 drivers/block/multi_pool_zram/multi_pool_zram_drv.c create mode 100644 drivers/block/multi_pool_zram/multi_pool_zram_drv.h create mode 100644 drivers/block/multi_pool_zram/zcomp.c create mode 100644 drivers/block/multi_pool_zram/zcomp.h diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig index 5b9d4aaebb81..861c959d16f6 100644 --- a/drivers/block/Kconfig +++ b/drivers/block/Kconfig @@ -107,6 +107,8 @@ source "drivers/block/mtip32xx/Kconfig" source "drivers/block/zram/Kconfig" +source "drivers/block/multi_pool_zram/Kconfig" + config BLK_DEV_UBD bool "Virtual block device" depends on UML diff --git a/drivers/block/Makefile b/drivers/block/Makefile index 101612cba303..bb5c00991e75 100644 --- a/drivers/block/Makefile +++ b/drivers/block/Makefile @@ -33,6 +33,7 @@ obj-$(CONFIG_BLK_DEV_RBD) += rbd.o obj-$(CONFIG_BLK_DEV_PCIESSD_MTIP32XX) += mtip32xx/ obj-$(CONFIG_ZRAM) += zram/ +obj-$(CONFIG_ZRAM) += multi_pool_zram/ obj-$(CONFIG_BLK_DEV_RNBD) += rnbd/ obj-$(CONFIG_BLK_DEV_NULL_BLK) += null_blk/ diff --git a/drivers/block/multi_pool_zram/Kconfig b/drivers/block/multi_pool_zram/Kconfig new file mode 100644 index 000000000000..a7fb0986b3de --- /dev/null +++ b/drivers/block/multi_pool_zram/Kconfig @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0 +config MULTI_POOL_ZRAM + tristate "Compressed RAM block device with multiple zspools" + depends on BLOCK && SYSFS && MMU + depends on CRYPTO_LZO || CRYPTO_ZSTD || CRYPTO_LZ4 || CRYPTO_LZ4HC || CRYPTO_842 + depends on MEMCG_ZRAM + select ZSMALLOC + help + Support zram driver with multiple zspools. Each zspool can + bind to a memcg. The memory allocated by the zspool will + be from the dynamoc pool of the bounded memcg, and IO from + the memcg will use the bounded zspool. diff --git a/drivers/block/multi_pool_zram/Makefile b/drivers/block/multi_pool_zram/Makefile new file mode 100644 index 000000000000..30b74db73730 --- /dev/null +++ b/drivers/block/multi_pool_zram/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only +multi_pool_zram-y := zcomp.o multi_pool_zram_drv.o + +obj-$(CONFIG_MULTI_POOL_ZRAM) += multi_pool_zram.o diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c new file mode 100644 index 000000000000..69a899ae98b5 --- /dev/null +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -0,0 +1,1128 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Compressed RAM block device with multiple zspool support + */ + +#define KMSG_COMPONENT "multi-pool zram" +#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt + +#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/bio.h> +#include <linux/bitops.h> +#include <linux/blkdev.h> +#include <linux/buffer_head.h> +#include <linux/device.h> +#include <linux/highmem.h> +#include <linux/slab.h> +#include <linux/backing-dev.h> +#include <linux/string.h> +#include <linux/vmalloc.h> +#include <linux/err.h> +#include <linux/idr.h> +#include <linux/sysfs.h> +#include <linux/debugfs.h> +#include <linux/cpuhotplug.h> +#include <linux/part_stat.h> + +#include "multi_pool_zram_drv.h" + +static DEFINE_IDR(zram_index_idr); +/* idr index must be protected */ +static DEFINE_MUTEX(zram_index_mutex); + +static int zram_major; +static const char *default_compressor = "lzo-rle"; + +/* Module params (documentation at end) */ +static unsigned int num_devices = 1; +/* + * Pages that compress to sizes equals or greater than this are stored + * uncompressed in memory. + */ +static size_t huge_class_size; + +static const struct block_device_operations zram_devops; + +static void zram_free_page(struct zram *zram, size_t index); +static int zram_read_page(struct zram *zram, struct page *page, u32 index, + struct bio *parent); + +static int zram_slot_trylock(struct zram *zram, u32 index) +{ + return bit_spin_trylock(ZRAM_LOCK, &zram->table[index].flags); +} + +static void zram_slot_lock(struct zram *zram, u32 index) +{ + bit_spin_lock(ZRAM_LOCK, &zram->table[index].flags); +} + +static void zram_slot_unlock(struct zram *zram, u32 index) +{ + bit_spin_unlock(ZRAM_LOCK, &zram->table[index].flags); +} + +static inline bool init_done(struct zram *zram) +{ + return zram->disksize; +} + +static inline struct zram *dev_to_zram(struct device *dev) +{ + return (struct zram *)dev_to_disk(dev)->private_data; +} + +static unsigned long zram_get_handle(struct zram *zram, u32 index) +{ + return zram->table[index].handle; +} + +static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) +{ + zram->table[index].handle = handle; +} + +/* flag operations require table entry bit_spin_lock() being held */ +static bool zram_test_flag(struct zram *zram, u32 index, + enum zram_pageflags flag) +{ + return zram->table[index].flags & BIT(flag); +} + +static void zram_set_flag(struct zram *zram, u32 index, + enum zram_pageflags flag) +{ + zram->table[index].flags |= BIT(flag); +} + +static void zram_clear_flag(struct zram *zram, u32 index, + enum zram_pageflags flag) +{ + zram->table[index].flags &= ~BIT(flag); +} + +static inline void zram_set_element(struct zram *zram, u32 index, + unsigned long element) +{ + zram->table[index].element = element; +} + +static unsigned long zram_get_element(struct zram *zram, u32 index) +{ + return zram->table[index].element; +} + +static size_t zram_get_obj_size(struct zram *zram, u32 index) +{ + return zram->table[index].flags & (BIT(ZRAM_FLAG_SHIFT) - 1); +} + +static void zram_set_obj_size(struct zram *zram, + u32 index, size_t size) +{ + unsigned long flags = zram->table[index].flags >> ZRAM_FLAG_SHIFT; + + zram->table[index].flags = (flags << ZRAM_FLAG_SHIFT) | size; +} + +#if PAGE_SIZE != 4096 +static inline bool is_partial_io(struct bio_vec *bvec) +{ + return bvec->bv_len != PAGE_SIZE; +} +#define ZRAM_PARTIAL_IO 1 +#else +static inline bool is_partial_io(struct bio_vec *bvec) +{ + return false; +} +#endif + +static inline void zram_fill_page(void *ptr, unsigned long len, + unsigned long value) +{ + WARN_ON_ONCE(!IS_ALIGNED(len, sizeof(unsigned long))); + memset_l(ptr, value, len / sizeof(unsigned long)); +} + +static bool page_same_filled(void *ptr, unsigned long *element) +{ + unsigned long *page; + unsigned long val; + unsigned int pos, last_pos = PAGE_SIZE / sizeof(*page) - 1; + + page = (unsigned long *)ptr; + val = page[0]; + + if (val != page[last_pos]) + return false; + + for (pos = 1; pos < last_pos; pos++) { + if (val != page[pos]) + return false; + } + + *element = val; + + return true; +} + +static ssize_t initstate_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + u32 val; + struct zram *zram = dev_to_zram(dev); + + down_read(&zram->init_lock); + val = init_done(zram); + up_read(&zram->init_lock); + + return scnprintf(buf, PAGE_SIZE, "%u\n", val); +} + +static ssize_t disksize_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct zram *zram = dev_to_zram(dev); + + return scnprintf(buf, PAGE_SIZE, "%llu\n", zram->disksize); +} + +static ssize_t comp_algorithm_store(struct device *dev, + struct device_attribute *attr, + const char *buf, + size_t len) +{ + struct zram *zram = dev_to_zram(dev); + char compressor[ARRAY_SIZE(zram->compressor)]; + size_t sz; + + sz = strlen(buf); + if (sz >= CRYPTO_MAX_ALG_NAME) + return -E2BIG; + + strscpy(compressor, buf, sizeof(compressor)); + /* ignore trailing newline */ + if (sz > 0 && compressor[sz - 1] == '\n') + compressor[sz - 1] = 0x00; + + if (!zcomp_available_algorithm(compressor)) + return -EINVAL; + + down_write(&zram->init_lock); + if (init_done(zram)) { + up_write(&zram->init_lock); + pr_info("Can't change algorithm for initialized device\n"); + return -EBUSY; + } + + strcpy(zram->compressor, compressor); + up_write(&zram->init_lock); + return 0; +} + +static ssize_t comp_algorithm_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct zram *zram = dev_to_zram(dev); + ssize_t sz; + + down_read(&zram->init_lock); + sz = zcomp_available_show(zram->compressor, buf); + up_read(&zram->init_lock); + + return sz; +} + +static ssize_t io_stat_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct zram *zram = dev_to_zram(dev); + ssize_t ret; + + down_read(&zram->init_lock); + ret = scnprintf(buf, PAGE_SIZE, + "%8llu %8llu 0 %8llu\n", + (u64)atomic64_read(&zram->stats.failed_reads), + (u64)atomic64_read(&zram->stats.failed_writes), + (u64)atomic64_read(&zram->stats.notify_free)); + up_read(&zram->init_lock); + + return ret; +} + +static ssize_t debug_stat_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + int version = 1; + struct zram *zram = dev_to_zram(dev); + ssize_t ret; + + down_read(&zram->init_lock); + ret = scnprintf(buf, PAGE_SIZE, + "version: %d\n%8llu %8llu\n", + version, + (u64)atomic64_read(&zram->stats.writestall), + (u64)atomic64_read(&zram->stats.miss_free)); + up_read(&zram->init_lock); + + return ret; +} + +static DEVICE_ATTR_RO(io_stat); +static DEVICE_ATTR_RO(debug_stat); + +static void zram_meta_free(struct zram *zram, u64 disksize) +{ + size_t num_pages = disksize >> PAGE_SHIFT; + size_t index; + + if (!zram->table) + return; + + /* Free all pages that are still in this zram device */ + for (index = 0; index < num_pages; index++) + zram_free_page(zram, index); + + zs_destroy_pool(zram->mem_pool); + vfree(zram->table); + zram->table = NULL; +} + +static bool zram_meta_alloc(struct zram *zram, u64 disksize) +{ + size_t num_pages = disksize >> PAGE_SHIFT; + unsigned long size = array_size(num_pages, sizeof(*zram->table)); + + zram->table = vzalloc(size); + if (!zram->table) + return false; + + zram->mem_pool = zs_create_pool(zram->disk->disk_name); + if (!zram->mem_pool) { + vfree(zram->table); + zram->table = NULL; + return false; + } + + if (!huge_class_size) + huge_class_size = zs_huge_class_size(zram->mem_pool); + return true; +} + +/* + * To protect concurrent access to the same index entry, + * caller should hold this table index entry's bit_spinlock to + * indicate this index entry is accessing. + */ +static void zram_free_page(struct zram *zram, size_t index) +{ + unsigned long handle; + + if (zram_test_flag(zram, index, ZRAM_HUGE)) { + zram_clear_flag(zram, index, ZRAM_HUGE); + atomic64_dec(&zram->stats.huge_pages); + } + + /* + * No memory is allocated for same element filled pages. + * Simply clear same page flag. + */ + if (zram_test_flag(zram, index, ZRAM_SAME)) { + zram_clear_flag(zram, index, ZRAM_SAME); + atomic64_dec(&zram->stats.same_pages); + goto out; + } + + handle = zram_get_handle(zram, index); + if (!handle) + return; + + zs_free(zram->mem_pool, handle); + + atomic64_sub(zram_get_obj_size(zram, index), + &zram->stats.compr_data_size); +out: + atomic64_dec(&zram->stats.pages_stored); + zram_set_handle(zram, index, 0); + zram_set_obj_size(zram, index, 0); + WARN_ON_ONCE(zram->table[index].flags & + ~(1UL << ZRAM_LOCK)); +} + +/* + * Reads (decompresses if needed) a page from zspool (zsmalloc). + * Corresponding ZRAM slot should be locked. + */ +static int zram_read_from_zspool(struct zram *zram, struct page *page, + u32 index) +{ + struct zcomp_strm *zstrm; + unsigned long handle; + unsigned int size; + void *src, *dst; + int ret; + + handle = zram_get_handle(zram, index); + if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { + unsigned long value; + void *mem; + + value = handle ? zram_get_element(zram, index) : 0; + mem = kmap_atomic(page); + zram_fill_page(mem, PAGE_SIZE, value); + kunmap_atomic(mem); + return 0; + } + + size = zram_get_obj_size(zram, index); + + if (size != PAGE_SIZE) + zstrm = zcomp_stream_get(zram->comp); + + src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + if (size == PAGE_SIZE) { + dst = kmap_atomic(page); + memcpy(dst, src, PAGE_SIZE); + kunmap_atomic(dst); + ret = 0; + } else { + dst = kmap_atomic(page); + ret = zcomp_decompress(zstrm, src, size, dst); + kunmap_atomic(dst); + zcomp_stream_put(zram->comp); + } + zs_unmap_object(zram->mem_pool, handle); + return ret; +} + +static int zram_read_page(struct zram *zram, struct page *page, u32 index, + struct bio *parent) +{ + int ret; + + zram_slot_lock(zram, index); + /* Slot should be locked through out the function call */ + ret = zram_read_from_zspool(zram, page, index); + zram_slot_unlock(zram, index); + + /* Should NEVER happen. Return bio error if it does. */ + if (WARN_ON(ret < 0)) + pr_err("Decompression failed! err=%d, page=%u\n", ret, index); + + return ret; +} + +/* + * Use a temporary buffer to decompress the page, as the decompressor + * always expects a full page for the output. + */ +static int zram_bvec_read_partial(struct zram *zram, struct bio_vec *bvec, + u32 index, int offset) +{ + struct page *page = alloc_page(GFP_NOIO); + int ret; + + if (!page) + return -ENOMEM; + ret = zram_read_page(zram, page, index, NULL); + if (likely(!ret)) + memcpy_to_bvec(bvec, page_address(page) + offset); + __free_page(page); + return ret; +} + +static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, + u32 index, int offset, struct bio *bio) +{ + if (is_partial_io(bvec)) + return zram_bvec_read_partial(zram, bvec, index, offset); + return zram_read_page(zram, bvec->bv_page, index, bio); +} + +static int zram_write_page(struct zram *zram, struct page *page, u32 index) +{ + int ret = 0; + unsigned long handle = -ENOMEM; + unsigned int comp_len = 0; + void *src, *dst, *mem; + struct zcomp_strm *zstrm; + unsigned long element = 0; + enum zram_pageflags flags = 0; + + mem = kmap_atomic(page); + if (page_same_filled(mem, &element)) { + kunmap_atomic(mem); + /* Free memory associated with this sector now. */ + flags = ZRAM_SAME; + atomic64_inc(&zram->stats.same_pages); + goto out; + } + kunmap_atomic(mem); + +compress_again: + zstrm = zcomp_stream_get(zram->comp); + src = kmap_atomic(page); + ret = zcomp_compress(zstrm, src, &comp_len); + kunmap_atomic(src); + + if (unlikely(ret)) { + zcomp_stream_put(zram->comp); + pr_err("Compression failed! err=%d\n", ret); + zs_free(zram->mem_pool, handle); + return ret; + } + + if (comp_len >= huge_class_size) + comp_len = PAGE_SIZE; + /* + * handle allocation has 2 paths: + * a) fast path is executed with preemption disabled (for + * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, + * since we can't sleep; + * b) slow path enables preemption and attempts to allocate + * the page with __GFP_DIRECT_RECLAIM bit set. we have to + * put per-cpu compression stream and, thus, to re-do + * the compression once handle is allocated. + * + * if we have a 'non-null' handle here then we are coming + * from the slow path and handle has already been allocated. + */ + if (IS_ERR_VALUE(handle)) + handle = zs_malloc(zram->mem_pool, comp_len, + __GFP_KSWAPD_RECLAIM | + __GFP_NOWARN | + __GFP_HIGHMEM | + __GFP_MOVABLE); + if (IS_ERR_VALUE(handle)) { + zcomp_stream_put(zram->comp); + atomic64_inc(&zram->stats.writestall); + handle = zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | + __GFP_MOVABLE); + if (IS_ERR_VALUE(handle)) + return PTR_ERR((void *)handle); + + if (comp_len != PAGE_SIZE) + goto compress_again; + /* + * If the page is not compressible, you need to acquire the + * lock and execute the code below. The zcomp_stream_get() + * call is needed to disable the cpu hotplug and grab the + * zstrm buffer back. It is necessary that the dereferencing + * of the zstrm variable below occurs correctly. + */ + zstrm = zcomp_stream_get(zram->comp); + } + + dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); + src = zstrm->buffer; + if (comp_len == PAGE_SIZE) + src = kmap_atomic(page); + memcpy(dst, src, comp_len); + if (comp_len == PAGE_SIZE) + kunmap_atomic(src); + + zcomp_stream_put(zram->comp); + zs_unmap_object(zram->mem_pool, handle); + atomic64_add(comp_len, &zram->stats.compr_data_size); +out: + /* + * Free memory associated with this sector + * before overwriting unused sectors. + */ + zram_slot_lock(zram, index); + zram_free_page(zram, index); + + if (comp_len == PAGE_SIZE) { + zram_set_flag(zram, index, ZRAM_HUGE); + atomic64_inc(&zram->stats.huge_pages); + atomic64_inc(&zram->stats.huge_pages_since); + } + + if (flags) { + zram_set_flag(zram, index, flags); + zram_set_element(zram, index, element); + } else { + zram_set_handle(zram, index, handle); + zram_set_obj_size(zram, index, comp_len); + } + zram_slot_unlock(zram, index); + + /* Update stats */ + atomic64_inc(&zram->stats.pages_stored); + return ret; +} + +/* + * This is a partial IO. Read the full page before writing the changes. + */ +static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec, + u32 index, int offset, struct bio *bio) +{ + struct page *page = alloc_page(GFP_NOIO); + int ret; + + if (!page) + return -ENOMEM; + + ret = zram_read_page(zram, page, index, bio); + if (!ret) { + memcpy_from_bvec(page_address(page) + offset, bvec); + ret = zram_write_page(zram, page, index); + } + __free_page(page); + return ret; +} + +static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, + u32 index, int offset, struct bio *bio) +{ + if (is_partial_io(bvec)) + return zram_bvec_write_partial(zram, bvec, index, offset, bio); + return zram_write_page(zram, bvec->bv_page, index); +} + +static void zram_bio_discard(struct zram *zram, struct bio *bio) +{ + size_t n = bio->bi_iter.bi_size; + u32 index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + u32 offset = (bio->bi_iter.bi_sector & (SECTORS_PER_PAGE - 1)) << + SECTOR_SHIFT; + + /* + * zram manages data in physical block size units. Because logical block + * size isn't identical with physical block size on some arch, we + * could get a discard request pointing to a specific offset within a + * certain physical block. Although we can handle this request by + * reading that physiclal block and decompressing and partially zeroing + * and re-compressing and then re-storing it, this isn't reasonable + * because our intent with a discard request is to save memory. So + * skipping this logical block is appropriate here. + */ + if (offset) { + if (n <= (PAGE_SIZE - offset)) + return; + + n -= (PAGE_SIZE - offset); + index++; + } + + while (n >= PAGE_SIZE) { + zram_slot_lock(zram, index); + zram_free_page(zram, index); + zram_slot_unlock(zram, index); + atomic64_inc(&zram->stats.notify_free); + index++; + n -= PAGE_SIZE; + } + + bio_endio(bio); +} + +static void zram_bio_read(struct zram *zram, struct bio *bio) +{ + unsigned long start_time = bio_start_io_acct(bio); + struct bvec_iter iter = bio->bi_iter; + + do { + u32 index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + u32 offset = (iter.bi_sector & (SECTORS_PER_PAGE - 1)) << + SECTOR_SHIFT; + struct bio_vec bv = bio_iter_iovec(bio, iter); + + bv.bv_len = min_t(u32, bv.bv_len, PAGE_SIZE - offset); + + if (zram_bvec_read(zram, &bv, index, offset, bio) < 0) { + atomic64_inc(&zram->stats.failed_reads); + bio->bi_status = BLK_STS_IOERR; + break; + } + flush_dcache_page(bv.bv_page); + bio_advance_iter_single(bio, &iter, bv.bv_len); + } while (iter.bi_size); + + bio_end_io_acct(bio, start_time); + bio_endio(bio); +} + +static void zram_bio_write(struct zram *zram, struct bio *bio) +{ + unsigned long start_time = bio_start_io_acct(bio); + struct bvec_iter iter = bio->bi_iter; + + do { + u32 index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + u32 offset = (iter.bi_sector & (SECTORS_PER_PAGE - 1)) << + SECTOR_SHIFT; + struct bio_vec bv = bio_iter_iovec(bio, iter); + + bv.bv_len = min_t(u32, bv.bv_len, PAGE_SIZE - offset); + + if (zram_bvec_write(zram, &bv, index, offset, bio) < 0) { + atomic64_inc(&zram->stats.failed_writes); + bio->bi_status = BLK_STS_IOERR; + break; + } + bio_advance_iter_single(bio, &iter, bv.bv_len); + } while (iter.bi_size); + + bio_end_io_acct(bio, start_time); + bio_endio(bio); +} + +/* + * Handler function for all zram I/O requests. + */ +static void zram_submit_bio(struct bio *bio) +{ + struct zram *zram = bio->bi_bdev->bd_disk->private_data; + + switch (bio_op(bio)) { + case REQ_OP_READ: + zram_bio_read(zram, bio); + break; + case REQ_OP_WRITE: + zram_bio_write(zram, bio); + break; + case REQ_OP_DISCARD: + case REQ_OP_WRITE_ZEROES: + zram_bio_discard(zram, bio); + break; + default: + WARN_ON_ONCE(1); + bio_endio(bio); + } +} + +static void zram_slot_free_notify(struct block_device *bdev, + unsigned long index) +{ + struct zram *zram; + + zram = bdev->bd_disk->private_data; + + atomic64_inc(&zram->stats.notify_free); + if (!zram_slot_trylock(zram, index)) { + atomic64_inc(&zram->stats.miss_free); + return; + } + + zram_free_page(zram, index); + zram_slot_unlock(zram, index); +} + +static void zram_reset_device(struct zram *zram) +{ + down_write(&zram->init_lock); + + set_capacity_and_notify(zram->disk, 0); + part_stat_set_all(zram->disk->part0, 0); + + /* I/O operation under all of CPU are done so let's free */ + zram_meta_free(zram, zram->disksize); + zram->disksize = 0; + if (zram->comp) { + zcomp_destroy(zram->comp); + zram->comp = NULL; + } + memset(&zram->stats, 0, sizeof(zram->stats)); + strscpy(zram->compressor, default_compressor, sizeof(zram->compressor)); + up_write(&zram->init_lock); +} + +static ssize_t disksize_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + u64 disksize; + struct zcomp *comp; + struct zram *zram = dev_to_zram(dev); + int err; + + disksize = memparse(buf, NULL); + if (!disksize) + return -EINVAL; + + down_write(&zram->init_lock); + if (init_done(zram)) { + pr_info("Cannot change disksize for initialized device\n"); + err = -EBUSY; + goto out_unlock; + } + + disksize = PAGE_ALIGN(disksize); + if (!zram_meta_alloc(zram, disksize)) { + err = -ENOMEM; + goto out_unlock; + } + + comp = zcomp_create(zram->compressor); + if (IS_ERR(comp)) { + pr_err("Cannot initialise %s compressing backend\n", + zram->compressor); + err = PTR_ERR(comp); + goto out_free_meta; + } + + zram->comp = comp; + zram->disksize = disksize; + set_capacity_and_notify(zram->disk, zram->disksize >> SECTOR_SHIFT); + up_write(&zram->init_lock); + + return len; + +out_free_meta: + zram_meta_free(zram, disksize); +out_unlock: + up_write(&zram->init_lock); + return err; +} + +static ssize_t reset_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + int ret; + unsigned short do_reset; + struct zram *zram; + struct gendisk *disk; + + ret = kstrtou16(buf, 10, &do_reset); + if (ret) + return ret; + + if (!do_reset) + return -EINVAL; + + zram = dev_to_zram(dev); + disk = zram->disk; + + mutex_lock(&disk->open_mutex); + /* Do not reset an active device or claimed device */ + if (disk_openers(disk) || zram->claim) { + mutex_unlock(&disk->open_mutex); + return -EBUSY; + } + + /* From now on, anyone can't open /dev/zram[0-9] */ + zram->claim = true; + mutex_unlock(&disk->open_mutex); + + /* Make sure all the pending I/O are finished */ + sync_blockdev(disk->part0); + zram_reset_device(zram); + + mutex_lock(&disk->open_mutex); + zram->claim = false; + mutex_unlock(&disk->open_mutex); + + return len; +} + +static int zram_open(struct gendisk *disk, blk_mode_t mode) +{ + struct zram *zram = disk->private_data; + + WARN_ON(!mutex_is_locked(&disk->open_mutex)); + + /* zram was claimed to reset so open request fails */ + if (zram->claim) + return -EBUSY; + return 0; +} + +static const struct block_device_operations zram_devops = { + .open = zram_open, + .submit_bio = zram_submit_bio, + .swap_slot_free_notify = zram_slot_free_notify, + .owner = THIS_MODULE +}; + +static DEVICE_ATTR_RW(disksize); +static DEVICE_ATTR_RO(initstate); +static DEVICE_ATTR_WO(reset); +static DEVICE_ATTR_RW(comp_algorithm); + +static struct attribute *zram_disk_attrs[] = { + &dev_attr_disksize.attr, + &dev_attr_initstate.attr, + &dev_attr_reset.attr, + &dev_attr_comp_algorithm.attr, + &dev_attr_io_stat.attr, + &dev_attr_debug_stat.attr, + NULL, +}; + +ATTRIBUTE_GROUPS(zram_disk); + +/* + * Allocate and initialize new zram device. the function returns + * '>= 0' device_id upon success, and negative value otherwise. + */ +static int zram_add(void) +{ + struct zram *zram; + int ret, device_id; + + zram = kzalloc(sizeof(struct zram), GFP_KERNEL); + if (!zram) + return -ENOMEM; + + ret = idr_alloc(&zram_index_idr, zram, 0, 0, GFP_KERNEL); + if (ret < 0) + goto out_free_dev; + device_id = ret; + + init_rwsem(&zram->init_lock); + + /* gendisk structure */ + zram->disk = blk_alloc_disk(NUMA_NO_NODE); + if (!zram->disk) { + pr_err("Error allocating disk structure for device %d\n", + device_id); + ret = -ENOMEM; + goto out_free_idr; + } + + zram->disk->major = zram_major; + zram->disk->first_minor = device_id; + zram->disk->minors = 1; + zram->disk->flags |= GENHD_FL_NO_PART; + zram->disk->fops = &zram_devops; + zram->disk->private_data = zram; + snprintf(zram->disk->disk_name, 16, "zram%d", device_id); + strscpy(zram->compressor, default_compressor, sizeof(zram->compressor)); + + /* Actual capacity set using sysfs (/sys/block/zram<id>/disksize */ + set_capacity(zram->disk, 0); + /* zram devices sort of resembles non-rotational disks */ + blk_queue_flag_set(QUEUE_FLAG_NONROT, zram->disk->queue); + blk_queue_flag_set(QUEUE_FLAG_SYNCHRONOUS, zram->disk->queue); + + /* + * To ensure that we always get PAGE_SIZE aligned + * and n*PAGE_SIZED sized I/O requests. + */ + blk_queue_physical_block_size(zram->disk->queue, PAGE_SIZE); + blk_queue_logical_block_size(zram->disk->queue, + ZRAM_LOGICAL_BLOCK_SIZE); + blk_queue_io_min(zram->disk->queue, PAGE_SIZE); + blk_queue_io_opt(zram->disk->queue, PAGE_SIZE); + zram->disk->queue->limits.discard_granularity = PAGE_SIZE; + blk_queue_max_discard_sectors(zram->disk->queue, UINT_MAX); + + /* + * zram_bio_discard() will clear all logical blocks if logical block + * size is identical with physical block size(PAGE_SIZE). But if it is + * different, we will skip discarding some parts of logical blocks in + * the part of the request range which isn't aligned to physical block + * size. So we can't ensure that all discarded logical blocks are + * zeroed. + */ + if (ZRAM_LOGICAL_BLOCK_SIZE == PAGE_SIZE) + blk_queue_max_write_zeroes_sectors(zram->disk->queue, UINT_MAX); + + blk_queue_flag_set(QUEUE_FLAG_STABLE_WRITES, zram->disk->queue); + ret = device_add_disk(NULL, zram->disk, zram_disk_groups); + if (ret) + goto out_cleanup_disk; + + pr_info("Added device: %s\n", zram->disk->disk_name); + return device_id; + +out_cleanup_disk: + put_disk(zram->disk); +out_free_idr: + idr_remove(&zram_index_idr, device_id); +out_free_dev: + kfree(zram); + return ret; +} + +static int zram_remove(struct zram *zram) +{ + bool claimed; + + mutex_lock(&zram->disk->open_mutex); + if (disk_openers(zram->disk)) { + mutex_unlock(&zram->disk->open_mutex); + return -EBUSY; + } + + claimed = zram->claim; + if (!claimed) + zram->claim = true; + mutex_unlock(&zram->disk->open_mutex); + + if (claimed) { + /* + * If we were claimed by reset_store(), del_gendisk() will + * wait until reset_store() is done, so nothing need to do. + */ + ; + } else { + /* Make sure all the pending I/O are finished */ + sync_blockdev(zram->disk->part0); + zram_reset_device(zram); + } + + pr_info("Removed device: %s\n", zram->disk->disk_name); + + del_gendisk(zram->disk); + + /* del_gendisk drains pending reset_store */ + WARN_ON_ONCE(claimed && zram->claim); + + /* + * disksize_store() may be called in between zram_reset_device() + * and del_gendisk(), so run the last reset to avoid leaking + * anything allocated with disksize_store() + */ + zram_reset_device(zram); + + put_disk(zram->disk); + kfree(zram); + return 0; +} + +/* zram-control sysfs attributes */ + +/* + * NOTE: hot_add attribute is not the usual read-only sysfs attribute. In a + * sense that reading from this file does alter the state of your system -- it + * creates a new un-initialized zram device and returns back this device's + * device_id (or an error code if it fails to create a new device). + */ +static ssize_t hot_add_show(const struct class *class, + const struct class_attribute *attr, + char *buf) +{ + int ret; + + mutex_lock(&zram_index_mutex); + ret = zram_add(); + mutex_unlock(&zram_index_mutex); + + if (ret < 0) + return ret; + return scnprintf(buf, PAGE_SIZE, "%d\n", ret); +} +/* This attribute must be set to 0400, so CLASS_ATTR_RO() can not be used */ +static struct class_attribute class_attr_hot_add = + __ATTR(hot_add, 0400, hot_add_show, NULL); + +static ssize_t hot_remove_store(const struct class *class, + const struct class_attribute *attr, + const char *buf, + size_t count) +{ + struct zram *zram; + int ret, dev_id; + + /* dev_id is gendisk->first_minor, which is `int' */ + ret = kstrtoint(buf, 10, &dev_id); + if (ret) + return ret; + if (dev_id < 0) + return -EINVAL; + + mutex_lock(&zram_index_mutex); + + zram = idr_find(&zram_index_idr, dev_id); + if (zram) { + ret = zram_remove(zram); + if (!ret) + idr_remove(&zram_index_idr, dev_id); + } else { + ret = -ENODEV; + } + + mutex_unlock(&zram_index_mutex); + return ret ? ret : count; +} +static CLASS_ATTR_WO(hot_remove); + +static struct attribute *zram_control_class_attrs[] = { + &class_attr_hot_add.attr, + &class_attr_hot_remove.attr, + NULL, +}; +ATTRIBUTE_GROUPS(zram_control_class); + +static struct class zram_control_class = { + .name = "zram-control", + .class_groups = zram_control_class_groups, +}; + +static int zram_remove_cb(int id, void *ptr, void *data) +{ + WARN_ON_ONCE(zram_remove(ptr)); + return 0; +} + +static void destroy_devices(void) +{ + class_unregister(&zram_control_class); + idr_for_each(&zram_index_idr, &zram_remove_cb, NULL); + idr_destroy(&zram_index_idr); + unregister_blkdev(zram_major, "zram"); + cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); +} + +static int __init zram_init(void) +{ + int ret; + + BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > BITS_PER_LONG); + + ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", + zcomp_cpu_up_prepare, zcomp_cpu_dead); + if (ret < 0) + return ret; + + ret = class_register(&zram_control_class); + if (ret) { + pr_err("Unable to register zram-control class\n"); + cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); + return ret; + } + + zram_major = register_blkdev(0, "zram"); + if (zram_major <= 0) { + pr_err("Unable to get major number\n"); + class_unregister(&zram_control_class); + cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); + return -EBUSY; + } + + while (num_devices != 0) { + mutex_lock(&zram_index_mutex); + ret = zram_add(); + mutex_unlock(&zram_index_mutex); + if (ret < 0) + goto out_error; + num_devices--; + } + + return 0; + +out_error: + destroy_devices(); + return ret; +} + +static void __exit zram_exit(void) +{ + destroy_devices(); +} + +module_init(zram_init); +module_exit(zram_exit); + +module_param(num_devices, uint, 0); +MODULE_PARM_DESC(num_devices, "Number of pre-created zram devices"); + +MODULE_LICENSE("Dual BSD/GPL"); +MODULE_AUTHOR("Nitin Gupta <ngupta@vflare.org>"); +MODULE_DESCRIPTION("Compressed RAM Block Device with multiple zspool support"); diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h new file mode 100644 index 000000000000..e320f395d121 --- /dev/null +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Compressed RAM block device with multiple zspool support + */ + +#ifndef _ZRAM_DRV_H_ +#define _ZRAM_DRV_H_ + +#include <linux/rwsem.h> +#include <linux/zsmalloc.h> +#include <linux/crypto.h> +#include <linux/memcontrol.h> + +#include "zcomp.h" + +#define SECTORS_PER_PAGE_SHIFT (PAGE_SHIFT - SECTOR_SHIFT) +#define SECTORS_PER_PAGE (1 << SECTORS_PER_PAGE_SHIFT) +#define ZRAM_LOGICAL_BLOCK_SHIFT 12 +#define ZRAM_LOGICAL_BLOCK_SIZE (1 << ZRAM_LOGICAL_BLOCK_SHIFT) +#define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ + (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) + + +/* + * ZRAM is mainly used for memory efficiency so we want to keep memory + * footprint small and thus squeeze size and zram pageflags into a flags + * member. The lower ZRAM_FLAG_SHIFT bits is for object size (excluding + * header), which cannot be larger than PAGE_SIZE (requiring PAGE_SHIFT + * bits), the higher bits are for zram_pageflags. + * + * We use BUILD_BUG_ON() to make sure that zram pageflags don't overflow. + */ +#define ZRAM_FLAG_SHIFT (PAGE_SHIFT + 1) + +/* Flags for zram pages (table[page_no].flags) */ +enum zram_pageflags { + /* zram slot is locked */ + ZRAM_LOCK = ZRAM_FLAG_SHIFT, + ZRAM_SAME, /* Page consists the same element */ + ZRAM_HUGE, /* Incompressible page */ + + __NR_ZRAM_PAGEFLAGS, +}; + +/*-- Data structures */ + +/* Allocated for each disk page */ +struct zram_table_entry { + union { + unsigned long handle; + unsigned long element; + }; + unsigned long flags; +}; + +struct zram_stats { + atomic64_t compr_data_size; /* compressed size of pages stored */ + atomic64_t failed_reads; /* can happen when memory is too low */ + atomic64_t failed_writes; /* can happen when memory is too low */ + atomic64_t notify_free; /* no. of swap slot free notifications */ + atomic64_t same_pages; /* no. of same element filled pages */ + atomic64_t huge_pages; /* no. of huge pages */ + atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ + atomic64_t pages_stored; /* no. of pages currently stored */ + atomic64_t writestall; /* no. of write slow paths */ + atomic64_t miss_free; /* no. of missed free */ +}; + +struct zram { + struct zram_table_entry *table; + struct zs_pool *mem_pool; + struct zcomp *comp; + struct gendisk *disk; + /* Prevent concurrent execution of device init */ + struct rw_semaphore init_lock; + + struct zram_stats stats; + /* + * This is the limit on amount of *uncompressed* worth of data + * we can store in a disk. + */ + u64 disksize; /* bytes */ + char compressor[CRYPTO_MAX_ALG_NAME]; + /* + * zram is claimed so open request will be failed + */ + bool claim; /* Protected by disk->open_mutex */ +}; +#endif diff --git a/drivers/block/multi_pool_zram/zcomp.c b/drivers/block/multi_pool_zram/zcomp.c new file mode 100644 index 000000000000..55af4efd7983 --- /dev/null +++ b/drivers/block/multi_pool_zram/zcomp.c @@ -0,0 +1,233 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2014 Sergey Senozhatsky. + */ + +#include <linux/kernel.h> +#include <linux/string.h> +#include <linux/err.h> +#include <linux/slab.h> +#include <linux/wait.h> +#include <linux/sched.h> +#include <linux/cpu.h> +#include <linux/crypto.h> + +#include "zcomp.h" + +static const char * const backends[] = { +#if IS_ENABLED(CONFIG_CRYPTO_LZO) + "lzo", + "lzo-rle", +#endif +#if IS_ENABLED(CONFIG_CRYPTO_LZ4) + "lz4", +#endif +#if IS_ENABLED(CONFIG_CRYPTO_LZ4HC) + "lz4hc", +#endif +#if IS_ENABLED(CONFIG_CRYPTO_842) + "842", +#endif +#if IS_ENABLED(CONFIG_CRYPTO_ZSTD) + "zstd", +#endif +}; + +static void zcomp_strm_free(struct zcomp_strm *zstrm) +{ + if (!IS_ERR_OR_NULL(zstrm->tfm)) + crypto_free_comp(zstrm->tfm); + free_pages((unsigned long)zstrm->buffer, 1); + zstrm->tfm = NULL; + zstrm->buffer = NULL; +} + +/* + * Initialize zcomp_strm structure with ->tfm initialized by backend, and + * ->buffer. Return a negative value on error. + */ +static int zcomp_strm_init(struct zcomp_strm *zstrm, struct zcomp *comp) +{ + zstrm->tfm = crypto_alloc_comp(comp->name, 0, 0); + /* + * allocate 2 pages. 1 for compressed data, plus 1 extra for the + * case when compressed size is larger than the original one + */ + zstrm->buffer = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 1); + if (IS_ERR_OR_NULL(zstrm->tfm) || !zstrm->buffer) { + zcomp_strm_free(zstrm); + return -ENOMEM; + } + return 0; +} + +bool zcomp_available_algorithm(const char *comp) +{ + /* + * Crypto does not ignore a trailing new line symbol, + * so make sure you don't supply a string containing + * one. + * This also means that we permit zcomp initialisation + * with any compressing algorithm known to crypto api. + */ + return crypto_has_comp(comp, 0, 0) == 1; +} + +/* show available compressors */ +ssize_t zcomp_available_show(const char *comp, char *buf) +{ + bool known_algorithm = false; + ssize_t sz = 0; + int i; + + for (i = 0; i < ARRAY_SIZE(backends); i++) { + if (!strcmp(comp, backends[i])) { + known_algorithm = true; + sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2, + "[%s] ", backends[i]); + } else { + sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2, + "%s ", backends[i]); + } + } + + /* + * Out-of-tree module known to crypto api or a missing + * entry in `backends'. + */ + if (!known_algorithm && crypto_has_comp(comp, 0, 0) == 1) + sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2, + "[%s] ", comp); + + sz += scnprintf(buf + sz, PAGE_SIZE - sz, "\n"); + return sz; +} + +struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) +{ + local_lock(&comp->stream->lock); + return this_cpu_ptr(comp->stream); +} + +void zcomp_stream_put(struct zcomp *comp) +{ + local_unlock(&comp->stream->lock); +} + +int zcomp_compress(struct zcomp_strm *zstrm, + const void *src, unsigned int *dst_len) +{ + /* + * Our dst memory (zstrm->buffer) is always `2 * PAGE_SIZE' sized + * because sometimes we can endup having a bigger compressed data + * due to various reasons: for example compression algorithms tend + * to add some padding to the compressed buffer. Speaking of padding, + * comp algorithm `842' pads the compressed length to multiple of 8 + * and returns -ENOSP when the dst memory is not big enough, which + * is not something that ZRAM wants to see. We can handle the + * `compressed_size > PAGE_SIZE' case easily in ZRAM, but when we + * receive -ERRNO from the compressing backend we can't help it + * anymore. To make `842' happy we need to tell the exact size of + * the dst buffer, zram_drv will take care of the fact that + * compressed buffer is too big. + */ + *dst_len = PAGE_SIZE * 2; + + return crypto_comp_compress(zstrm->tfm, + src, PAGE_SIZE, + zstrm->buffer, dst_len); +} + +int zcomp_decompress(struct zcomp_strm *zstrm, + const void *src, unsigned int src_len, void *dst) +{ + unsigned int dst_len = PAGE_SIZE; + + return crypto_comp_decompress(zstrm->tfm, + src, src_len, + dst, &dst_len); +} + +int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) +{ + struct zcomp *comp = hlist_entry(node, struct zcomp, node); + struct zcomp_strm *zstrm; + int ret; + + zstrm = per_cpu_ptr(comp->stream, cpu); + local_lock_init(&zstrm->lock); + + ret = zcomp_strm_init(zstrm, comp); + if (ret) + pr_err("Can't allocate a compression stream\n"); + return ret; +} + +int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) +{ + struct zcomp *comp = hlist_entry(node, struct zcomp, node); + struct zcomp_strm *zstrm; + + zstrm = per_cpu_ptr(comp->stream, cpu); + zcomp_strm_free(zstrm); + return 0; +} + +static int zcomp_init(struct zcomp *comp) +{ + int ret; + + comp->stream = alloc_percpu(struct zcomp_strm); + if (!comp->stream) + return -ENOMEM; + + ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); + if (ret < 0) + goto cleanup; + return 0; + +cleanup: + free_percpu(comp->stream); + return ret; +} + +void zcomp_destroy(struct zcomp *comp) +{ + cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node); + free_percpu(comp->stream); + kfree(comp); +} + +/* + * search available compressors for requested algorithm. + * allocate new zcomp and initialize it. return compressing + * backend pointer or ERR_PTR if things went bad. ERR_PTR(-EINVAL) + * if requested algorithm is not supported, ERR_PTR(-ENOMEM) in + * case of allocation error, or any other error potentially + * returned by zcomp_init(). + */ +struct zcomp *zcomp_create(const char *alg) +{ + struct zcomp *comp; + int error; + + /* + * Crypto API will execute /sbin/modprobe if the compression module + * is not loaded yet. We must do it here, otherwise we are about to + * call /sbin/modprobe under CPU hot-plug lock. + */ + if (!zcomp_available_algorithm(alg)) + return ERR_PTR(-EINVAL); + + comp = kzalloc(sizeof(struct zcomp), GFP_KERNEL); + if (!comp) + return ERR_PTR(-ENOMEM); + + comp->name = alg; + error = zcomp_init(comp); + if (error) { + kfree(comp); + return ERR_PTR(error); + } + return comp; +} diff --git a/drivers/block/multi_pool_zram/zcomp.h b/drivers/block/multi_pool_zram/zcomp.h new file mode 100644 index 000000000000..cdefdef93da8 --- /dev/null +++ b/drivers/block/multi_pool_zram/zcomp.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2014 Sergey Senozhatsky. + */ + +#ifndef _ZCOMP_H_ +#define _ZCOMP_H_ +#include <linux/local_lock.h> + +struct zcomp_strm { + /* The members ->buffer and ->tfm are protected by ->lock. */ + local_lock_t lock; + /* compression/decompression buffer */ + void *buffer; + struct crypto_comp *tfm; +}; + +/* dynamic per-device compression frontend */ +struct zcomp { + struct zcomp_strm __percpu *stream; + const char *name; + struct hlist_node node; +}; + +int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node); +int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node); +ssize_t zcomp_available_show(const char *comp, char *buf); +bool zcomp_available_algorithm(const char *comp); + +struct zcomp *zcomp_create(const char *alg); +void zcomp_destroy(struct zcomp *comp); + +struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); +void zcomp_stream_put(struct zcomp *comp); + +int zcomp_compress(struct zcomp_strm *zstrm, + const void *src, unsigned int *dst_len); + +int zcomp_decompress(struct zcomp_strm *zstrm, + const void *src, unsigned int src_len, void *dst); + +bool zcomp_set_max_streams(struct zcomp *comp, int num_strm); +#endif /* _ZCOMP_H_ */ -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- zram_slot_free_notify() tries to lock zram slot and may fail. In that case. the zspool memory allocated to the zram slot couldn't be freed, and then we couldn't unbind memcg for the zspool. So, we should guarantee this function frees memory successfully. commit 3c9959e02547 ("zram: fix lockdep warning of free block handling") uses zram_slot_trylock() in zram_slot_free_notify() to avoid deadlock, due to zram_slot_free_notify() could be called in softirq context by end_swap_bio_read(). However, after commit e914d8f00391 ("mm: fix unexpected zeroed page mapping with zram swap"), zram_slot_free_notify() couldn't be called in softirq context. So, use zram_slot_lock() in zram_slot_free_notify(). Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- drivers/block/multi_pool_zram/multi_pool_zram_drv.c | 13 ++----------- drivers/block/multi_pool_zram/multi_pool_zram_drv.h | 1 - 2 files changed, 2 insertions(+), 12 deletions(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 69a899ae98b5..987dadebaf07 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -48,11 +48,6 @@ static void zram_free_page(struct zram *zram, size_t index); static int zram_read_page(struct zram *zram, struct page *page, u32 index, struct bio *parent); -static int zram_slot_trylock(struct zram *zram, u32 index) -{ - return bit_spin_trylock(ZRAM_LOCK, &zram->table[index].flags); -} - static void zram_slot_lock(struct zram *zram, u32 index) { bit_spin_lock(ZRAM_LOCK, &zram->table[index].flags); @@ -265,7 +260,7 @@ static ssize_t debug_stat_show(struct device *dev, "version: %d\n%8llu %8llu\n", version, (u64)atomic64_read(&zram->stats.writestall), - (u64)atomic64_read(&zram->stats.miss_free)); + 0ULL); up_read(&zram->init_lock); return ret; @@ -705,11 +700,7 @@ static void zram_slot_free_notify(struct block_device *bdev, zram = bdev->bd_disk->private_data; atomic64_inc(&zram->stats.notify_free); - if (!zram_slot_trylock(zram, index)) { - atomic64_inc(&zram->stats.miss_free); - return; - } - + zram_slot_lock(zram, index); zram_free_page(zram, index); zram_slot_unlock(zram, index); } diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h index e320f395d121..7d96139a94df 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h @@ -63,7 +63,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic64_t writestall; /* no. of write slow paths */ - atomic64_t miss_free; /* no. of missed free */ }; struct zram { -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Prepare to support multiple pool. Currently, only zspool 0 is used. Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- .../multi_pool_zram/multi_pool_zram_drv.c | 72 +++++++++++++++---- .../multi_pool_zram/multi_pool_zram_drv.h | 14 +++- 2 files changed, 73 insertions(+), 13 deletions(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 987dadebaf07..a5f8a7ba799e 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -78,6 +78,18 @@ static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) zram->table[index].handle = handle; } +static struct zs_pool *zram_get_pool(struct zram *zram, u32 index) +{ + int pool_index = zram->table[index].pool_index; + + return zram->mem_pool[pool_index]; +} + +static void zram_set_pool_index(struct zram *zram, u32 index, int pool_index) +{ + zram->table[index].pool_index = pool_index; +} + /* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) @@ -265,6 +277,31 @@ static ssize_t debug_stat_show(struct device *dev, return ret; } +static void zram_pool_destroy(struct zram *zram) +{ + int i; + + for (i = 0; i < NR_ZPOOL; ++i) + if (zram->mem_pool[i]) + zs_destroy_pool(zram->mem_pool[i]); +} + +static int zram_pool_create(struct zram *zram) +{ + char name[64]; + int i; + + for (i = 0; i < NR_ZPOOL; ++i) { + snprintf(name, 64, "%s-%d", zram->disk->disk_name, i); + zram->mem_pool[i] = zs_create_pool(name); + if (!zram->mem_pool[i]) { + zram_pool_destroy(zram); + return -ENOMEM; + } + } + + return 0; +} static DEVICE_ATTR_RO(io_stat); static DEVICE_ATTR_RO(debug_stat); @@ -281,7 +318,7 @@ static void zram_meta_free(struct zram *zram, u64 disksize) for (index = 0; index < num_pages; index++) zram_free_page(zram, index); - zs_destroy_pool(zram->mem_pool); + zram_pool_destroy(zram); vfree(zram->table); zram->table = NULL; } @@ -290,20 +327,21 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) { size_t num_pages = disksize >> PAGE_SHIFT; unsigned long size = array_size(num_pages, sizeof(*zram->table)); + int ret; zram->table = vzalloc(size); if (!zram->table) return false; - zram->mem_pool = zs_create_pool(zram->disk->disk_name); - if (!zram->mem_pool) { + ret = zram_pool_create(zram); + if (ret) { vfree(zram->table); zram->table = NULL; return false; } if (!huge_class_size) - huge_class_size = zs_huge_class_size(zram->mem_pool); + huge_class_size = zs_huge_class_size(zram->mem_pool[DEFAULT_ZPOOL_INDEX]); return true; } @@ -315,6 +353,7 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; + struct zs_pool *pool; if (zram_test_flag(zram, index, ZRAM_HUGE)) { zram_clear_flag(zram, index, ZRAM_HUGE); @@ -335,13 +374,15 @@ static void zram_free_page(struct zram *zram, size_t index) if (!handle) return; - zs_free(zram->mem_pool, handle); + pool = zram_get_pool(zram, index); + zs_free(pool, handle); atomic64_sub(zram_get_obj_size(zram, index), &zram->stats.compr_data_size); out: atomic64_dec(&zram->stats.pages_stored); zram_set_handle(zram, index, 0); + zram_set_pool_index(zram, index, DEFAULT_ZPOOL_INDEX); zram_set_obj_size(zram, index, 0); WARN_ON_ONCE(zram->table[index].flags & ~(1UL << ZRAM_LOCK)); @@ -355,6 +396,7 @@ static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index) { struct zcomp_strm *zstrm; + struct zs_pool *pool; unsigned long handle; unsigned int size; void *src, *dst; @@ -372,12 +414,13 @@ static int zram_read_from_zspool(struct zram *zram, struct page *page, return 0; } + pool = zram_get_pool(zram, index); size = zram_get_obj_size(zram, index); if (size != PAGE_SIZE) zstrm = zcomp_stream_get(zram->comp); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_map_object(pool, handle, ZS_MM_RO); if (size == PAGE_SIZE) { dst = kmap_atomic(page); memcpy(dst, src, PAGE_SIZE); @@ -389,7 +432,7 @@ static int zram_read_from_zspool(struct zram *zram, struct page *page, kunmap_atomic(dst); zcomp_stream_put(zram->comp); } - zs_unmap_object(zram->mem_pool, handle); + zs_unmap_object(pool, handle); return ret; } @@ -446,6 +489,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) struct zcomp_strm *zstrm; unsigned long element = 0; enum zram_pageflags flags = 0; + struct zs_pool *pool; + int pool_index; mem = kmap_atomic(page); if (page_same_filled(mem, &element)) { @@ -457,6 +502,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } kunmap_atomic(mem); + pool_index = DEFAULT_ZPOOL_INDEX; + pool = zram->mem_pool[pool_index]; compress_again: zstrm = zcomp_stream_get(zram->comp); src = kmap_atomic(page); @@ -466,7 +513,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zram->comp); pr_err("Compression failed! err=%d\n", ret); - zs_free(zram->mem_pool, handle); + zs_free(pool, handle); return ret; } @@ -486,7 +533,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) * from the slow path and handle has already been allocated. */ if (IS_ERR_VALUE(handle)) - handle = zs_malloc(zram->mem_pool, comp_len, + handle = zs_malloc(pool, comp_len, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN | __GFP_HIGHMEM | @@ -494,7 +541,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zram->comp); atomic64_inc(&zram->stats.writestall); - handle = zs_malloc(zram->mem_pool, comp_len, + handle = zs_malloc(pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) @@ -512,7 +559,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zstrm = zcomp_stream_get(zram->comp); } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); + dst = zs_map_object(pool, handle, ZS_MM_WO); src = zstrm->buffer; if (comp_len == PAGE_SIZE) src = kmap_atomic(page); @@ -521,7 +568,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) kunmap_atomic(src); zcomp_stream_put(zram->comp); - zs_unmap_object(zram->mem_pool, handle); + zs_unmap_object(pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); out: /* @@ -542,6 +589,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zram_set_element(zram, index, element); } else { zram_set_handle(zram, index, handle); + zram_set_pool_index(zram, index, pool_index); zram_set_obj_size(zram, index, comp_len); } zram_slot_unlock(zram, index); diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h index 7d96139a94df..0304149b1697 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h @@ -20,6 +20,17 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) +/* + * the 1~256 zpools can be bounded to memcgs. The memory + * allocated by the zspool will be charged to be bounded + * memcg, and IO from the memcg will use the corresponding + * zspool. + * + * The first pool is used if write IO isn't bounded to memcg + * or the memcg isn't bounded to any zpool. + */ +#define NR_ZPOOL 257 +#define DEFAULT_ZPOOL_INDEX 0 /* * ZRAM is mainly used for memory efficiency so we want to keep memory @@ -50,6 +61,7 @@ struct zram_table_entry { unsigned long handle; unsigned long element; }; + int pool_index; unsigned long flags; }; @@ -67,7 +79,7 @@ struct zram_stats { struct zram { struct zram_table_entry *table; - struct zs_pool *mem_pool; + struct zs_pool *mem_pool[NR_ZPOOL]; struct zcomp *comp; struct gendisk *disk; /* Prevent concurrent execution of device init */ -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Add a sysfs file for each zspool to bind memcg. The file path is /sys/block/zramX/zs_pool/zspool-xxx/mem_cgroup. For zspool 0, this file isn't exported, because zspool 0 is used if write IO isn't bounded to memcg or the memcg isn't bounded to any zpool. Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- .../multi_pool_zram/multi_pool_zram_drv.c | 213 +++++++++++++++++- .../multi_pool_zram/multi_pool_zram_drv.h | 13 ++ include/linux/swap_slots.h | 4 + include/linux/zsmalloc.h | 1 + mm/swap_slots.c | 9 + mm/zsmalloc.c | 3 +- 6 files changed, 241 insertions(+), 2 deletions(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index a5f8a7ba799e..9890afa6e4ee 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -24,6 +24,7 @@ #include <linux/debugfs.h> #include <linux/cpuhotplug.h> #include <linux/part_stat.h> +#include <linux/swap_slots.h> #include "multi_pool_zram_drv.h" @@ -133,6 +134,63 @@ static void zram_set_obj_size(struct zram *zram, zram->table[index].flags = (flags << ZRAM_FLAG_SHIFT) | size; } +static int zspool_bind_memcg(struct zram *zram, int pool_index, + struct mem_cgroup *memcg) +{ + bool drained = false; + int ret = -EBUSY; + + down_write(&zram->init_lock); + if (!init_done(zram)) { + if (zram->pool_memcg[pool_index]) + css_put(&zram->pool_memcg[pool_index]->css); + zram->pool_memcg[pool_index] = memcg; + ret = 0; + goto out_uninited; + } + +restart: + down_write(&zram->rebind_memcg_lock); + spin_lock(&zram->rebind_memcg_spinlock); + if (zs_get_total_pages(zram->mem_pool[pool_index]) != 0) { + if (!drained) { + spin_unlock(&zram->rebind_memcg_spinlock); + up_write(&zram->rebind_memcg_lock); + drain_swap_slots_cache(); + drained = true; + goto restart; + } + goto out; + } + + if (zram->pool_memcg[pool_index]) + css_put(&zram->pool_memcg[pool_index]->css); + zram->pool_memcg[pool_index] = memcg; + zs_set_memcg(zram->mem_pool[pool_index], memcg); + ret = 0; +out: + spin_unlock(&zram->rebind_memcg_spinlock); + up_write(&zram->rebind_memcg_lock); +out_uninited: + up_write(&zram->init_lock); + return ret; +} + +static void reset_memcg(struct zram *zram) +{ + int i; + + for (i = 0; i < NR_ZPOOL; i++) { + if (i == DEFAULT_ZPOOL_INDEX) + continue; + + if (zram->pool_memcg[i]) { + css_put(&zram->pool_memcg[i]->css); + zram->pool_memcg[i] = NULL; + } + } +} + #if PAGE_SIZE != 4096 static inline bool is_partial_io(struct bio_vec *bvec) { @@ -298,7 +356,10 @@ static int zram_pool_create(struct zram *zram) zram_pool_destroy(zram); return -ENOMEM; } + zs_set_memcg(zram->mem_pool[i], zram->pool_memcg[i]); } + init_rwsem(&zram->rebind_memcg_lock); + spin_lock_init(&zram->rebind_memcg_spinlock); return 0; } @@ -723,6 +784,7 @@ static void zram_submit_bio(struct bio *bio) { struct zram *zram = bio->bi_bdev->bd_disk->private_data; + down_read(&zram->rebind_memcg_lock); switch (bio_op(bio)) { case REQ_OP_READ: zram_bio_read(zram, bio); @@ -738,6 +800,7 @@ static void zram_submit_bio(struct bio *bio) WARN_ON_ONCE(1); bio_endio(bio); } + up_read(&zram->rebind_memcg_lock); } static void zram_slot_free_notify(struct block_device *bdev, @@ -746,11 +809,12 @@ static void zram_slot_free_notify(struct block_device *bdev, struct zram *zram; zram = bdev->bd_disk->private_data; - atomic64_inc(&zram->stats.notify_free); + spin_lock(&zram->rebind_memcg_spinlock); zram_slot_lock(zram, index); zram_free_page(zram, index); zram_slot_unlock(zram, index); + spin_unlock(&zram->rebind_memcg_spinlock); } static void zram_reset_device(struct zram *zram) @@ -769,6 +833,7 @@ static void zram_reset_device(struct zram *zram) } memset(&zram->stats, 0, sizeof(zram->stats)); strscpy(zram->compressor, default_compressor, sizeof(zram->compressor)); + reset_memcg(zram); up_write(&zram->init_lock); } @@ -895,6 +960,145 @@ static struct attribute *zram_disk_attrs[] = { ATTRIBUTE_GROUPS(zram_disk); +static int kobj_to_pool_index(struct zram *zram, struct kobject *kobj) +{ + int i; + + for (i = 0; i < NR_ZPOOL; i++) + if (zram->pool_kobj[i] == kobj) + return i; + + VM_BUG_ON(1); + return 0; +} + +static ssize_t mem_cgroup_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct kobject *kobj = &dev->kobj; + struct mem_cgroup *memcg; + struct zram *zram; + int pool_index; + + zram = dev_to_zram(kobj_to_dev(kobj->parent->parent)); + pool_index = kobj_to_pool_index(zram, kobj); + memcg = zram->pool_memcg[pool_index]; + + if (!memcg) + return scnprintf(buf, PAGE_SIZE, "none\n"); + + if (!cgroup_path(memcg->css.cgroup, buf, PATH_MAX)) + return scnprintf(buf, PAGE_SIZE, "none\n"); + + return scnprintf(buf, PAGE_SIZE, "%s\n", buf); +} + +static ssize_t mem_cgroup_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t len) +{ + struct kobject *kobj = &dev->kobj; + struct mem_cgroup *memcg; + struct zram *zram; + int pool_index; + char *kbuf; + size_t sz; + int ret = 0; + + kbuf = kmalloc(PATH_MAX, GFP_KERNEL); + if (!kbuf) + return -ENOMEM; + + strlcpy(kbuf, buf, PATH_MAX); + sz = strlen(kbuf); + if (sz > 0 && kbuf[sz - 1] == '\n') + kbuf[sz - 1] = 0x00; + + if (!strcmp(kbuf, "none")) { + memcg = NULL; + } else { + memcg = memcg_get_from_path(kbuf, PATH_MAX); + if (!memcg) { + ret = -EINVAL; + goto out; + } + } + + zram = dev_to_zram(kobj_to_dev(kobj->parent->parent)); + pool_index = kobj_to_pool_index(zram, kobj); + ret = zspool_bind_memcg(zram, pool_index, memcg); + if (ret && memcg) + css_put(&memcg->css); +out: + kfree(kbuf); + return ret ? ret : len; +} +static DEVICE_ATTR_RW(mem_cgroup); + +static struct attribute *mem_pool_attrs[] = { + &dev_attr_mem_cgroup.attr, + NULL, +}; + +static const struct attribute_group mem_pool_attr_group = { + .attrs = mem_pool_attrs, +}; + +static void zram_pool_kobject_destroy(struct zram *zram) +{ + int i; + + for (i = 0; i < NR_ZPOOL; i++) { + if (i == DEFAULT_ZPOOL_INDEX) + continue; + if (!zram->pool_kobj[i]) + continue; + + sysfs_remove_group(zram->pool_kobj[i], &mem_pool_attr_group); + kobject_put(zram->pool_kobj[i]); + zram->pool_kobj[i] = NULL; + } + + if (zram->pool_parent_kobj) { + kobject_put(zram->pool_parent_kobj); + zram->pool_parent_kobj = NULL; + } +} + +static int zram_pool_kobject_create(struct zram *zram) +{ + char name[64]; + int ret, i; + + zram->pool_parent_kobj = kobject_create_and_add("zs_pool", + &disk_to_dev(zram->disk)->kobj); + if (!zram->pool_parent_kobj) + return -ENOMEM; + + for (i = 0; i < NR_ZPOOL; i++) { + if (i == DEFAULT_ZPOOL_INDEX) + continue; + + snprintf(name, 64, "zspool-%03d", i); + zram->pool_kobj[i] = kobject_create_and_add(name, zram->pool_parent_kobj); + if (!zram->pool_kobj[i]) { + ret = -ENOMEM; + goto err; + } + + ret = sysfs_create_group(zram->pool_kobj[i], &mem_pool_attr_group); + if (ret) { + kobject_put(zram->pool_kobj[i]); + zram->pool_kobj[i] = NULL; + goto err; + } + } + + return 0; +err: + zram_pool_kobject_destroy(zram); + return ret; +} + /* * Allocate and initialize new zram device. the function returns * '>= 0' device_id upon success, and negative value otherwise. @@ -967,9 +1171,15 @@ static int zram_add(void) if (ret) goto out_cleanup_disk; + ret = zram_pool_kobject_create(zram); + if (ret < 0) + goto out_del_disk; + pr_info("Added device: %s\n", zram->disk->disk_name); return device_id; +out_del_disk: + del_gendisk(zram->disk); out_cleanup_disk: put_disk(zram->disk); out_free_idr: @@ -1019,6 +1229,7 @@ static int zram_remove(struct zram *zram) * anything allocated with disksize_store() */ zram_reset_device(zram); + zram_pool_kobject_destroy(zram); put_disk(zram->disk); kfree(zram); diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h index 0304149b1697..e8298e9cfd7b 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h @@ -80,6 +80,16 @@ struct zram_stats { struct zram { struct zram_table_entry *table; struct zs_pool *mem_pool[NR_ZPOOL]; + /* + * Use the below two locks to serialize zspool update + * and use. + * + * zram_slot_free_notify() may be called with holding + * spinlock, so can't use rebind_memcg_lock, use + * rebind_memcg_spinlock instead. + */ + struct rw_semaphore rebind_memcg_lock; + spinlock_t rebind_memcg_spinlock; struct zcomp *comp; struct gendisk *disk; /* Prevent concurrent execution of device init */ @@ -96,5 +106,8 @@ struct zram { * zram is claimed so open request will be failed */ bool claim; /* Protected by disk->open_mutex */ + struct kobject *pool_parent_kobj; + struct kobject *pool_kobj[NR_ZPOOL]; + struct mem_cgroup *pool_memcg[NR_ZPOOL]; }; #endif diff --git a/include/linux/swap_slots.h b/include/linux/swap_slots.h index 77521ac11dca..ccd6753af7b5 100644 --- a/include/linux/swap_slots.h +++ b/include/linux/swap_slots.h @@ -26,6 +26,10 @@ void reenable_swap_slots_cache_unlock(void); void enable_swap_slots_cache(int type); void free_swap_slot(swp_entry_t entry); +#ifdef CONFIG_MEMCG_ZRAM +void drain_swap_slots_cache(void); +#endif + extern bool swap_slot_cache_enabled; #endif /* _LINUX_SWAP_SLOTS_H */ diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 2c09676f9178..9a5d6c24d62d 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -43,6 +43,7 @@ struct zs_pool; struct zs_pool *zs_create_pool(const char *name); #ifdef CONFIG_MEMCG_ZRAM struct zs_pool *zs_create_pool_with_memcg(const char *name, void *memcg); +void zs_set_memcg(struct zs_pool *pool, void *memcg); #endif void zs_destroy_pool(struct zs_pool *pool); diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 7af3b93d4c8c..04c22efee496 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -93,6 +93,15 @@ void reenable_swap_slots_cache_unlock(void) mutex_unlock(&swap_slots_cache_enable_mutex); } +#ifdef CONFIG_MEMCG_ZRAM +void drain_swap_slots_cache(void) +{ + disable_swap_slots_cache_lock(); + reenable_swap_slots_cache_unlock(); +} +EXPORT_SYMBOL_GPL(drain_swap_slots_cache); +#endif + static bool check_cache_active(void) { long pages; diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 1fb34d1dd563..b106b4f9981b 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -2280,11 +2280,12 @@ struct zs_pool *zs_create_pool(const char *name) EXPORT_SYMBOL_GPL(zs_create_pool); #ifdef CONFIG_MEMCG_ZRAM -static inline void zs_set_memcg(struct zs_pool *pool, void *memcg) +void zs_set_memcg(struct zs_pool *pool, void *memcg) { if (pool) pool->memcg = memcg; } +EXPORT_SYMBOL_GPL(zs_set_memcg); struct zs_pool *zs_create_pool_with_memcg(const char *name, void *memcg) { -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- If the page to write comes from memcg and the memcg is bounded with a zspool, use the zspool. Otherwise, use zspool 0. Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- .../multi_pool_zram/multi_pool_zram_drv.c | 29 ++++++++++++++++++- include/linux/memcontrol.h | 2 ++ mm/memcontrol.c | 18 ++++++++++++ 3 files changed, 48 insertions(+), 1 deletion(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 9890afa6e4ee..c3f29d44a9fc 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -191,6 +191,33 @@ static void reset_memcg(struct zram *zram) } } +static int get_pool_index_from_page(struct zram *zram, struct page *page) +{ + struct folio *folio = page_folio(page); + struct mem_cgroup *memcg; + int res = DEFAULT_ZPOOL_INDEX; + int i; + + folio_memcg_lock(folio); + memcg = page_memcg(page); + if (!memcg) + goto out; + + for (i = 0; i < NR_ZPOOL; i++) { + if (i == DEFAULT_ZPOOL_INDEX) + continue; + + if (multi_pool_zram_is_target_memcg(memcg, zram->pool_memcg[i])) { + res = i; + break; + } + } + +out: + folio_memcg_unlock(folio); + return res; +} + #if PAGE_SIZE != 4096 static inline bool is_partial_io(struct bio_vec *bvec) { @@ -563,7 +590,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } kunmap_atomic(mem); - pool_index = DEFAULT_ZPOOL_INDEX; + pool_index = get_pool_index_from_page(zram, page); pool = zram->mem_pool[pool_index]; compress_again: zstrm = zcomp_stream_get(zram->comp); diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index c0e30821c94b..dcc42b60bc7a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1326,6 +1326,8 @@ int mem_cgroup_force_empty(struct mem_cgroup *memcg); struct mem_cgroup *memcg_get_from_path(char *path, size_t buflen); void memcg_charge_zram(struct mem_cgroup *memcg, unsigned int nr_pages); void memcg_uncharge_zram(struct mem_cgroup *memcg, unsigned int nr_pages); +bool multi_pool_zram_is_target_memcg(struct mem_cgroup *memcg, + struct mem_cgroup *target_memcg); #endif #else /* CONFIG_MEMCG */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 960028758d92..feb48eccf578 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2234,6 +2234,7 @@ void folio_memcg_lock(struct folio *folio) memcg->move_lock_task = current; memcg->move_lock_flags = flags; } +EXPORT_SYMBOL(folio_memcg_lock); static void __folio_memcg_unlock(struct mem_cgroup *memcg) { @@ -2261,6 +2262,7 @@ void folio_memcg_unlock(struct folio *folio) { __folio_memcg_unlock(folio_memcg(folio)); } +EXPORT_SYMBOL(folio_memcg_unlock); struct memcg_stock_pcp { local_lock_t stock_lock; @@ -4205,6 +4207,22 @@ static u64 mem_cgroup_zram_usage(struct cgroup_subsys_state *css, return (u64)atomic64_read(&memcg->zram_usage) * PAGE_SIZE; } + +bool multi_pool_zram_is_target_memcg(struct mem_cgroup *memcg, + struct mem_cgroup *target_memcg) +{ + int ret = false; + + rcu_read_lock(); + for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) + if (memcg == target_memcg) { + ret = true; + break; + } + rcu_read_unlock(); + return ret; +} +EXPORT_SYMBOL(multi_pool_zram_is_target_memcg); #else static inline void memcg_zram_usage_init(struct mem_cgroup *memcg) { -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Add mm_stat interface. Compared to zram, mem_used_total and pages_compacted are calcualted with multiple zspool. mem_limit and mem_used_max aren't supported, and always displayed as 0. Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- .../multi_pool_zram/multi_pool_zram_drv.c | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index c3f29d44a9fc..5df356ad6759 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -362,6 +362,64 @@ static ssize_t debug_stat_show(struct device *dev, return ret; } + +static u64 zram_pool_get_total_pages(struct zram *zram) +{ + u64 mem_used = 0; + int i; + + for (i = 0; i < NR_ZPOOL; ++i) + mem_used += zs_get_total_pages(zram->mem_pool[i]); + + return mem_used; +} + +static long zram_pool_get_pages_compacted(struct zram *zram) +{ + struct zs_pool_stats pool_stats; + long pages_compacted = 0; + int i; + + for (i = 0; i < NR_ZPOOL; ++i) { + zs_pool_stats(zram->mem_pool[i], &pool_stats); + pages_compacted += atomic_long_read(&pool_stats.pages_compacted); + } + + return pages_compacted; +} + +static ssize_t mm_stat_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct zram *zram = dev_to_zram(dev); + u64 orig_size, mem_used = 0; + long pages_compacted = 0; + ssize_t ret; + + down_read(&zram->init_lock); + if (init_done(zram)) { + mem_used = zram_pool_get_total_pages(zram); + pages_compacted = zram_pool_get_pages_compacted(zram); + } + + orig_size = atomic64_read(&zram->stats.pages_stored); + + ret = scnprintf(buf, PAGE_SIZE, + "%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu %8llu\n", + orig_size << PAGE_SHIFT, + (u64)atomic64_read(&zram->stats.compr_data_size), + mem_used << PAGE_SHIFT, + 0L, + 0L, + (u64)atomic64_read(&zram->stats.same_pages), + pages_compacted, + (u64)atomic64_read(&zram->stats.huge_pages), + (u64)atomic64_read(&zram->stats.huge_pages_since)); + up_read(&zram->init_lock); + + return ret; +} + static void zram_pool_destroy(struct zram *zram) { int i; @@ -392,6 +450,7 @@ static int zram_pool_create(struct zram *zram) } static DEVICE_ATTR_RO(io_stat); +static DEVICE_ATTR_RO(mm_stat); static DEVICE_ATTR_RO(debug_stat); static void zram_meta_free(struct zram *zram, u64 disksize) @@ -981,6 +1040,7 @@ static struct attribute *zram_disk_attrs[] = { &dev_attr_reset.attr, &dev_attr_comp_algorithm.attr, &dev_attr_io_stat.attr, + &dev_attr_mm_stat.attr, &dev_attr_debug_stat.attr, NULL, }; -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Add mm_stat interface for each zspool, the format is same to the global interface. Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- .../multi_pool_zram/multi_pool_zram_drv.c | 74 +++++++++++++++++-- .../multi_pool_zram/multi_pool_zram_drv.h | 9 +++ 2 files changed, 77 insertions(+), 6 deletions(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 5df356ad6759..08987c974a70 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -499,12 +499,19 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) */ static void zram_free_page(struct zram *zram, size_t index) { + struct zram_zspool_stats *zspool_stats = NULL; unsigned long handle; struct zs_pool *pool; + size_t obj_size; + int pool_index; + pool_index = zram->table[index].pool_index; + pool = zram->mem_pool[pool_index]; + zspool_stats = &zram->zspool_stats[pool_index]; if (zram_test_flag(zram, index, ZRAM_HUGE)) { zram_clear_flag(zram, index, ZRAM_HUGE); atomic64_dec(&zram->stats.huge_pages); + atomic64_dec(&zspool_stats->huge_pages); } /* @@ -514,6 +521,7 @@ static void zram_free_page(struct zram *zram, size_t index) if (zram_test_flag(zram, index, ZRAM_SAME)) { zram_clear_flag(zram, index, ZRAM_SAME); atomic64_dec(&zram->stats.same_pages); + atomic64_dec(&zspool_stats->same_pages); goto out; } @@ -521,13 +529,14 @@ static void zram_free_page(struct zram *zram, size_t index) if (!handle) return; - pool = zram_get_pool(zram, index); zs_free(pool, handle); - atomic64_sub(zram_get_obj_size(zram, index), - &zram->stats.compr_data_size); + obj_size = zram_get_obj_size(zram, index); + atomic64_sub(obj_size, &zram->stats.compr_data_size); + atomic64_sub(obj_size, &zspool_stats->compr_data_size); out: atomic64_dec(&zram->stats.pages_stored); + atomic64_dec(&zspool_stats->pages_stored); zram_set_handle(zram, index, 0); zram_set_pool_index(zram, index, DEFAULT_ZPOOL_INDEX); zram_set_obj_size(zram, index, 0); @@ -636,21 +645,24 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) struct zcomp_strm *zstrm; unsigned long element = 0; enum zram_pageflags flags = 0; + struct zram_zspool_stats *zspool_stats = NULL; struct zs_pool *pool; int pool_index; + pool_index = get_pool_index_from_page(zram, page); + pool = zram->mem_pool[pool_index]; + zspool_stats = &zram->zspool_stats[pool_index]; mem = kmap_atomic(page); if (page_same_filled(mem, &element)) { kunmap_atomic(mem); /* Free memory associated with this sector now. */ flags = ZRAM_SAME; atomic64_inc(&zram->stats.same_pages); + atomic64_inc(&zspool_stats->same_pages); goto out; } kunmap_atomic(mem); - pool_index = get_pool_index_from_page(zram, page); - pool = zram->mem_pool[pool_index]; compress_again: zstrm = zcomp_stream_get(zram->comp); src = kmap_atomic(page); @@ -717,6 +729,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zcomp_stream_put(zram->comp); zs_unmap_object(pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); + atomic64_add(comp_len, &zspool_stats->compr_data_size); out: /* * Free memory associated with this sector @@ -728,7 +741,9 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (comp_len == PAGE_SIZE) { zram_set_flag(zram, index, ZRAM_HUGE); atomic64_inc(&zram->stats.huge_pages); + atomic64_inc(&zspool_stats->huge_pages); atomic64_inc(&zram->stats.huge_pages_since); + atomic64_inc(&zspool_stats->huge_pages_since); } if (flags) { @@ -736,13 +751,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zram_set_element(zram, index, element); } else { zram_set_handle(zram, index, handle); - zram_set_pool_index(zram, index, pool_index); zram_set_obj_size(zram, index, comp_len); } + zram_set_pool_index(zram, index, pool_index); zram_slot_unlock(zram, index); /* Update stats */ atomic64_inc(&zram->stats.pages_stored); + atomic64_inc(&zspool_stats->pages_stored); return ret; } @@ -1121,8 +1137,54 @@ static ssize_t mem_cgroup_store(struct device *dev, } static DEVICE_ATTR_RW(mem_cgroup); +static ssize_t zspool_mm_stat_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct zram_zspool_stats *zram_zspool_stats; + struct kobject *kobj = &dev->kobj; + struct zs_pool_stats pool_stats; + u64 orig_size, mem_used = 0; + struct zs_pool *pool; + struct zram *zram; + int pool_index; + ssize_t ret; + + zram = dev_to_zram(kobj_to_dev(kobj->parent->parent)); + pool_index = kobj_to_pool_index(zram, kobj); + pool = zram->mem_pool[pool_index]; + zram_zspool_stats = &zram->zspool_stats[pool_index]; + memset(&pool_stats, 0x0, sizeof(struct zs_pool_stats)); + + down_read(&zram->init_lock); + if (init_done(zram)) { + mem_used = zs_get_total_pages(pool); + zs_pool_stats(pool, &pool_stats); + } + + orig_size = atomic64_read(&zram_zspool_stats->pages_stored); + ret = scnprintf(buf, PAGE_SIZE, + "%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu %8llu\n", + orig_size << PAGE_SHIFT, + (u64)atomic64_read(&zram_zspool_stats->compr_data_size), + mem_used << PAGE_SHIFT, + 0L, 0L, + (u64)atomic64_read(&zram_zspool_stats->same_pages), + atomic_long_read(&pool_stats.pages_compacted), + (u64)atomic64_read(&zram_zspool_stats->huge_pages), + (u64)atomic64_read(&zram_zspool_stats->huge_pages_since)); + up_read(&zram->init_lock); + + return ret; +} + +static struct device_attribute dev_attr_zspool_mm_stat = { + .attr = { .name = "mm_stat", .mode = 0444 }, + .show = zspool_mm_stat_show, +}; + static struct attribute *mem_pool_attrs[] = { &dev_attr_mem_cgroup.attr, + &dev_attr_zspool_mm_stat.attr, NULL, }; diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h index e8298e9cfd7b..7c192e81a3de 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.h +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.h @@ -77,6 +77,14 @@ struct zram_stats { atomic64_t writestall; /* no. of write slow paths */ }; +struct zram_zspool_stats { + atomic64_t compr_data_size; /* compressed size of pages stored */ + atomic64_t pages_stored; /* no. of pages currently stored */ + atomic64_t same_pages; /* no. of same element filled pages */ + atomic64_t huge_pages; /* no. of huge pages */ + atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ +}; + struct zram { struct zram_table_entry *table; struct zs_pool *mem_pool[NR_ZPOOL]; @@ -109,5 +117,6 @@ struct zram { struct kobject *pool_parent_kobj; struct kobject *pool_kobj[NR_ZPOOL]; struct mem_cgroup *pool_memcg[NR_ZPOOL]; + struct zram_zspool_stats zspool_stats[NR_ZPOOL]; }; #endif -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Even if all processes that use the zspool exits, some swapcache pages may still exist and occupy zspool memory. When unbinding memcg for the zspool, the zspool should be empty. If the zspool memory is still used, try to reclaim swapcache pages. Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- .../multi_pool_zram/multi_pool_zram_drv.c | 24 +++++++ include/linux/swap.h | 10 +++ mm/swapfile.c | 65 +++++++++++++++++++ 3 files changed, 99 insertions(+) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 08987c974a70..732a084539cc 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -134,9 +134,26 @@ static void zram_set_obj_size(struct zram *zram, zram->table[index].flags = (flags << ZRAM_FLAG_SHIFT) | size; } +struct zram_is_used_arg { + struct zram *zram; + int pool_index; +}; + +static bool zram_is_used(void *arg) +{ + struct zram_is_used_arg *zram_arg = arg; + struct zram *zram = zram_arg->zram; + int pool_index = zram_arg->pool_index; + + return (zs_get_total_pages(zram->mem_pool[pool_index]) != 0) + || (atomic64_read(&zram->zspool_stats[pool_index].same_pages) != 0); +} + static int zspool_bind_memcg(struct zram *zram, int pool_index, struct mem_cgroup *memcg) { + struct zram_is_used_arg arg = {.zram = zram, .pool_index = pool_index}; + bool swapcache_reclaimed = false; bool drained = false; int ret = -EBUSY; @@ -160,6 +177,13 @@ static int zspool_bind_memcg(struct zram *zram, int pool_index, drained = true; goto restart; } + if (!swapcache_reclaimed) { + spin_unlock(&zram->rebind_memcg_spinlock); + up_write(&zram->rebind_memcg_lock); + try_to_reclaim_swap_cache(zram->disk->part0->bd_inode, zram_is_used, &arg); + swapcache_reclaimed = true; + goto restart; + } goto out; } diff --git a/include/linux/swap.h b/include/linux/swap.h index d31b74bd7eb1..1e07f2e23104 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -733,5 +733,15 @@ static inline bool mem_cgroup_swap_full(struct folio *folio) } #endif +struct zs_pool; +#ifdef CONFIG_MEMCG_ZRAM +void try_to_reclaim_swap_cache(struct inode *inode, bool (*continue_reclaim)(void *arg), void *arg); +#else +static inline +void try_to_reclaim_swap_cache(struct inode *inode, bool (*continue_reclaim)(void *arg), void *arg) +{ +} +#endif + #endif /* __KERNEL__*/ #endif /* _LINUX_SWAP_H */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 889a22c23a94..1b9d60f6ebb3 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3887,6 +3887,71 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp) } #endif +#ifdef CONFIG_MEMCG_ZRAM +static unsigned int find_next_to_reclaim(struct swap_info_struct *si, + unsigned int prev) +{ + unsigned int i; + unsigned char count; + + for (i = prev + 1; i < si->max; i++) { + count = READ_ONCE(si->swap_map[i]); + if (count == SWAP_HAS_CACHE) + break; + if ((i % LATENCY_LIMIT) == 0) + cond_resched(); + } + + if (i == si->max) + i = 0; + + return i; +} + +/* Refer to frontswap_shrink(). */ +void try_to_reclaim_swap_cache(struct inode *inode, bool (*continue_reclaim)(void *arg), void *arg) +{ + struct swap_info_struct *si = NULL; + unsigned int i = 0; + struct folio *folio; + bool found = false; + swp_entry_t entry; + int type; + + spin_lock(&swap_lock); + plist_for_each_entry(si, &swap_active_head, list) { + if (si->swap_file->f_mapping == inode->i_mapping) { + found = true; + type = si->type; + break; + } + } + spin_unlock(&swap_lock); + if (!found) + return; + + disable_swap_slots_cache_lock(); + while (READ_ONCE(si->inuse_pages) && continue_reclaim(arg)) { + i = find_next_to_reclaim(si, i); + if (!i) + break; + + entry = swp_entry(type, i); + folio = filemap_get_folio(swap_address_space(entry), i); + if (IS_ERR(folio)) + continue; + + folio_lock(folio); + folio_wait_writeback(folio); + folio_free_swap(folio); + folio_unlock(folio); + folio_put(folio); + } + reenable_swap_slots_cache_unlock(); +} +EXPORT_SYMBOL_GPL(try_to_reclaim_swap_cache); +#endif + static int __init swapfile_init(void) { int nid; -- 2.43.0
hulk inclusion category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Currently, zram usage is only charged in memory.usage_in_bytes, and isn't charged in memory.memsw.usage_in_bytes. This leads to memory.swap.max fail to restrict the swap usage, because memsw - memory is smaller than swap usage. To fix it, charge zram usage to memory.memsw.usage_in_bytes too. Fixes: 03cbd28fc3ca ("[Huawei] zram: charge zram usage in memory cgroup") Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- mm/memcontrol.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index feb48eccf578..0d0a3a0c8bd9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4183,6 +4183,7 @@ void memcg_charge_zram(struct mem_cgroup *memcg, unsigned int nr_pages) return; page_counter_charge(&memcg->memory, nr_pages); + page_counter_charge(&memcg->memsw, nr_pages); atomic_long_add(nr_pages, &memcg->zram_usage); } EXPORT_SYMBOL_GPL(memcg_charge_zram); @@ -4196,6 +4197,7 @@ void memcg_uncharge_zram(struct mem_cgroup *memcg, unsigned int nr_pages) return; page_counter_uncharge(&memcg->memory, nr_pages); + page_counter_uncharge(&memcg->memsw, nr_pages); atomic_long_sub(nr_pages, &memcg->zram_usage); } EXPORT_SYMBOL_GPL(memcg_uncharge_zram); -- 2.43.0
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- Enable CONFIG_MULTI_POOL_ZRAM by default. Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- arch/arm64/configs/openeuler_defconfig | 1 + arch/x86/configs/openeuler_defconfig | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index 100af4496175..87a077c87b89 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -2477,6 +2477,7 @@ CONFIG_ZRAM_DEF_COMP="lzo-rle" CONFIG_ZRAM_WRITEBACK=y # CONFIG_ZRAM_MEMORY_TRACKING is not set CONFIG_ZRAM_MULTI_COMP=y +CONFIG_MULTI_POOL_ZRAM=m CONFIG_BLK_DEV_LOOP=m CONFIG_BLK_DEV_LOOP_MIN_COUNT=0 CONFIG_BLK_DEV_DRBD=m diff --git a/arch/x86/configs/openeuler_defconfig b/arch/x86/configs/openeuler_defconfig index 4beff1733c18..0d3146960f77 100644 --- a/arch/x86/configs/openeuler_defconfig +++ b/arch/x86/configs/openeuler_defconfig @@ -2417,6 +2417,7 @@ CONFIG_ZRAM_DEF_COMP="lzo-rle" CONFIG_ZRAM_WRITEBACK=y # CONFIG_ZRAM_MEMORY_TRACKING is not set CONFIG_ZRAM_MULTI_COMP=y +CONFIG_MULTI_POOL_ZRAM=m CONFIG_BLK_DEV_LOOP=m CONFIG_BLK_DEV_LOOP_MIN_COUNT=0 # CONFIG_BLK_DEV_DRBD is not set -- 2.43.0
hulk inclusion category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- When updating comp_algorithm successfully, comp_algorithm_store() should return len intead of zero. Otherwise, echo zstd > comp_algorithm will retry infinitely. Fixes: 82ca43d5a1c3 ("[Huawei] multi_pool_zram: copy from zram and delete unused code") Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- drivers/block/multi_pool_zram/multi_pool_zram_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 732a084539cc..85534cede5a3 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -335,7 +335,7 @@ static ssize_t comp_algorithm_store(struct device *dev, strcpy(zram->compressor, compressor); up_write(&zram->init_lock); - return 0; + return len; } static ssize_t comp_algorithm_show(struct device *dev, -- 2.43.0
hulk inclusion category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQ ---------------------------------------- zspool_mm_stat_show() takes pool without init_lock held, it may race with disksize_store(), and lead to NULL-ptr dereference. The race is as follows. CPU0 (zspool_mm_stat_show) CPU1 (disksize_store) pool = zram->mem_pool[pool_index]; // NULL zram_meta_alloc() zram->disksize = disksize; if (init_done(zram)) // true zs_get_total_pages(pool);// NULL-ptr dereference To fix it, zspool_mm_stat_show() takes pool under init_lock, to avoid race with disksize_store(). Fixes: 44bffb3bd10b ("[Huawei] multi_pool_zram: add mm_stat interface for each zspool") Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- drivers/block/multi_pool_zram/multi_pool_zram_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 85534cede5a3..c498b57b82b7 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -1175,12 +1175,12 @@ static ssize_t zspool_mm_stat_show(struct device *dev, zram = dev_to_zram(kobj_to_dev(kobj->parent->parent)); pool_index = kobj_to_pool_index(zram, kobj); - pool = zram->mem_pool[pool_index]; zram_zspool_stats = &zram->zspool_stats[pool_index]; memset(&pool_stats, 0x0, sizeof(struct zs_pool_stats)); down_read(&zram->init_lock); if (init_done(zram)) { + pool = zram->mem_pool[pool_index]; mem_used = zs_get_total_pages(pool); zs_pool_stats(pool, &pool_stats); } -- 2.43.0
hulk inclusion category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- zram_reset_device() clears zram->stats but not zram->zspool_stats[]. After reset, zspool_mm_stat shows stale values. On re-init + new IO, counters accumulate on top of stale values, causing permanently incorrect per-pool statistics. In particular, huge_pages_since is a monotonically increasing counter that is never decremented by zram_free_page(), so its stale value can never self-correct. Fix by adding memset for each zspool_stats entry in zram_reset_device(). Fixes: 8f7294e42b50 ("[Huawei] multi_pool_zram: add mm_stat interface for each zspool") Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- drivers/block/multi_pool_zram/multi_pool_zram_drv.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index c498b57b82b7..5af9ee71f8c1 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -945,6 +945,8 @@ static void zram_slot_free_notify(struct block_device *bdev, static void zram_reset_device(struct zram *zram) { + int i; + down_write(&zram->init_lock); set_capacity_and_notify(zram->disk, 0); @@ -958,6 +960,8 @@ static void zram_reset_device(struct zram *zram) zram->comp = NULL; } memset(&zram->stats, 0, sizeof(zram->stats)); + for (i = 0; i < NR_ZPOOL; i++) + memset(&zram->zspool_stats[i], 0, sizeof(struct zram_zspool_stats)); strscpy(zram->compressor, default_compressor, sizeof(zram->compressor)); reset_memcg(zram); up_write(&zram->init_lock); -- 2.43.0
hulk inclusion category: bugfix bugzilla: https://gitee.com/openeuler/kernel/issues/IAFWQH ---------------------------------------- mem_cgroup_show() reads pool_memcg[] without any lock, while writers (zspool_bind_memcg, reset_memcg) hold init_lock write and css_put the old memcg before assigning a new value or NULL. A reader can obtain a pointer to an already-freed memcg and dereference it via cgroup_path(), causing use-after-free. The race is as follows. CPU0 (mem_cgroup_show) CPU1 (zspool_bind_memcg) memcg = zram->pool_memcg[pool_index]; down_write(&zram->init_lock); css_put(&old_memcg->css); // freed zram->pool_memcg[pool_index] = new_memcg; up_write(&zram->init_lock); cgroup_path(memcg->css.cgroup, ...); // UAF Fix by reading pool_memcg[] under init_lock read lock, which serializes with all writers that hold init_lock write lock. Fixes: 0f9e909b4624 ("[Huawei] multi_pool_zram: add sysfs file to support binding memcg for zpool") Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> --- .../block/multi_pool_zram/multi_pool_zram_drv.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c index 5af9ee71f8c1..ffec4d3baa26 100644 --- a/drivers/block/multi_pool_zram/multi_pool_zram_drv.c +++ b/drivers/block/multi_pool_zram/multi_pool_zram_drv.c @@ -1110,18 +1110,22 @@ static ssize_t mem_cgroup_show(struct device *dev, struct mem_cgroup *memcg; struct zram *zram; int pool_index; + int ret; zram = dev_to_zram(kobj_to_dev(kobj->parent->parent)); pool_index = kobj_to_pool_index(zram, kobj); + down_read(&zram->init_lock); memcg = zram->pool_memcg[pool_index]; if (!memcg) - return scnprintf(buf, PAGE_SIZE, "none\n"); - - if (!cgroup_path(memcg->css.cgroup, buf, PATH_MAX)) - return scnprintf(buf, PAGE_SIZE, "none\n"); + ret = scnprintf(buf, PAGE_SIZE, "none\n"); + else if (!cgroup_path(memcg->css.cgroup, buf, PATH_MAX)) + ret = scnprintf(buf, PAGE_SIZE, "none\n"); + else + ret = scnprintf(buf, PAGE_SIZE, "%s\n", buf); - return scnprintf(buf, PAGE_SIZE, "%s\n", buf); + up_read(&zram->init_lock); + return ret; } static ssize_t mem_cgroup_store(struct device *dev, -- 2.43.0
反馈： 您发送到kernel@openeuler.org的补丁/补丁集，已成功转换为PR！ PR链接地址： https://atomgit.com/openeuler/kernel/merge_requests/29883 邮件列表地址：https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/EYA... FeedBack: The patch(es) which you have sent to kernel@openeuler.org mailing list has been converted to a pull request successfully! Pull request link: https://atomgit.com/openeuler/kernel/merge_requests/29883 Mailing list address: https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/EYA...
participants (2)
-
Jinjiang Tu -
patchwork bot