From: David Hildenbrand david@redhat.com

mainline inclusion from linux-5.7-rc1 commit 956f8b445061667c3545baa24778f890d1d522f4 category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

Patch series "mm/memory_hotplug: allow to specify a default online_type", v3.

Distributions nowadays use udev rules ([1] [2]) to specify if and how to online hotplugged memory. The rules seem to get more complex with many special cases. Due to the various special cases, CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE cannot be used. All memory hotplug is handled via udev rules.

Every time we hotplug memory, the udev rule will come to the same conclusion. Especially Hyper-V (but also soon virtio-mem) add a lot of memory in separate memory blocks and wait for memory to get onlined by user space before continuing to add more memory blocks (to not add memory faster than it is getting onlined). This of course slows down the whole memory hotplug process.

To make the job of distributions easier and to avoid udev rules that get more and more complicated, let's extend the mechanism provided by - /sys/devices/system/memory/auto_online_blocks - "memhp_default_state=" on the kernel cmdline to be able to specify also "online_movable" as well as "online_kernel" Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com

=== Example /usr/libexec/config-memhotplug ===

#!/bin/bash

VIRT=`systemd-detect-virt --vm` ARCH=`uname -p`

sense_virtio_mem() { if [ -d "/sys/bus/virtio/drivers/virtio_mem/" ]; then DEVICES=`find /sys/bus/virtio/drivers/virtio_mem/ -maxdepth 1 -type l | wc -l` if [ $DEVICES != "0" ]; then return 0 fi fi return 1 }

if [ ! -e "/sys/devices/system/memory/auto_online_blocks" ]; then echo "Memory hotplug configuration support missing in the kernel" exit 1 fi

if grep "memhp_default_state=" /proc/cmdline > /dev/null; then echo "Memory hotplug configuration overridden in kernel cmdline (memhp_default_state=)" exit 1 fi

if [ $VIRT == "microsoft" ]; then echo "Detected Hyper-V on $ARCH" # Hyper-V wants all memory in ZONE_NORMAL ONLINE_TYPE="online_kernel" elif sense_virtio_mem; then echo "Detected virtio-mem on $ARCH" # virtio-mem wants all memory in ZONE_NORMAL ONLINE_TYPE="online_kernel" elif [ $ARCH == "s390x" ] || [ $ARCH == "s390" ]; then echo "Detected $ARCH" # standby memory should not be onlined automatically ONLINE_TYPE="offline" elif [ $ARCH == "ppc64" ] || [ $ARCH == "ppc64le" ]; then echo "Detected" $ARCH # PPC64 onlines all hotplugged memory right from the kernel ONLINE_TYPE="offline" elif [ $VIRT == "none" ]; then echo "Detected bare-metal on $ARCH" # Bare metal users expect hotplugged memory to be unpluggable. We assume # that ZONE imbalances on such enterpise servers cannot happen and is # properly documented ONLINE_TYPE="online_movable" else # TODO: Hypervisors that want to unplug DIMMs and can guarantee that ZONE # imbalances won't happen echo "Detected $VIRT on $ARCH" # Usually, ballooning is used in virtual environments, so memory should go to # ZONE_NORMAL. However, sometimes "movable_node" is relevant. ONLINE_TYPE="online" fi

echo "Selected online_type:" $ONLINE_TYPE

# Configure what to do with memory that will be hotplugged in the future echo $ONLINE_TYPE 2>/dev/null > /sys/devices/system/memory/auto_online_blocks if [ $? != "0" ]; then echo "Memory hotplug cannot be configured (e.g., old kernel or missing permissions)" # A backup udev rule should handle old kernels if necessary exit 1 fi

# Process all already pluggedd blocks (e.g., DIMMs, but also Hyper-V or virtio-mem) if [ $ONLINE_TYPE != "offline" ]; then for MEMORY in /sys/devices/system/memory/memory*; do STATE=`cat $MEMORY/state` if [ $STATE == "offline" ]; then echo $ONLINE_TYPE > $MEMORY/state fi done fi

=== Example /usr/lib/systemd/system/config-memhotplug.service ===

[Unit] Description=Configure memory hotplug behavior DefaultDependencies=no Conflicts=shutdown.target Before=sysinit.target shutdown.target After=systemd-modules-load.service ConditionPathExists=|/sys/devices/system/memory/auto_online_blocks

[Service] ExecStart=/usr/libexec/config-memhotplug Type=oneshot TimeoutSec=0 RemainAfterExit=yes

[Install] WantedBy=sysinit.target

=== Example modification to the 40-redhat.rules [2] ===

: diff --git a/40-redhat.rules b/40-redhat.rules-new : index 2c690e5..168fd03 100644 : --- a/40-redhat.rules : +++ b/40-redhat.rules-new : @@ -6,6 +6,9 @@ SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online} : # Memory hotadd request : SUBSYSTEM!="memory", GOTO="memory_hotplug_end" : ACTION!="add", GOTO="memory_hotplug_end" : +# memory hotplug behavior configured : +PROGRAM=="grep online /sys/devices/system/memory/auto_online_blocks", GOTO="memory_hotplug_end" : + : PROGRAM="/bin/uname -p", RESULT=="s390*", GOTO="memory_hotplug_end" : : ENV{.state}="online"

===

[1] https://github.com/lnykryn/systemd-rhel/pull/281 [2] https://github.com/lnykryn/systemd-rhel/blob/staging/rules/40-redhat.rules

This patch (of 8):

The name is misleading and it's not really clear what is "kept". Let's just name it like the online_type name we expose to user space ("online").

Add some documentation to the types.

Signed-off-by: David Hildenbrand david@redhat.com Signed-off-by: Andrew Morton akpm@linux-foundation.org Reviewed-by: Wei Yang richard.weiyang@gmail.com Reviewed-by: Baoquan He bhe@redhat.com Acked-by: Pankaj Gupta pankaj.gupta.linux@gmail.com Cc: Greg Kroah-Hartman gregkh@linuxfoundation.org Cc: Michal Hocko mhocko@kernel.org Cc: Oscar Salvador osalvador@suse.de Cc: "Rafael J. Wysocki" rafael@kernel.org Cc: Wei Yang richard.weiyang@gmail.com Cc: Vitaly Kuznetsov vkuznets@redhat.com Cc: Yumei Huang yuhuang@redhat.com Cc: Igor Mammedov imammedo@redhat.com Cc: Eduardo Habkost ehabkost@redhat.com Cc: Benjamin Herrenschmidt benh@kernel.crashing.org Cc: Haiyang Zhang haiyangz@microsoft.com Cc: K. Y. Srinivasan kys@microsoft.com Cc: Michael Ellerman mpe@ellerman.id.au (powerpc) Cc: Paul Mackerras paulus@samba.org Cc: Stephen Hemminger sthemmin@microsoft.com Cc: Wei Liu wei.liu@kernel.org Link: http://lkml.kernel.org/r/20200319131221.14044-1-david@redhat.com Link: http://lkml.kernel.org/r/20200317104942.11178-2-david@redhat.com Signed-off-by: Linus Torvalds torvalds@linux-foundation.org Signed-off-by: Ma Wupeng mawupeng1@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- drivers/base/memory.c | 9 +++++---- include/linux/memory_hotplug.h | 6 +++++- 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 35472a2b9d1a6..74d9bf9dddaab 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -298,7 +298,7 @@ static int memory_subsys_online(struct device *dev) * attribute and need to set the online_type. */ if (mem->online_type < 0) - mem->online_type = MMOP_ONLINE_KEEP; + mem->online_type = MMOP_ONLINE;

ret = memory_block_change_state(mem, MEM_ONLINE, MEM_OFFLINE);

@@ -337,7 +337,7 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr, else if (sysfs_streq(buf, "online_movable")) online_type = MMOP_ONLINE_MOVABLE; else if (sysfs_streq(buf, "online")) - online_type = MMOP_ONLINE_KEEP; + online_type = MMOP_ONLINE; else if (sysfs_streq(buf, "offline")) online_type = MMOP_OFFLINE; else { @@ -348,7 +348,7 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr, switch (online_type) { case MMOP_ONLINE_KERNEL: case MMOP_ONLINE_MOVABLE: - case MMOP_ONLINE_KEEP: + case MMOP_ONLINE: /* mem->online_type is protected by device_hotplug_lock */ mem->online_type = online_type; ret = device_online(&mem->dev); @@ -429,7 +429,8 @@ static ssize_t valid_zones_show(struct device *dev, }

nid = mem->nid; - default_zone = zone_for_pfn_range(MMOP_ONLINE_KEEP, nid, start_pfn, nr_pages); + default_zone = zone_for_pfn_range(MMOP_ONLINE, nid, start_pfn, + nr_pages); strcat(buf, default_zone->name);

print_allowed_zone(buf, nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL, diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 9d28fca5bbde8..09a05938576a3 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -47,9 +47,13 @@ enum {

/* Types for control the zone type of onlined and offlined memory */ enum { + /* Offline the memory. */ MMOP_OFFLINE = -1, - MMOP_ONLINE_KEEP, + /* Online the memory. Zone depends, see default_zone_for_pfn(). */ + MMOP_ONLINE, + /* Online the memory to ZONE_NORMAL. */ MMOP_ONLINE_KERNEL, + /* Online the memory to ZONE_MOVABLE. */ MMOP_ONLINE_MOVABLE, };

From: David Hildenbrand david@redhat.com

mainline inclusion from linux-5.7-rc1 commit 4dc8207bfd45799525f882e1039e63e9438d605e category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

Let's use a simple array which we can reuse soon. While at it, move the string->mmop conversion out of the device hotplug lock.

Signed-off-by: David Hildenbrand david@redhat.com Signed-off-by: Andrew Morton akpm@linux-foundation.org Reviewed-by: Wei Yang richard.weiyang@gmail.com Reviewed-by: Baoquan He bhe@redhat.com Acked-by: Michal Hocko mhocko@suse.com Acked-by: Pankaj Gupta pankaj.gupta.linux@gmail.com Cc: Greg Kroah-Hartman gregkh@linuxfoundation.org Cc: Oscar Salvador osalvador@suse.de Cc: "Rafael J. Wysocki" rafael@kernel.org Cc: Wei Yang richard.weiyang@gmail.com Cc: Benjamin Herrenschmidt benh@kernel.crashing.org Cc: Eduardo Habkost ehabkost@redhat.com Cc: Haiyang Zhang haiyangz@microsoft.com Cc: Igor Mammedov imammedo@redhat.com Cc: "K. Y. Srinivasan" kys@microsoft.com Cc: Michael Ellerman mpe@ellerman.id.au Cc: Paul Mackerras paulus@samba.org Cc: Stephen Hemminger sthemmin@microsoft.com Cc: Vitaly Kuznetsov vkuznets@redhat.com Cc: Wei Liu wei.liu@kernel.org Cc: Yumei Huang yuhuang@redhat.com Link: http://lkml.kernel.org/r/20200317104942.11178-4-david@redhat.com Signed-off-by: Linus Torvalds torvalds@linux-foundation.org Signed-off-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- drivers/base/memory.c | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-)

diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 5dff2c0d56b91..1b6f89fea15d4 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -30,6 +30,24 @@ static DEFINE_MUTEX(mem_sysfs_mutex);

#define MEMORY_CLASS_NAME "memory"

+static const char *const online_type_to_str[] = { + [MMOP_OFFLINE] = "offline", + [MMOP_ONLINE] = "online", + [MMOP_ONLINE_KERNEL] = "online_kernel", + [MMOP_ONLINE_MOVABLE] = "online_movable", +}; + +static int memhp_online_type_from_str(const char *str) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(online_type_to_str); i++) { + if (sysfs_streq(str, online_type_to_str[i])) + return i; + } + return -EINVAL; +} + #define to_memory_block(dev) container_of(dev, struct memory_block, dev)

static int sections_per_block; @@ -322,26 +340,17 @@ static int memory_subsys_offline(struct device *dev) static ssize_t state_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { + const int online_type = memhp_online_type_from_str(buf); struct memory_block *mem = to_memory_block(dev); - int ret, online_type; + int ret; + + if (online_type < 0) + return -EINVAL;

ret = lock_device_hotplug_sysfs(); if (ret) return ret;

- if (sysfs_streq(buf, "online_kernel")) - online_type = MMOP_ONLINE_KERNEL; - else if (sysfs_streq(buf, "online_movable")) - online_type = MMOP_ONLINE_MOVABLE; - else if (sysfs_streq(buf, "online")) - online_type = MMOP_ONLINE; - else if (sysfs_streq(buf, "offline")) - online_type = MMOP_OFFLINE; - else { - ret = -EINVAL; - goto err; - } - switch (online_type) { case MMOP_ONLINE_KERNEL: case MMOP_ONLINE_MOVABLE: @@ -357,7 +366,6 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr, ret = -EINVAL; /* should never happen */ }

-err: unlock_device_hotplug();

if (ret < 0)

From: David Hildenbrand david@redhat.com

mainline inclusion from linux-5.7-rc1 commit 862919e568356cc36288a11b42cd88ec3a7100e9 category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

... and rename it to memhp_default_online_type. This is a preparation for more detailed default online behavior.

Signed-off-by: David Hildenbrand david@redhat.com Signed-off-by: Andrew Morton akpm@linux-foundation.org Reviewed-by: Wei Yang richard.weiyang@gmail.com Reviewed-by: Baoquan He bhe@redhat.com Acked-by: Michal Hocko mhocko@suse.com Acked-by: Pankaj Gupta pankaj.gupta.linux@gmail.com Cc: Greg Kroah-Hartman gregkh@linuxfoundation.org Cc: Oscar Salvador osalvador@suse.de Cc: "Rafael J. Wysocki" rafael@kernel.org Cc: Wei Yang richard.weiyang@gmail.com Cc: Benjamin Herrenschmidt benh@kernel.crashing.org Cc: Eduardo Habkost ehabkost@redhat.com Cc: Haiyang Zhang haiyangz@microsoft.com Cc: Igor Mammedov imammedo@redhat.com Cc: "K. Y. Srinivasan" kys@microsoft.com Cc: Michael Ellerman mpe@ellerman.id.au Cc: Paul Mackerras paulus@samba.org Cc: Stephen Hemminger sthemmin@microsoft.com Cc: Vitaly Kuznetsov vkuznets@redhat.com Cc: Wei Liu wei.liu@kernel.org Cc: Yumei Huang yuhuang@redhat.com Link: http://lkml.kernel.org/r/20200317104942.11178-8-david@redhat.com Signed-off-by: Linus Torvalds torvalds@linux-foundation.org [Wupeng: keep memhp_auto_online for kabi] Signed-off-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- drivers/base/memory.c | 10 ++++------ include/linux/memory_hotplug.h | 2 ++ mm/memory_hotplug.c | 9 ++++++--- 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 1b6f89fea15d4..f7c41be57976b 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -473,10 +473,8 @@ static DEVICE_ATTR_RO(block_size_bytes); static ssize_t auto_online_blocks_show(struct device *dev, struct device_attribute *attr, char *buf) { - if (memhp_auto_online) - return sprintf(buf, "online\n"); - else - return sprintf(buf, "offline\n"); + return sprintf(buf, "%s\n", + online_type_to_str[memhp_default_online_type]); }

static ssize_t auto_online_blocks_store(struct device *dev, @@ -484,9 +482,9 @@ static ssize_t auto_online_blocks_store(struct device *dev, const char *buf, size_t count) { if (sysfs_streq(buf, "online")) - memhp_auto_online = true; + memhp_default_online_type = MMOP_ONLINE; else if (sysfs_streq(buf, "offline")) - memhp_auto_online = false; + memhp_default_online_type = MMOP_OFFLINE; else return -EINVAL;

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index dfb5d5b6f1063..1ba6a688094de 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -105,6 +105,8 @@ extern void __online_page_free(struct page *page); extern int try_online_node(int nid);

extern bool memhp_auto_online; +/* Default online_type (MMOP_*) when new memory blocks are added. */ +extern int memhp_default_online_type; /* If movable_node boot option specified */ extern bool movable_node_enabled; static inline bool movable_node_is_enabled(void) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index f13144a65b127..3b9d409c902b9 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -68,8 +68,10 @@ void put_online_mems(void) bool movable_node_enabled = false;

#ifndef CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE +int memhp_default_online_type = MMOP_OFFLINE; bool memhp_auto_online; #else +int memhp_default_online_type = MMOP_ONLINE; bool memhp_auto_online = true; #endif EXPORT_SYMBOL_GPL(memhp_auto_online); @@ -77,9 +79,9 @@ EXPORT_SYMBOL_GPL(memhp_auto_online); static int __init setup_memhp_default_state(char *str) { if (!strcmp(str, "online")) - memhp_auto_online = true; + memhp_default_online_type = MMOP_ONLINE; else if (!strcmp(str, "offline")) - memhp_auto_online = false; + memhp_default_online_type = MMOP_OFFLINE;

return 1; } @@ -1042,6 +1044,7 @@ static int check_hotplug_memory_range(u64 start, u64 size)

static int online_memory_block(struct memory_block *mem, void *arg) { + mem->online_type = memhp_default_online_type; return device_online(&mem->dev); }

@@ -1114,7 +1117,7 @@ int __ref add_memory_resource(int nid, struct resource *res) mem_hotplug_done();

/* online pages if requested */ - if (memhp_auto_online) + if (memhp_default_online_type != MMOP_OFFLINE) walk_memory_range(PFN_DOWN(start), PFN_UP(start + size - 1), NULL, online_memory_block);

From: Ma Wupeng mawupeng1@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

Make efi_print_memmap() public in preparation for adding fake memory support for architecture with efi support, eg, arm64.

Co-developed-by: Jing Xiangfeng jingxiangfeng@huawei.com Signed-off-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- arch/x86/include/asm/efi.h | 1 - arch/x86/platform/efi/efi.c | 16 ---------------- drivers/firmware/efi/memmap.c | 16 ++++++++++++++++ include/linux/efi.h | 1 + 4 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/arch/x86/include/asm/efi.h b/arch/x86/include/asm/efi.h index baa549f8e9188..00aa9de5abbb1 100644 --- a/arch/x86/include/asm/efi.h +++ b/arch/x86/include/asm/efi.h @@ -123,7 +123,6 @@ extern void __init efi_set_executable(efi_memory_desc_t *md, bool executable); extern int __init efi_memblock_x86_reserve_range(void); extern pgd_t * __init efi_call_phys_prolog(void); extern void __init efi_call_phys_epilog(pgd_t *save_pgd); -extern void __init efi_print_memmap(void); extern void __init efi_memory_uc(u64 addr, unsigned long size); extern void __init efi_map_region(efi_memory_desc_t *md); extern void __init efi_map_region_fixed(efi_memory_desc_t *md); diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c index 9e5229139761b..a01663c9f740e 100644 --- a/arch/x86/platform/efi/efi.c +++ b/arch/x86/platform/efi/efi.c @@ -278,22 +278,6 @@ static void __init efi_clean_memmap(void) } }

-void __init efi_print_memmap(void) -{ - efi_memory_desc_t *md; - int i = 0; - - for_each_efi_memory_desc(md) { - char buf[64]; - - pr_info("mem%02u: %s range=[0x%016llx-0x%016llx] (%lluMB)\n", - i++, efi_md_typeattr_format(buf, sizeof(buf), md), - md->phys_addr, - md->phys_addr + (md->num_pages << EFI_PAGE_SHIFT) - 1, - (md->num_pages >> (20 - EFI_PAGE_SHIFT))); - } -} - static int __init efi_systab_init(void *phys) { if (efi_enabled(EFI_64BIT)) { diff --git a/drivers/firmware/efi/memmap.c b/drivers/firmware/efi/memmap.c index 1907db2b38d81..16c160569e1a1 100644 --- a/drivers/firmware/efi/memmap.c +++ b/drivers/firmware/efi/memmap.c @@ -29,6 +29,22 @@ static phys_addr_t __init __efi_memmap_alloc_late(unsigned long size) return PFN_PHYS(page_to_pfn(p)); }

+void __init efi_print_memmap(void) +{ + efi_memory_desc_t *md; + int i = 0; + + for_each_efi_memory_desc(md) { + char buf[64]; + + pr_info("mem%02u: %s range=[0x%016llx-0x%016llx] (%lluMB)\n", + i++, efi_md_typeattr_format(buf, sizeof(buf), md), + md->phys_addr, + md->phys_addr + (md->num_pages << EFI_PAGE_SHIFT) - 1, + (md->num_pages >> (20 - EFI_PAGE_SHIFT))); + } +} + /** * efi_memmap_alloc - Allocate memory for the EFI memory map * @num_entries: Number of entries in the allocated map. diff --git a/include/linux/efi.h b/include/linux/efi.h index 34c255c2a487c..1582fa572651e 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -1039,6 +1039,7 @@ extern int __init efi_memmap_split_count(efi_memory_desc_t *md, struct range *range); extern void __init efi_memmap_insert(struct efi_memory_map *old_memmap, void *buf, struct efi_mem_range *mem); +extern void __init efi_print_memmap(void);

extern int efi_config_init(efi_config_table_type_t *arch_tables); #ifdef CONFIG_EFI_ESRT

From: Ma Wupeng mawupeng1@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

Commit b05b9f5f9dcf ("x86, mirror: x86 enabling - find mirrored memory ranges") introduce the efi_find_mirror function on x86. In order to reuse the API we make it public in preparation for arm64 to support mirrord memory.

Co-developed-by: Jing Xiangfeng jingxiangfeng@huawei.com Signed-off-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- arch/x86/platform/efi/efi.c | 20 -------------------- drivers/firmware/efi/efi.c | 20 ++++++++++++++++++++ include/linux/efi.h | 4 +++- 3 files changed, 23 insertions(+), 21 deletions(-)

diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c index a01663c9f740e..b8cb57edd424a 100644 --- a/arch/x86/platform/efi/efi.c +++ b/arch/x86/platform/efi/efi.c @@ -101,26 +101,6 @@ static efi_status_t __init phys_efi_set_virtual_address_map( return status; }

-void __init efi_find_mirror(void) -{ - efi_memory_desc_t *md; - u64 mirror_size = 0, total_size = 0; - - for_each_efi_memory_desc(md) { - unsigned long long start = md->phys_addr; - unsigned long long size = md->num_pages << EFI_PAGE_SHIFT; - - total_size += size; - if (md->attribute & EFI_MEMORY_MORE_RELIABLE) { - memblock_mark_mirror(start, size); - mirror_size += size; - } - } - if (mirror_size) - pr_info("Memory: %lldM/%lldM mirrored memory\n", - mirror_size>>20, total_size>>20); -} - /* * Tell the kernel about the EFI memory map. This might include * more than the max 128 entries that can fit in the e820 legacy diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index 0f3ce03c1116d..a159ae07d66f8 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -404,6 +404,26 @@ static int __init efisubsys_init(void)

subsys_initcall(efisubsys_init);

+void __init efi_find_mirror(void) +{ + efi_memory_desc_t *md; + u64 mirror_size = 0, total_size = 0; + + for_each_efi_memory_desc(md) { + unsigned long long start = md->phys_addr; + unsigned long long size = md->num_pages << EFI_PAGE_SHIFT; + + total_size += size; + if (md->attribute & EFI_MEMORY_MORE_RELIABLE) { + memblock_mark_mirror(start, size); + mirror_size += size; + } + } + if (mirror_size) + pr_info("Memory: %lldM/%lldM mirrored memory\n", + mirror_size>>20, total_size>>20); +} + /* * Find the efi memory descriptor for a given physical address. Given a * physical address, determine if it exists within an EFI Memory Map entry, diff --git a/include/linux/efi.h b/include/linux/efi.h index 1582fa572651e..b90423d1128bc 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -1017,7 +1017,6 @@ extern void efi_free_boot_services(void); extern efi_status_t efi_query_variable_store(u32 attributes, unsigned long size, bool nonblocking); -extern void efi_find_mirror(void); #else static inline void efi_free_boot_services(void) {}

@@ -1181,6 +1180,7 @@ static inline bool efi_enabled(int feature) extern void efi_reboot(enum reboot_mode reboot_mode, const char *__unused);

extern bool efi_is_table_address(unsigned long phys_addr); +extern void efi_find_mirror(void); #else static inline bool efi_enabled(int feature) { @@ -1199,6 +1199,8 @@ static inline bool efi_is_table_address(unsigned long phys_addr) { return false; } + +static inline void efi_find_mirror(void) {} #endif

extern int efi_status_to_err(efi_status_t status);

From: Ma Wupeng mawupeng1@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

Introduction Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com

============

Memory reliable feature is a memory tiering mechanism. It is based on kernel mirror feature, which splits memory into two sperate regions, mirrored(reliable) region and non-mirrored (non-reliable) region.

for kernel mirror feature:

- allocate kernel memory from mirrored region by default - allocate user memory from non-mirrored region by default

non-mirrored region will be arranged into ZONE_MOVABLE.

for kernel reliable feature, it has additional features below:

- normal user tasks never alloc memory from mirrored region with userspace apis(malloc, mmap, etc.) - special user tasks will allocate memory from mirrored region by default - tmpfs/pagecache allocate memory from mirrored region by default - upper limit of mirrored region allcated for user tasks, tmpfs and pagecache

Support Reliable fallback mechanism which allows special user tasks, tmpfs and pagecache can fallback to alloc non-mirrored region, it's the default setting.

In order to fulfil the goal

- ___GFP_RELIABILITY flag added for alloc memory from mirrored region.

- the high_zoneidx for special user tasks/tmpfs/pagecache is set to ZONE_NORMAL.

- normal user tasks could only alloc from ZONE_MOVABLE.

This patch is just the main framework, memory reliable support for special user tasks, pagecache and tmpfs has own patches.

To enable this function, mirrored(reliable) memory is needed and "kernelcore=reliable" should be added to kernel parameters.

Signed-off-by: Ma Wupeng mawupeng1@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- .../admin-guide/kernel-parameters.txt | 5 +- include/linux/gfp.h | 12 +++ include/linux/mem_reliable.h | 64 +++++++++++++++ include/linux/mm.h | 3 + mm/Kconfig | 18 +++++ mm/Makefile | 1 + mm/mem_reliable.c | 78 +++++++++++++++++++ mm/page_alloc.c | 46 ++++++++++- 8 files changed, 224 insertions(+), 3 deletions(-) create mode 100644 include/linux/mem_reliable.h create mode 100644 mm/mem_reliable.c

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 886c900323f14..cc5eec8959a07 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1945,7 +1945,7 @@ keepinitrd [HW,ARM]

kernelcore= [KNL,X86,IA-64,PPC,ARM64] - Format: nn[KMGTPE] | nn% | "mirror" + Format: nn[KMGTPE] | nn% | "mirror" | "reliable" This parameter specifies the amount of memory usable by the kernel for non-movable allocations. The requested amount is spread evenly throughout all nodes in the @@ -1969,6 +1969,9 @@ for Movable pages. "nn[KMGTPE]", "nn%", and "mirror" are exclusive, so you cannot specify multiple forms.

+ Option "reliable" is base on option "mirror", but make + some extension. These two features are alternatives. + kgdbdbgp= [KGDB,HW] kgdb over EHCI usb debug port. Format: <Controller#>[,poll interval] The controller # is the number of the ehci usb debug diff --git a/include/linux/gfp.h b/include/linux/gfp.h index f78d1e89593fd..152cb9bdf4365 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -45,6 +45,12 @@ struct vm_area_struct; #define ___GFP_NOLOCKDEP 0 #endif /* If the above are modified, __GFP_BITS_SHIFT may need updating */ +#ifdef CONFIG_MEMORY_RELIABLE +/* add flag at the end of gfp_mask to aovid kapi change */ +#define ___GFP_RELIABILITY 0x40000000u +#else +#define ___GFP_RELIABILITY 0 +#endif

/* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -446,6 +452,12 @@ static inline enum zone_type gfp_zone(gfp_t flags) z = (GFP_ZONE_TABLE >> (bit * GFP_ZONES_SHIFT)) & ((1 << GFP_ZONES_SHIFT) - 1); VM_BUG_ON((GFP_ZONE_BAD >> bit) & 1); + +#ifdef CONFIG_MEMORY_RELIABLE + if (z == ZONE_MOVABLE && flags & ___GFP_RELIABILITY) + return ZONE_NORMAL; +#endif + return z; }

diff --git a/include/linux/mem_reliable.h b/include/linux/mem_reliable.h new file mode 100644 index 0000000000000..b03108441e37a --- /dev/null +++ b/include/linux/mem_reliable.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_MEM_RELIABLE__ +#define __MM_MEM_RELIABLE__ + +#include <linux/stddef.h> +#include <linux/gfp.h> +#include <linux/mmzone.h> +#include <linux/mm_types.h> +#include <linux/sched.h> + + +#ifdef CONFIG_MEMORY_RELIABLE + +extern struct static_key_false mem_reliable; + +extern bool reliable_enabled; + +extern void add_reliable_mem_size(long sz); +extern void mem_reliable_init(bool has_unmirrored_mem, + unsigned long *zone_movable_pfn); + +static inline bool mem_reliable_is_enabled(void) +{ + return static_branch_likely(&mem_reliable); +} + +static inline bool zone_reliable(struct zone *zone) +{ + return mem_reliable_is_enabled() && zone_idx(zone) < ZONE_MOVABLE; +} + +static inline bool skip_none_movable_zone(gfp_t gfp, struct zoneref *z) +{ + if (!mem_reliable_is_enabled()) + return false; + + if (!current->mm || (current->flags & PF_KTHREAD)) + return false; + + /* user tasks can only alloc memory from non-mirrored region */ + if (!(gfp & ___GFP_RELIABILITY) && (gfp & __GFP_HIGHMEM) && + (gfp & __GFP_MOVABLE)) { + if (zonelist_zone_idx(z) < ZONE_MOVABLE) + return true; + } + + return false; +} +#else +#define reliable_enabled 0 + +static inline bool mem_reliable_is_enabled(void) { return false; } +static inline void add_reliable_mem_size(long sz) {} +static inline void mem_reliable_init(bool has_unmirrored_mem, + unsigned long *zone_movable_pfn) {} +static inline bool zone_reliable(struct zone *zone) { return false; } +static inline bool skip_none_movable_zone(gfp_t gfp, struct zoneref *z) +{ + return false; +} + +#endif + +#endif diff --git a/include/linux/mm.h b/include/linux/mm.h index be0be448c3f19..630b103065f4c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -28,6 +28,9 @@ #include <linux/memremap.h> #include <linux/overflow.h>

+/* added to mm.h to avoid every caller adding new header file */ +#include <linux/mem_reliable.h> + struct mempolicy; struct anon_vma; struct anon_vma_chain; diff --git a/mm/Kconfig b/mm/Kconfig index 12601505c4a4a..80d7b47ca9f53 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -831,4 +831,22 @@ config PID_RESERVE We record the pid of dump task in the reserve memory, and reserve the pids before init task start. In restore process, free the reserved pids and realloc them for use. + +config MEMORY_RELIABLE + bool "Support for memory reliable" + depends on ARM64 + default n + help + Memory reliable is based on mirror memory. It has the following + additional features: + a) normal user tasks never alloc memory from mirrored region; + b) special user tasks will allocate memory from mirrored region + by default; c) upper limit of mirrored region allcated for user + tasks, tmpfs and pagecache. + Special user tasks and tmpfs/pagecache can fallback to + non-mirrored region if you enable reliable fallback mechanism. + + To enable this function, mirrored memory is needed and + "kernelcore=reliable" need to be added in kernel parameters. + endmenu diff --git a/mm/Makefile b/mm/Makefile index 8fba091be3868..741f9c250914c 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -109,3 +109,4 @@ obj-$(CONFIG_ASCEND_AUTO_TUNING_HUGEPAGE) += hugepage_tuning.o obj-$(CONFIG_PIN_MEMORY) += pin_mem.o obj-$(CONFIG_ASCEND_SHARE_POOL) += share_pool.o obj-$(CONFIG_MEMCG_MEMFS_INFO) += memcg_memfs_info.o +obj-$(CONFIG_MEMORY_RELIABLE) += mem_reliable.o diff --git a/mm/mem_reliable.c b/mm/mem_reliable.c new file mode 100644 index 0000000000000..2e21839ca49fb --- /dev/null +++ b/mm/mem_reliable.c @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#define pr_fmt(fmt) "mem reliable: " fmt + + +#include <linux/mm.h> +#include <linux/memory.h> +#include <linux/memory_hotplug.h> + +DEFINE_STATIC_KEY_FALSE(mem_reliable); + +bool reliable_enabled; + +static atomic_long_t total_reliable_mem; + +void add_reliable_mem_size(long sz) +{ + atomic_long_add(sz, &total_reliable_mem); +} + +static int reliable_mem_notifier(struct notifier_block *nb, + unsigned long action, void *arg) +{ + struct memory_notify *m_arg = arg; + struct zone *zone; + + switch (action) { + case MEM_ONLINE: + zone = page_zone(pfn_to_page(m_arg->start_pfn)); + if (zone_reliable(zone)) + add_reliable_mem_size(m_arg->nr_pages * PAGE_SIZE); + break; + case MEM_OFFLINE: + zone = page_zone(pfn_to_page(m_arg->start_pfn)); + if (zone_reliable(zone)) + add_reliable_mem_size(-m_arg->nr_pages * PAGE_SIZE); + break; + default: + break; + } + + return NOTIFY_OK; +} + +static struct notifier_block reliable_notifier_block = { + .notifier_call = reliable_mem_notifier, +}; + +void mem_reliable_init(bool has_unmirrored_mem, unsigned long *zone_movable_pfn) +{ + if (!reliable_enabled) + return; + + if (atomic_long_read(&total_reliable_mem) == 0) { + memset(zone_movable_pfn, 0, + sizeof(unsigned long) * MAX_NUMNODES); + + pr_err("init failed, mirrored memory size is zero."); + + return; + } + + if (!has_unmirrored_mem) { + pr_err("init failed, unmirrored memory size is zero."); + + return; + } + + if (register_hotmemory_notifier(&reliable_notifier_block)) { + pr_err("init failed, register memory notifier failed."); + return; + } + + static_branch_enable(&mem_reliable); + + pr_info("init succeed, mirrored memory size(%lu)", + atomic_long_read(&total_reliable_mem)); +} diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4cad86f1e3a91..e1e513e851dec 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3454,6 +3454,10 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, struct page *page; unsigned long mark;

+ /* skip non-movable zone for normal user tasks */ + if (skip_none_movable_zone(gfp_mask, z)) + continue; + /* * CDM nodes get skipped if the requested gfp flag * does not have __GFP_THISNODE set or the nodemask @@ -4557,6 +4561,18 @@ static inline void finalise_ac(gfp_t gfp_mask, struct alloc_context *ac) ac->high_zoneidx, ac->nodemask); }

+static inline void prepare_before_alloc(gfp_t *gfp_mask) +{ + gfp_t gfp_ori = *gfp_mask; + *gfp_mask &= gfp_allowed_mask; + + if (!mem_reliable_is_enabled()) + return; + + if (gfp_ori & ___GFP_RELIABILITY) + *gfp_mask |= ___GFP_RELIABILITY; +} + /* * This is the 'heart' of the zoned buddy allocator. */ @@ -4578,7 +4594,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, return NULL; }

- gfp_mask &= gfp_allowed_mask; + prepare_before_alloc(&gfp_mask); + alloc_mask = gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags)) return NULL; @@ -6912,10 +6929,13 @@ static void __init find_zone_movable_pfns_for_nodes(void) */ if (mirrored_kernelcore) { bool mem_below_4gb_not_mirrored = false; + bool has_unmirrored_mem = false;

for_each_memblock(memory, r) { - if (memblock_is_mirror(r)) + if (memblock_is_mirror(r)) { + add_reliable_mem_size(r->size); continue; + }

nid = r->nid;

@@ -6926,6 +6946,7 @@ static void __init find_zone_movable_pfns_for_nodes(void) continue; }

+ has_unmirrored_mem = true; zone_movable_pfn[nid] = zone_movable_pfn[nid] ? min(usable_startpfn, zone_movable_pfn[nid]) : usable_startpfn; @@ -6934,6 +6955,8 @@ static void __init find_zone_movable_pfns_for_nodes(void) if (mem_below_4gb_not_mirrored) pr_warn("This configuration results in unmirrored kernel memory.");

+ mem_reliable_init(has_unmirrored_mem, zone_movable_pfn); + goto out2; }

@@ -7226,9 +7249,28 @@ static int __init cmdline_parse_kernelcore(char *p) { /* parse kernelcore=mirror */ if (parse_option_str(p, "mirror")) { + if (reliable_enabled) { + pr_info("kernelcore=reliable and kernelcore=mirror are alternative."); + return -EINVAL; + } + + mirrored_kernelcore = true; + return 0; + } + +#ifdef CONFIG_MEMORY_RELIABLE + /* parse kernelcore=reliable */ + if (parse_option_str(p, "reliable")) { + if (!reliable_enabled && mirrored_kernelcore) { + pr_info("kernelcore=mirror and kernelcore=reliable are alternative."); + return -EINVAL; + } + + reliable_enabled = true; mirrored_kernelcore = true; return 0; } +#endif

return cmdline_parse_core(p, &required_kernelcore, &required_kernelcore_percent);

From: Peng Wu wupeng58@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

------------------------------------------

Adding reliable flag for user task. User task with reliable flag can only alloc memory from mirrored region. PF_RELIABLE is added to represent the task's reliable flag.

- For init task, which is regarded as as special task which alloc memory from mirrored region.

- For normal user tasks, The reliable flag can be set via procfs interface shown as below and can be inherited via fork().

User can change a user task's reliable flag by

$ echo [0/1] > /proc/<pid>/reliable

and check a user task's reliable flag by

$ cat /proc/<pid>/reliable

Note, global init task's reliable file can not be accessed.

Signed-off-by: Peng Wu wupeng58@huawei.com Signed-off-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- fs/proc/base.c | 96 +++++++++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + mm/page_alloc.c | 3 ++ 3 files changed, 100 insertions(+)

diff --git a/fs/proc/base.c b/fs/proc/base.c index 349c01c68e576..d7e94f7b5ad3e 100644 --- a/fs/proc/base.c +++ b/fs/proc/base.c @@ -1222,6 +1222,96 @@ static const struct file_operations proc_oom_score_adj_operations = { .llseek = default_llseek, };

+#ifdef CONFIG_MEMORY_RELIABLE +static inline int reliable_check(struct task_struct *task, struct pid *pid) +{ + if (!mem_reliable_is_enabled()) + return -EPERM; + + if (is_global_init(task)) + return -EPERM; + + if (!task->mm || (task->flags & PF_KTHREAD) || + (task->flags & PF_EXITING)) + return -EPERM; + + return 0; +} + +static ssize_t reliable_read(struct file *file, char __user *buf, + size_t count, loff_t *ppos) +{ + struct task_struct *task = get_proc_task(file_inode(file)); + struct pid *pid = proc_pid(file_inode(file)); + char buffer[PROC_NUMBUF]; + size_t len; + short val; + int err; + + if (!task) + return -ESRCH; + + err = reliable_check(task, pid); + if (err) { + put_task_struct(task); + return err; + } + + val = task->flags & PF_RELIABLE ? 1 : 0; + put_task_struct(task); + len = snprintf(buffer, sizeof(buffer), "%hd\n", val); + return simple_read_from_buffer(buf, count, ppos, buffer, len); +} + +static ssize_t reliable_write(struct file *file, const char __user *buf, + size_t count, loff_t *ppos) +{ + struct task_struct *task = get_proc_task(file_inode(file)); + struct pid *pid = proc_pid(file_inode(file)); + char buffer[PROC_NUMBUF]; + int val; + int err; + + if (!task) + return -ESRCH; + + err = reliable_check(task, pid); + if (err) + goto out; + + memset(buffer, 0, sizeof(buffer)); + if (count > sizeof(buffer) - 1) + count = sizeof(buffer) - 1; + if (copy_from_user(buffer, buf, count)) { + err = -EFAULT; + goto out; + } + + err = kstrtoint(strstrip(buffer), 0, &val); + if (err) + goto out; + if (val != 0 && val != 1) { + err = -EINVAL; + goto out; + } + + if (val == 1) + task->flags |= PF_RELIABLE; + else + task->flags &= ~PF_RELIABLE; + +out: + put_task_struct(task); + return err < 0 ? err : count; +} + +static const struct file_operations proc_reliable_operations = { + .read = reliable_read, + .write = reliable_write, + .llseek = generic_file_llseek, +}; +#endif + #ifdef CONFIG_AUDITSYSCALL #define TMPBUFLEN 11 static ssize_t proc_loginuid_read(struct file * file, char __user * buf, @@ -3029,6 +3119,9 @@ static const struct pid_entry tgid_base_stuff[] = { ONE("oom_score", S_IRUGO, proc_oom_score), REG("oom_adj", S_IRUGO|S_IWUSR, proc_oom_adj_operations), REG("oom_score_adj", S_IRUGO|S_IWUSR, proc_oom_score_adj_operations), +#ifdef CONFIG_MEMORY_RELIABLE + REG("reliable", S_IRUGO|S_IWUSR, proc_reliable_operations), +#endif #ifdef CONFIG_AUDITSYSCALL REG("loginuid", S_IWUSR|S_IRUGO, proc_loginuid_operations), REG("sessionid", S_IRUGO, proc_sessionid_operations), @@ -3419,6 +3512,9 @@ static const struct pid_entry tid_base_stuff[] = { ONE("oom_score", S_IRUGO, proc_oom_score), REG("oom_adj", S_IRUGO|S_IWUSR, proc_oom_adj_operations), REG("oom_score_adj", S_IRUGO|S_IWUSR, proc_oom_score_adj_operations), +#ifdef CONFIG_MEMORY_RELIABLE + REG("reliable", S_IRUGO|S_IWUSR, proc_reliable_operations), +#endif #ifdef CONFIG_AUDITSYSCALL REG("loginuid", S_IWUSR|S_IRUGO, proc_loginuid_operations), REG("sessionid", S_IRUGO, proc_sessionid_operations), diff --git a/include/linux/sched.h b/include/linux/sched.h index 945a57ecd9a51..677cb6ace36f5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1421,6 +1421,7 @@ extern struct pid *cad_pid; */ #define PF_IDLE 0x00000002 /* I am an IDLE thread */ #define PF_EXITING 0x00000004 /* Getting shut down */ +#define PF_RELIABLE 0x00000008 /* Allocate from reliable memory */ #define PF_VCPU 0x00000010 /* I'm a virtual CPU */ #define PF_WQ_WORKER 0x00000020 /* I'm a workqueue worker */ #define PF_FORKNOEXEC 0x00000040 /* Forked but didn't exec */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e1e513e851dec..95d2450cf1771 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4571,6 +4571,9 @@ static inline void prepare_before_alloc(gfp_t *gfp_mask)

if (gfp_ori & ___GFP_RELIABILITY) *gfp_mask |= ___GFP_RELIABILITY; + + if (current->flags & PF_RELIABLE || is_global_init(current)) + *gfp_mask |= ___GFP_RELIABILITY; }

/*

From: Peng Wu wupeng58@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

----------------------------------------------

Counting reliable memory allocated by the reliable user tasks.

The policy of counting reliable memory usage is based on RSS statistics. Anywhere with counter of mm need count reliable pages too. Reliable page which is checked by page_reliable() need to update the reliable page counter by calling reliable_page_counter().

Updating the reliable pages should be considered if the following logic is added: - add_mm_counter - dec_mm_counter - inc_mm_counter_fast - dec_mm_counter_fast - rss[mm_counter(page)]

Signed-off-by: Peng Wu wupeng58@huawei.com Signed-off-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- Documentation/filesystems/proc.txt | 2 ++ fs/proc/task_mmu.c | 1 + include/linux/mem_reliable.h | 17 +++++++++++++++++ kernel/events/uprobes.c | 2 ++ mm/huge_memory.c | 8 ++++++++ mm/khugepaged.c | 1 + mm/ksm.c | 1 + mm/mem_reliable.c | 15 ++++++++++++++- mm/memory.c | 16 ++++++++++++++++ mm/migrate.c | 1 + mm/rmap.c | 5 +++++ mm/shmem.c | 1 + mm/swapfile.c | 1 + mm/userfaultfd.c | 1 + 14 files changed, 71 insertions(+), 1 deletion(-)

diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt index 690db5b3eb53b..1ef781f33b376 100644 --- a/Documentation/filesystems/proc.txt +++ b/Documentation/filesystems/proc.txt @@ -181,6 +181,7 @@ read the file /proc/PID/status: VmPTE: 20 kb VmSwap: 0 kB HugetlbPages: 0 kB + Reliable: 1608 KB CoreDumping: 0 Threads: 1 SigQ: 0/28578 @@ -254,6 +255,7 @@ Table 1-2: Contents of the status files (as of 4.8) VmSwap amount of swap used by anonymous private data (shmem swap usage is not included) HugetlbPages size of hugetlb memory portions + Reliable size of reliable memory used CoreDumping process's memory is currently being dumped (killing the process may lead to a corrupted core) Threads number of threads diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 495044e1990bd..78ce353d0dfad 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -77,6 +77,7 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) SEQ_PUT_DEC(" kB\nVmSwap:\t", swap); seq_puts(m, " kB\n"); hugetlb_report_usage(m, mm); + reliable_report_usage(m, mm); } #undef SEQ_PUT_DEC

diff --git a/include/linux/mem_reliable.h b/include/linux/mem_reliable.h index 31be68fac330b..a18a843c7b52f 100644 --- a/include/linux/mem_reliable.h +++ b/include/linux/mem_reliable.h @@ -14,11 +14,14 @@ extern struct static_key_false mem_reliable;

extern bool reliable_enabled; +extern atomic_long_t reliable_user_used_nr_page;

extern void add_reliable_mem_size(long sz); extern void mem_reliable_init(bool has_unmirrored_mem, unsigned long *zone_movable_pfn); extern void reliable_report_meminfo(struct seq_file *m); +extern bool page_reliable(struct page *page); +extern void reliable_report_usage(struct seq_file *m, struct mm_struct *mm);

static inline bool mem_reliable_is_enabled(void) { @@ -47,6 +50,15 @@ static inline bool skip_none_movable_zone(gfp_t gfp, struct zoneref *z)

return false; } + +static inline void reliable_page_counter(struct page *page, + struct mm_struct *mm, int val) +{ + if (page_reliable(page)) { + atomic_long_add(val, &mm->reliable_nr_page); + atomic_long_add(val, &reliable_user_used_nr_page); + } +} #else #define reliable_enabled 0

@@ -60,6 +72,11 @@ static inline bool skip_none_movable_zone(gfp_t gfp, struct zoneref *z) return false; } static inline void reliable_report_meminfo(struct seq_file *m) {} +static inline bool page_reliable(struct page *page) { return false; } +static inline void reliable_page_counter(struct page *page, + struct mm_struct *mm, int val) {} +static inline void reliable_report_usage(struct seq_file *m, + struct mm_struct *mm) {}

#endif

diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index c173e4131df88..de64e29830824 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -191,7 +191,9 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,

if (!PageAnon(old_page)) { dec_mm_counter(mm, mm_counter_file(old_page)); + reliable_page_counter(old_page, mm, -1); inc_mm_counter(mm, MM_ANONPAGES); + reliable_page_counter(new_page, mm, 1); }

flush_cache_page(vma, addr, pte_pfn(*pvmw.pte)); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 31f1c580ba9c0..f8319265c1cf3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -673,6 +673,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable); set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry); add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); + reliable_page_counter(page, vma->vm_mm, HPAGE_PMD_NR); mm_inc_nr_ptes(vma->vm_mm); spin_unlock(vmf->ptl); count_vm_event(THP_FAULT_ALLOC); @@ -1080,6 +1081,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, get_page(src_page); page_dup_rmap(src_page, true); add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + reliable_page_counter(src_page, dst_mm, HPAGE_PMD_NR); mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);

@@ -1468,6 +1470,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); if (!page) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); + reliable_page_counter(new_page, vma->vm_mm, + HPAGE_PMD_NR); } else { VM_BUG_ON_PAGE(!PageHead(page), page); page_remove_rmap(page, true); @@ -1850,10 +1854,12 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (PageAnon(page)) { zap_deposited_table(tlb->mm, pmd); add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR); + reliable_page_counter(page, tlb->mm, -HPAGE_PMD_NR); } else { if (arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); add_mm_counter(tlb->mm, mm_counter_file(page), -HPAGE_PMD_NR); + reliable_page_counter(page, tlb->mm, -HPAGE_PMD_NR); }

spin_unlock(ptl); @@ -2209,6 +2215,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, put_page(page); } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); + reliable_page_counter(page, mm, -HPAGE_PMD_NR); return; }

@@ -3170,6 +3177,7 @@ vm_fault_t do_anon_huge_page_remap(struct vm_area_struct *vma, unsigned long add pgtable_trans_huge_deposit(vma->vm_mm, pmd, pgtable); set_pmd_at(vma->vm_mm, address, pmd, entry); add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); + reliable_page_counter(page, vma->vm_mm, HPAGE_PMD_NR); mm_inc_nr_ptes(vma->vm_mm); spin_unlock(ptl); count_vm_event(THP_FAULT_ALLOC); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 55f171ed2d08a..5ac2486327528 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -648,6 +648,7 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { clear_user_highpage(page, address); add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1); + reliable_page_counter(page, vma->vm_mm, 1); if (is_zero_pfn(pte_pfn(pteval))) { /* * ptl mostly unnecessary. diff --git a/mm/ksm.c b/mm/ksm.c index 9749729a5381a..b656fa77f92ff 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1184,6 +1184,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, * when tearing down the mm. */ dec_mm_counter(mm, MM_ANONPAGES); + reliable_page_counter(page, mm, -1); }

flush_cache_page(vma, addr, pte_pfn(*ptep)); diff --git a/mm/mem_reliable.c b/mm/mem_reliable.c index c03c77090cf5b..d6aec08638923 100644 --- a/mm/mem_reliable.c +++ b/mm/mem_reliable.c @@ -12,14 +12,19 @@ DEFINE_STATIC_KEY_FALSE(mem_reliable);

bool reliable_enabled; - static atomic_long_t total_reliable_mem; +atomic_long_t reliable_user_used_nr_page;

void add_reliable_mem_size(long sz) { atomic_long_add(sz, &total_reliable_mem); }

+bool page_reliable(struct page *page) +{ + return mem_reliable_is_enabled() && page_zonenum(page) < ZONE_MOVABLE; +} + static int reliable_mem_notifier(struct notifier_block *nb, unsigned long action, void *arg) { @@ -105,3 +110,11 @@ void reliable_report_meminfo(struct seq_file *m) used_reliable_mem_sz() >> 10); } } + +void reliable_report_usage(struct seq_file *m, struct mm_struct *mm) +{ + if (mem_reliable_is_enabled()) { + seq_printf(m, "Reliable:\t%8lu kB\n", + atomic_long_read(&mm->reliable_nr_page)); + } +} diff --git a/mm/memory.c b/mm/memory.c index 054e62292902a..d4853970a7c10 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -740,6 +740,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,

rss[mm_counter(page)]++;

+ reliable_page_counter(page, dst_mm, 1); if (is_write_migration_entry(entry) && is_cow_mapping(vm_flags)) { /* @@ -766,6 +767,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, */ get_page(page); rss[mm_counter(page)]++; + reliable_page_counter(page, dst_mm, 1); page_dup_rmap(page, false);

/* @@ -807,6 +809,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, get_page(page); page_dup_rmap(page, false); rss[mm_counter(page)]++; + reliable_page_counter(page, dst_mm, 1); } else if (pte_devmap(pte)) { page = pte_page(pte);

@@ -819,6 +822,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, get_page(page); page_dup_rmap(page, false); rss[mm_counter(page)]++; + reliable_page_counter(page, dst_mm, 1); } }

@@ -1102,6 +1106,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, mark_page_accessed(page); } rss[mm_counter(page)]--; + reliable_page_counter(page, mm, -1); page_remove_rmap(page, false); if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); @@ -1130,6 +1135,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,

pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; + reliable_page_counter(page, mm, -1); page_remove_rmap(page, false); put_page(page); continue; @@ -1147,6 +1153,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,

page = migration_entry_to_page(entry); rss[mm_counter(page)]--; + reliable_page_counter(page, mm, -1); } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); @@ -1490,6 +1497,7 @@ static int insert_page(struct vm_area_struct *vma, unsigned long addr, /* Ok, finally just insert the thing.. */ get_page(page); inc_mm_counter_fast(mm, mm_counter_file(page)); + reliable_page_counter(page, mm, 1); page_add_file_rmap(page, false); set_pte_at(mm, addr, pte, mk_pte(page, prot));

@@ -2489,10 +2497,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) if (!PageAnon(old_page)) { dec_mm_counter_fast(mm, mm_counter_file(old_page)); + reliable_page_counter(old_page, mm, -1); inc_mm_counter_fast(mm, MM_ANONPAGES); + reliable_page_counter(new_page, mm, 1); } } else { inc_mm_counter_fast(mm, MM_ANONPAGES); + reliable_page_counter(new_page, mm, 1); } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); @@ -3051,6 +3062,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */

inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); + reliable_page_counter(page, vma->vm_mm, 1); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); pte = mk_pte(page, vma->vm_page_prot); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { @@ -3216,6 +3228,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) }

inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); + reliable_page_counter(page, vma->vm_mm, 1); page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); lru_cache_add_active_or_unevictable(page, vma); @@ -3416,6 +3429,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);

add_mm_counter(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); + reliable_page_counter(page, vma->vm_mm, HPAGE_PMD_NR); page_add_file_rmap(page, true); /* * deposit and withdraw with pmd lock held @@ -3489,6 +3503,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* copy-on-write page */ + reliable_page_counter(page, vma->vm_mm, 1); if (write && !(vma->vm_flags & VM_SHARED)) { inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); page_add_new_anon_rmap(page, vma, vmf->address, false); @@ -4910,6 +4925,7 @@ vm_fault_t do_anon_page_remap(struct vm_area_struct *vma, unsigned long address, if (ret) goto release; inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); + reliable_page_counter(page, vma->vm_mm, 1); page_add_new_anon_rmap(page, vma, address, false); mem_cgroup_commit_charge(page, memcg, false, false); lru_cache_add_active_or_unevictable(page, vma); diff --git a/mm/migrate.c b/mm/migrate.c index 90aa493faa602..eb27e8e2bf213 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2714,6 +2714,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, }

inc_mm_counter(mm, MM_ANONPAGES); + reliable_page_counter(page, mm, 1); page_add_new_anon_rmap(page, vma, addr, false); mem_cgroup_commit_charge(page, memcg, false, false); if (!is_zone_device_page(page)) diff --git a/mm/rmap.c b/mm/rmap.c index 7debdf0cc6785..224fac084ad0e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1548,6 +1548,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, vma_mmu_pagesize(vma)); } else { dec_mm_counter(mm, mm_counter(page)); + reliable_page_counter(page, mm, -1); set_pte_at(mm, address, pvmw.pte, pteval); }

@@ -1563,6 +1564,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, * copied pages. */ dec_mm_counter(mm, mm_counter(page)); + reliable_page_counter(page, mm, -1); /* We have to invalidate as we cleared the pte */ mmu_notifier_invalidate_range(mm, address, address + PAGE_SIZE); @@ -1617,6 +1619,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, mmu_notifier_invalidate_range(mm, address, address + PAGE_SIZE); dec_mm_counter(mm, MM_ANONPAGES); + reliable_page_counter(page, mm, -1); goto discard; }

@@ -1650,6 +1653,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, spin_unlock(&mmlist_lock); } dec_mm_counter(mm, MM_ANONPAGES); + reliable_page_counter(page, mm, -1); inc_mm_counter(mm, MM_SWAPENTS); swp_pte = swp_entry_to_pte(entry); if (pte_soft_dirty(pteval)) @@ -1670,6 +1674,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, * See Documentation/vm/mmu_notifier.rst */ dec_mm_counter(mm, mm_counter_file(page)); + reliable_page_counter(page, mm, -1); } discard: /* diff --git a/mm/shmem.c b/mm/shmem.c index 8d32d49a4d7ba..16bb7806a25e6 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2473,6 +2473,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, spin_unlock_irq(&info->lock);

inc_mm_counter(dst_mm, mm_counter_file(page)); + reliable_page_counter(page, dst_mm, 1); page_add_file_rmap(page, false); set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte);

diff --git a/mm/swapfile.c b/mm/swapfile.c index 4028994a51ae6..2619729400d32 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1869,6 +1869,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,

dec_mm_counter(vma->vm_mm, MM_SWAPENTS); inc_mm_counter(vma->vm_mm, MM_ANONPAGES); + reliable_page_counter(page, vma->vm_mm, 1); get_page(page); set_pte_at(vma->vm_mm, addr, pte, pte_mkold(mk_pte(page, vma->vm_page_prot))); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 1c86abd41c6d7..c26dd2040624f 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -116,6 +116,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, #endif

inc_mm_counter(dst_mm, MM_ANONPAGES); + reliable_page_counter(page, dst_mm, 1); page_add_new_anon_rmap(page, dst_vma, dst_addr, false); mem_cgroup_commit_charge(page, memcg, false, false); lru_cache_add_active_or_unevictable(page, dst_vma);

From: Peng Wu wupeng58@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

----------------------------------------------

there is a upper limit for special user tasks's memory allocation. special user task means user task with reliable flag.

Init tasks will alloc memory from non-mirrored region if their allocation trigger limit.

The limit can be set or access via /proc/sys/vm/task_reliable_limit

This limit's default value is ULONG_MAX.

Signed-off-by: Peng Wu wupeng58@huawei.com Signed-off-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- include/linux/mem_reliable.h | 20 +++++++++ lib/show_mem.c | 1 + mm/khugepaged.c | 14 ++++++ mm/mem_reliable.c | 86 ++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 50 ++++++++++++++++++--- 5 files changed, 164 insertions(+), 7 deletions(-)

diff --git a/include/linux/mem_reliable.h b/include/linux/mem_reliable.h index a18a843c7b52f..e4097f0cff679 100644 --- a/include/linux/mem_reliable.h +++ b/include/linux/mem_reliable.h @@ -15,6 +15,7 @@ extern struct static_key_false mem_reliable;

extern bool reliable_enabled; extern atomic_long_t reliable_user_used_nr_page; +extern unsigned long task_reliable_limit __read_mostly;

extern void add_reliable_mem_size(long sz); extern void mem_reliable_init(bool has_unmirrored_mem, @@ -22,6 +23,9 @@ extern void mem_reliable_init(bool has_unmirrored_mem, extern void reliable_report_meminfo(struct seq_file *m); extern bool page_reliable(struct page *page); extern void reliable_report_usage(struct seq_file *m, struct mm_struct *mm); +extern void reliable_show_mem_info(void); +extern void mem_reliable_out_of_memory(gfp_t gfp_mask, unsigned int order, + int preferred_nid, nodemask_t *nodemask);

static inline bool mem_reliable_is_enabled(void) { @@ -59,6 +63,12 @@ static inline void reliable_page_counter(struct page *page, atomic_long_add(val, &reliable_user_used_nr_page); } } + +static inline bool reliable_mem_limit_check(unsigned long nr_page) +{ + return atomic_long_read(&reliable_user_used_nr_page) + nr_page <= + task_reliable_limit / PAGE_SIZE; +} #else #define reliable_enabled 0

@@ -78,6 +88,16 @@ static inline void reliable_page_counter(struct page *page, static inline void reliable_report_usage(struct seq_file *m, struct mm_struct *mm) {}

+static inline bool reliable_mem_limit_check(unsigned long nr_page) +{ + return false; +} +static inline void reliable_show_mem_info(void) {} +static inline void mem_reliable_out_of_memory(gfp_t gfp_mask, + unsigned int order, + int preferred_nid, + nodemask_t *nodemask) {} + #endif

#endif diff --git a/lib/show_mem.c b/lib/show_mem.c index 0beaa1d899aae..0f85331ba91b9 100644 --- a/lib/show_mem.c +++ b/lib/show_mem.c @@ -49,4 +49,5 @@ void show_mem(unsigned int filter, nodemask_t *nodemask) #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif + reliable_show_mem_info(); } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 7f37633a886e0..2b154ff6ee734 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1235,6 +1235,12 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (ret) { + if (reliable && + !reliable_mem_limit_check(1 << HPAGE_PMD_ORDER)) { + ret = SCAN_ALLOC_HUGE_PAGE_FAIL; + goto out; + } + node = khugepaged_find_target_node(); /* collapse_huge_page will return with the mmap_sem released */ collapse_huge_page(mm, address, hpage, node, @@ -1692,6 +1698,12 @@ static void khugepaged_scan_shmem(struct mm_struct *mm, if (present < HPAGE_PMD_NR - khugepaged_max_ptes_none) { result = SCAN_EXCEED_NONE_PTE; } else { + if (reliable && + !reliable_mem_limit_check(1 << HPAGE_PMD_ORDER)) { + result = SCAN_ALLOC_HUGE_PAGE_FAIL; + goto out; + } + node = khugepaged_find_target_node(); collapse_shmem(mm, mapping, start, hpage, node, reliable); @@ -1699,6 +1711,8 @@ static void khugepaged_scan_shmem(struct mm_struct *mm, }

/* TODO: tracepoints */ +out: + return; } #else static void khugepaged_scan_shmem(struct mm_struct *mm, diff --git a/mm/mem_reliable.c b/mm/mem_reliable.c index d6aec08638923..c24c5b7cbca33 100644 --- a/mm/mem_reliable.c +++ b/mm/mem_reliable.c @@ -8,12 +8,15 @@ #include <linux/memory_hotplug.h> #include <linux/seq_file.h> #include <linux/mmzone.h> +#include <linux/oom.h>

DEFINE_STATIC_KEY_FALSE(mem_reliable);

bool reliable_enabled; static atomic_long_t total_reliable_mem; atomic_long_t reliable_user_used_nr_page; +/* reliable user limit for user tasks with reliable flag */ +unsigned long task_reliable_limit = ULONG_MAX;

void add_reliable_mem_size(long sz) { @@ -118,3 +121,86 @@ void reliable_report_usage(struct seq_file *m, struct mm_struct *mm) atomic_long_read(&mm->reliable_nr_page)); } } + +#ifdef CONFIG_SYSCTL +int reliable_limit_handler(struct ctl_table *table, int write, + void __user *buffer, size_t *length, loff_t *ppos) +{ + unsigned long old = task_reliable_limit; + int ret; + + ret = proc_doulongvec_minmax(table, write, buffer, length, ppos); + if (ret == 0 && write) { + if (task_reliable_limit > total_reliable_mem_sz()) { + task_reliable_limit = old; + return -EINVAL; + } + } + + return ret; +} + +static struct ctl_table reliable_ctl_table[] = { + { + .procname = "task_reliable_limit", + .data = &task_reliable_limit, + .maxlen = sizeof(task_reliable_limit), + .mode = 0644, + .proc_handler = reliable_limit_handler, + }, + {} +}; + +static struct ctl_table reliable_dir_table[] = { + { + .procname = "vm", + .maxlen = 0, + .mode = 0555, + .child = reliable_ctl_table, + }, + {} +}; + +static int __init reliable_sysctl_init(void) +{ + if (!mem_reliable_is_enabled()) + return 0; + + if (!register_sysctl_table(reliable_dir_table)) { + pr_err("register sysctl failed."); + return -1; + } + + return 0; +} +late_initcall(reliable_sysctl_init); +#endif + +void reliable_show_mem_info(void) +{ + if (mem_reliable_is_enabled()) { + pr_info("ReliableTotal: %lu kB", total_reliable_mem_sz() >> 10); + pr_info("ReliableUsed: %lu kB", used_reliable_mem_sz() >> 10); + pr_info("task_reliable_limit: %lu kB", + task_reliable_limit >> 10); + pr_info("reliable_user_used: %ld kB", + atomic_long_read(&reliable_user_used_nr_page) * 4); + } +} + +void mem_reliable_out_of_memory(gfp_t gfp_mask, unsigned int order, + int preferred_nid, nodemask_t *nodemask) +{ + struct oom_control oc = { + .zonelist = node_zonelist(preferred_nid, gfp_mask), + .nodemask = nodemask, + .memcg = NULL, + .gfp_mask = gfp_mask, + .order = order, + }; + + if (!mutex_trylock(&oom_lock)) + return; + out_of_memory(&oc); + mutex_unlock(&oom_lock); +} diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 95d2450cf1771..bfc0c2d1825cd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4561,19 +4561,51 @@ static inline void finalise_ac(gfp_t gfp_mask, struct alloc_context *ac) ac->high_zoneidx, ac->nodemask); }

-static inline void prepare_before_alloc(gfp_t *gfp_mask) +/* + * return false means this allocation is limit by reliable user limit and + * this will lead to pagefault_out_of_memory() + */ +static inline bool prepare_before_alloc(gfp_t *gfp_mask, unsigned int order) { gfp_t gfp_ori = *gfp_mask; *gfp_mask &= gfp_allowed_mask;

if (!mem_reliable_is_enabled()) - return; + return true;

- if (gfp_ori & ___GFP_RELIABILITY) + if (gfp_ori & ___GFP_RELIABILITY) { *gfp_mask |= ___GFP_RELIABILITY; + return true; + }

- if (current->flags & PF_RELIABLE || is_global_init(current)) - *gfp_mask |= ___GFP_RELIABILITY; + /* + * Init tasks will alloc memory from non-mirrored region if their + * allocation trigger task_reliable_limit + */ + if (is_global_init(current)) { + if (reliable_mem_limit_check(1 << order)) + *gfp_mask |= ___GFP_RELIABILITY; + return true; + } + + /* + * This only check task_reliable_limit without ___GFP_RELIABILITY + * or this process is global init. + * For kernel internal mechanism(hugepaged collapse and others) + * If they alloc memory for user and obey task_reliable_limit, they + * need to check this limit before allocing pages. + */ + if ((current->flags & PF_RELIABLE) && (gfp_ori & __GFP_HIGHMEM) && + (gfp_ori & __GFP_MOVABLE)) { + if (reliable_mem_limit_check(1 << order)) { + *gfp_mask |= ___GFP_RELIABILITY; + return true; + } + + return false; + } + + return true; }

/* @@ -4583,7 +4615,7 @@ struct page * __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, nodemask_t *nodemask) { - struct page *page; + struct page *page = NULL; unsigned int alloc_flags = ALLOC_WMARK_LOW; gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */ struct alloc_context ac = { }; @@ -4597,7 +4629,11 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, return NULL; }

- prepare_before_alloc(&gfp_mask); + if (!prepare_before_alloc(&gfp_mask, order)) { + mem_reliable_out_of_memory(gfp_mask, order, preferred_nid, + nodemask); + goto out; + }

alloc_mask = gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags))

From: Zhou Guanghui zhouguanghui1@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

------------------------------------------

This feature depends on the overall memory reliable feature. When the shared memory reliable feature is enabled, the pages used by the shared memory are allocated from the mirrored region by default. If the mirrored region is insufficient, you can allocate resources from the non-mirrored region.

Signed-off-by: Zhou Guanghui zhouguanghui1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- Documentation/admin-guide/kernel-parameters.txt | 3 ++- include/linux/mem_reliable.h | 10 ++++++++++ mm/mem_reliable.c | 16 ++++++++++++++++ mm/shmem.c | 12 ++++++++++++ 4 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 3fc729aab31a6..8b6f7071072b7 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1973,13 +1973,14 @@ some extension. These two features are alternatives.

reliable_debug= [ARM64] - Format: [F] + Format: [F][,S] Only works with CONFIG_MEMORY_RELIABLE and "kernelcore=reliable" is configured. F: User tasks with PF_RELIABLE will not allocate memory from non-mirrored region if this allocation from mirrored region failed. Pagecache and tmpfs will follow this rule too. + S: The shmem does not use the reliable memory.

kgdbdbgp= [KGDB,HW] kgdb over EHCI usb debug port. Format: <Controller#>[,poll interval] diff --git a/include/linux/mem_reliable.h b/include/linux/mem_reliable.h index c9c4d94a4df46..0641c7a88c786 100644 --- a/include/linux/mem_reliable.h +++ b/include/linux/mem_reliable.h @@ -17,10 +17,12 @@ extern bool reliable_enabled; extern atomic_long_t reliable_user_used_nr_page; extern unsigned long task_reliable_limit __read_mostly; extern bool reliable_allow_fallback; +extern bool shmem_reliable;

extern void add_reliable_mem_size(long sz); extern void mem_reliable_init(bool has_unmirrored_mem, unsigned long *zone_movable_pfn); +extern void shmem_reliable_init(void); extern void reliable_report_meminfo(struct seq_file *m); extern bool page_reliable(struct page *page); extern void reliable_report_usage(struct seq_file *m, struct mm_struct *mm); @@ -75,6 +77,12 @@ static inline bool reliable_allow_fb_enabled(void) { return reliable_allow_fallback; } + +static inline bool shmem_reliable_is_enabled(void) +{ + return shmem_reliable; +} + #else #define reliable_enabled 0 #define reliable_allow_fb_enabled() false @@ -83,6 +91,7 @@ static inline bool mem_reliable_is_enabled(void) { return false; } static inline void add_reliable_mem_size(long sz) {} static inline void mem_reliable_init(bool has_unmirrored_mem, unsigned long *zone_movable_pfn) {} +static inline void shmem_reliable_init(void) {} static inline bool zone_reliable(struct zone *zone) { return false; } static inline bool skip_none_movable_zone(gfp_t gfp, struct zoneref *z) { @@ -104,6 +113,7 @@ static inline void mem_reliable_out_of_memory(gfp_t gfp_mask, unsigned int order, int preferred_nid, nodemask_t *nodemask) {} +static inline bool shmem_reliable_is_enabled(void) { return false; }

#endif

diff --git a/mm/mem_reliable.c b/mm/mem_reliable.c index 60a214e3b28f7..32a0270b494d2 100644 --- a/mm/mem_reliable.c +++ b/mm/mem_reliable.c @@ -18,6 +18,7 @@ atomic_long_t reliable_user_used_nr_page; /* reliable user limit for user tasks with reliable flag */ unsigned long task_reliable_limit = ULONG_MAX; bool reliable_allow_fallback __read_mostly = true; +bool shmem_reliable __read_mostly = true;

void add_reliable_mem_size(long sz) { @@ -88,6 +89,17 @@ void mem_reliable_init(bool has_unmirrored_mem, unsigned long *zone_movable_pfn) atomic_long_read(&total_reliable_mem)); }

+void shmem_reliable_init(void) +{ + if (!shmem_reliable_is_enabled()) + return; + + if (!mem_reliable_is_enabled()) { + shmem_reliable = false; + pr_info("shmem reliable disabled.\n"); + } +} + static unsigned long total_reliable_mem_sz(void) { return atomic_long_read(&total_reliable_mem); @@ -223,6 +235,10 @@ static int __init setup_reliable_debug(char *str) reliable_allow_fallback = false; pr_info("fallback disabled."); break; + case 'S': + shmem_reliable = false; + pr_info("shmem reliable disabled."); + break; default: pr_err("reliable_debug option '%c' unknown. skipped\n", *str); diff --git a/mm/shmem.c b/mm/shmem.c index 16bb7806a25e6..e27fc90bab412 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1593,6 +1593,14 @@ static struct page *shmem_alloc_page(gfp_t gfp, return page; }

+static inline void shmem_prepare_alloc(gfp_t *gfp_mask) +{ + if (!shmem_reliable_is_enabled()) + return; + + *gfp_mask |= ___GFP_RELIABILITY; +} + static struct page *shmem_alloc_and_acct_page(gfp_t gfp, struct inode *inode, pgoff_t index, bool huge, int node_id) @@ -1609,6 +1617,8 @@ static struct page *shmem_alloc_and_acct_page(gfp_t gfp, if (!shmem_inode_acct_block(inode, nr)) goto failed;

+ shmem_prepare_alloc(&gfp); + if (huge) page = shmem_alloc_hugepage(gfp, info, index, node_id); else @@ -3941,6 +3951,8 @@ int __init shmem_init(void) else shmem_huge = 0; /* just in case it was patched */ #endif + + shmem_reliable_init(); return 0;

out1:

From: Chen Wandun chenwandun@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

__page_cache_alloc is used to alloc page cache in most file system, such as ext4, f2fs, so add ___GFP_RELIABILITY flag to support feature CONFIG_MEMORY_RELIABLE when alloc page.

Signed-off-by: Chen Wandun chenwandun@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- include/linux/pagemap.h | 1 + mm/filemap.c | 1 + 2 files changed, 2 insertions(+)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 085aed892ce58..a6457acd7462e 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -249,6 +249,7 @@ extern struct page *__page_cache_alloc(gfp_t gfp); #else static inline struct page *__page_cache_alloc(gfp_t gfp) { + gfp |= ___GFP_RELIABILITY; return alloc_pages(gfp, 0); } #endif diff --git a/mm/filemap.c b/mm/filemap.c index f578d4e3e2c86..c30e5c1eb77c2 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1042,6 +1042,7 @@ struct page *__page_cache_alloc(gfp_t gfp) int n; struct page *page;

+ gfp |= ___GFP_RELIABILITY; if (cpuset_do_page_mem_spread()) { unsigned int cpuset_mems_cookie; do {

From: Chen Wandun chenwandun@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

-------------------------------- Item "FileCache" in /proc/meminfo show the number of page cache in LRU(active + inactive).

Signed-off-by: Chen Wandun chenwandun@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- mm/mem_reliable.c | 11 +++++++++++ 1 file changed, 11 insertions(+)

diff --git a/mm/mem_reliable.c b/mm/mem_reliable.c index 796892d5d0560..d4458d7401945 100644 --- a/mm/mem_reliable.c +++ b/mm/mem_reliable.c @@ -165,6 +165,17 @@ void reliable_report_meminfo(struct seq_file *m) &reliable_shmem_used_nr_page) << (PAGE_SHIFT - 10); seq_printf(m, "ReliableShmem: %8lu kB\n", shmem); } + + if (pagecache_reliable_is_enabled()) { + unsigned long num = 0; + + num += global_node_page_state(NR_LRU_BASE + + LRU_ACTIVE_FILE); + num += global_node_page_state(NR_LRU_BASE + + LRU_INACTIVE_FILE); + seq_printf(m, "FileCache: %8lu kB\n", + num << (PAGE_SHIFT - 10)); + } } }

From: Chen Wandun chenwandun@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

-------------------------------------------------------

In function cache_limit_ratio_sysctl_handler and cache_limit_mbytes_sysctl_handler, it will shrink page cache even if vm_cache_reclaim_enable is false, it is unexpected.

Signed-off-by: Chen Wandun chenwandun@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- mm/page_alloc.c | 6 ++++-- mm/vmscan.c | 3 ++- 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 455525e49c727..3fb21ea5dcf9b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8600,7 +8600,8 @@ int cache_limit_ratio_sysctl_handler(struct ctl_table *table, int write, vm_cache_limit_ratio); else pr_warn("page cache limit off\n"); - while (vm_cache_limit_mbytes && page_cache_over_limit()) + while (vm_cache_limit_mbytes && page_cache_over_limit() && + vm_cache_reclaim_enable) shrink_page_cache(GFP_KERNEL); }

@@ -8628,7 +8629,8 @@ int cache_limit_mbytes_sysctl_handler(struct ctl_table *table, int write, else pr_warn("page cache limit off\n");

- while (vm_cache_limit_mbytes && page_cache_over_limit()) + while (vm_cache_limit_mbytes && page_cache_over_limit() && + vm_cache_reclaim_enable) shrink_page_cache(GFP_KERNEL); }

diff --git a/mm/vmscan.c b/mm/vmscan.c index 208ee8653da62..39b06e6eaa0f0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3562,7 +3562,8 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) kernel_swap_check(&sc);

#ifdef CONFIG_SHRINK_PAGECACHE - if (vm_cache_limit_mbytes && page_cache_over_limit()) + if (vm_cache_limit_mbytes && page_cache_over_limit() && + vm_cache_reclaim_enable) shrink_page_cache(GFP_KERNEL); #endif

From: Chen Wandun chenwandun@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

The number of page cache should be limited in a range if enable CONFIG_MEMORY_RELIABLE, so only page cache instead of both file + anono page should be reclaimed during page cache reclaimtion.

Signed-off-by: Chen Wandun chenwandun@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- mm/vmscan.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c index 39b06e6eaa0f0..a9fca07dba3ba 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@

#include <linux/swapops.h> #include <linux/balloon_compaction.h> +#include <linux/mem_reliable.h>

#include "internal.h"

@@ -3968,7 +3969,7 @@ static unsigned long __shrink_page_cache(gfp_t mask) .nr_to_reclaim = SWAP_CLUSTER_MAX * (unsigned long)vm_cache_reclaim_weight, .may_unmap = 1, - .may_swap = 1, + .may_swap = mem_reliable_is_enabled() ? 0 : 1, .order = 0, .priority = DEF_PRIORITY, .target_mem_cgroup = NULL,

From: Chen Wandun chenwandun@huawei.com

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

-------------------------------- Add two proc interfaces to set page cache limit. Both vm_cache_limit_mbytes and vm_cache_limit_ratio will be update when writing either of the two interfaces.

Signed-off-by: Chen Wandun chenwandun@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- kernel/sysctl.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)

diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 31c35a65fbb78..542611081c610 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1468,6 +1468,24 @@ static struct ctl_table vm_table[] = { .extra1 = &zero, .extra2 = &one, }, + { + .procname = "cache_limit_mbytes", + .data = &vm_cache_limit_mbytes, + .maxlen = sizeof(vm_cache_limit_mbytes), + .mode = 0644, + .proc_handler = cache_limit_mbytes_sysctl_handler, + .extra1 = &vm_cache_limit_mbytes_min, + .extra2 = &vm_cache_limit_mbytes_max, + }, + { + .procname = "cache_limit_ratio", + .data = &vm_cache_limit_ratio, + .maxlen = sizeof(vm_cache_limit_ratio), + .mode = 0644, + .proc_handler = cache_limit_ratio_sysctl_handler, + .extra1 = &vm_cache_limit_ratio_min, + .extra2 = &vm_cache_limit_ratio_max, + }, #endif #ifdef CONFIG_HUGETLB_PAGE {

From: Alexander Duyck alexander.h.duyck@linux.intel.com

stable inclusion from stable-5.10.88 commit 8204e0c1113d6b7f599bcd7ebfbfde72e76c102f bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S

--------------------------------

Provide a new function, queue_work_node, which is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.

For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.

As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.

Acked-by: Tejun Heo tj@kernel.org Reviewed-by: Bart Van Assche bvanassche@acm.org Acked-by: Dan Williams dan.j.williams@intel.com Signed-off-by: Alexander Duyck alexander.h.duyck@linux.intel.com Signed-off-by: Greg Kroah-Hartman gregkh@linuxfoundation.org Signed-off-by: Yu Liao liaoyu15@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- include/linux/workqueue.h | 2 + kernel/workqueue.c | 84 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)

diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 93b87cd67ee4a..6f2b042fc44cd 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -481,6 +481,8 @@ int workqueue_set_unbound_cpumask(cpumask_var_t cpumask);

extern bool queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work); +extern bool queue_work_node(int node, struct workqueue_struct *wq, + struct work_struct *work); extern bool queue_delayed_work_on(int cpu, struct workqueue_struct *wq, struct delayed_work *work, unsigned long delay); extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq, diff --git a/kernel/workqueue.c b/kernel/workqueue.c index a46532be3bcb0..04b558a267aea 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1490,6 +1490,90 @@ bool queue_work_on(int cpu, struct workqueue_struct *wq, } EXPORT_SYMBOL(queue_work_on);

+/** + * workqueue_select_cpu_near - Select a CPU based on NUMA node + * @node: NUMA node ID that we want to select a CPU from + * + * This function will attempt to find a "random" cpu available on a given + * node. If there are no CPUs available on the given node it will return + * WORK_CPU_UNBOUND indicating that we should just schedule to any + * available CPU if we need to schedule this work. + */ +static int workqueue_select_cpu_near(int node) +{ + int cpu; + + /* No point in doing this if NUMA isn't enabled for workqueues */ + if (!wq_numa_enabled) + return WORK_CPU_UNBOUND; + + /* Delay binding to CPU if node is not valid or online */ + if (node < 0 || node >= MAX_NUMNODES || !node_online(node)) + return WORK_CPU_UNBOUND; + + /* Use local node/cpu if we are already there */ + cpu = raw_smp_processor_id(); + if (node == cpu_to_node(cpu)) + return cpu; + + /* Use "random" otherwise know as "first" online CPU of node */ + cpu = cpumask_any_and(cpumask_of_node(node), cpu_online_mask); + + /* If CPU is valid return that, otherwise just defer */ + return cpu < nr_cpu_ids ? cpu : WORK_CPU_UNBOUND; +} + +/** + * queue_work_node - queue work on a "random" cpu for a given NUMA node + * @node: NUMA node that we are targeting the work for + * @wq: workqueue to use + * @work: work to queue + * + * We queue the work to a "random" CPU within a given NUMA node. The basic + * idea here is to provide a way to somehow associate work with a given + * NUMA node. + * + * This function will only make a best effort attempt at getting this onto + * the right NUMA node. If no node is requested or the requested node is + * offline then we just fall back to standard queue_work behavior. + * + * Currently the "random" CPU ends up being the first available CPU in the + * intersection of cpu_online_mask and the cpumask of the node, unless we + * are running on the node. In that case we just use the current CPU. + * + * Return: %false if @work was already on a queue, %true otherwise. + */ +bool queue_work_node(int node, struct workqueue_struct *wq, + struct work_struct *work) +{ + unsigned long flags; + bool ret = false; + + /* + * This current implementation is specific to unbound workqueues. + * Specifically we only return the first available CPU for a given + * node instead of cycling through individual CPUs within the node. + * + * If this is used with a per-cpu workqueue then the logic in + * workqueue_select_cpu_near would need to be updated to allow for + * some round robin type logic. + */ + WARN_ON_ONCE(!(wq->flags & WQ_UNBOUND)); + + local_irq_save(flags); + + if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) { + int cpu = workqueue_select_cpu_near(node); + + __queue_work(cpu, wq, work); + ret = true; + } + + local_irq_restore(flags); + return ret; +} +EXPORT_SYMBOL_GPL(queue_work_node); + void delayed_work_timer_fn(struct timer_list *t) { struct delayed_work *dwork = from_timer(dwork, t, timer);

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I4SK3S CVE: NA

--------------------------------

Enable CONFIG_MEMORY_RELIABLE CONFIG_CLEAR_FREELIST_PAGE and CONFIG_EFI_FAKE_MEMMAP for test

Reviewed-by: Chen Wandun chenwandun@huawei.com Signed-off-by: Yang Yingliang yangyingliang@huawei.com --- arch/arm64/configs/hulk_defconfig | 3 +++ 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/configs/hulk_defconfig b/arch/arm64/configs/hulk_defconfig index f3cf87b98f3ef..f3a15e856b5f7 100644 --- a/arch/arm64/configs/hulk_defconfig +++ b/arch/arm64/configs/hulk_defconfig @@ -608,6 +608,7 @@ CONFIG_EFI_VARS=y CONFIG_EFI_ESRT=y CONFIG_EFI_VARS_PSTORE=y CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y +CONFIG_EFI_FAKE_MEMMAP=y CONFIG_EFI_PARAMS_FROM_FDT=y CONFIG_EFI_RUNTIME_WRAPPERS=y CONFIG_EFI_ARMSTUB=y @@ -1003,6 +1004,8 @@ CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y CONFIG_ARCH_HAS_PTE_SPECIAL=y CONFIG_PIN_MEMORY=y CONFIG_PID_RESERVE=y +CONFIG_MEMORY_RELIABLE=y +CONFIG_CLEAR_FREELIST_PAGE=y CONFIG_NET=y CONFIG_NET_INGRESS=y CONFIG_NET_EGRESS=y