From: Marc Zyngier <maz@kernel.org> mainline inclusion from mainline-v6.9-rc1 commit cfc680bb04c54e61faa51a34d8383a0aa25b583f category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9795 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... ---------------------------------------- ARMv9.5 has infroduced ID_AA64MMFR4_EL1 with a bunch of new features. Add the corresponding layout. This is extracted from the public ARM SysReg_xml_A_profile-2023-09 delivery, timestamped d55f5af8e09052abe92a02adf820deea2eaed717. Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Reviewed-by: Miguel Luis <miguel.luis@oracle.com> Link: https://lore.kernel.org/r/20240122181344.258974-5-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev> Signed-off-by: Zeng Heng <zengheng4@huawei.com> --- arch/arm64/tools/sysreg | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index c4ff6cf83623..864f7d0b67ed 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -2240,6 +2240,43 @@ UnsignedEnum 3:0 TCRX EndEnum EndSysreg +Sysreg ID_AA64MMFR4_EL1 3 0 0 7 4 +Res0 63:40 +UnsignedEnum 39:36 E3DSE + 0b0000 NI + 0b0001 IMP +EndEnum +Res0 35:28 +SignedEnum 27:24 E2H0 + 0b0000 IMP + 0b1110 NI_NV1 + 0b1111 NI +EndEnum +UnsignedEnum 23:20 NV_frac + 0b0000 NV_NV2 + 0b0001 NV2_ONLY +EndEnum +UnsignedEnum 19:16 FGWTE3 + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 15:12 HACDBS + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 11:8 ASID2 + 0b0000 NI + 0b0001 IMP +EndEnum +SignedEnum 7:4 EIESB + 0b0000 NI + 0b0001 ToEL3 + 0b0010 ToELx + 0b1111 ANY +EndEnum +Res0 3:0 +EndSysreg + Sysreg SCTLR_EL1 3 0 1 0 0 Field 63 TIDCP Field 62 SPINTMASK -- 2.43.0

From: Marc Zyngier <maz@kernel.org> mainline inclusion from mainline-v6.16-rc1 commit eef33835bf6f297faa222f48bf941d57d2f8bda0 category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9795 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... ---------------------------------------- Resync the ID_AA64MMFR4_EL1 with the architectue description. This results in: - the new PoPS field - the new NV2P1 value for the NV_frac field - the new RMEGDI field - the new SRMASK field These fields have been generated from the reference JSON file. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Zeng Heng <zengheng4@huawei.com> --- arch/arm64/tools/sysreg | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 864f7d0b67ed..4770bc4b7690 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -2241,12 +2241,21 @@ EndEnum EndSysreg Sysreg ID_AA64MMFR4_EL1 3 0 0 7 4 -Res0 63:40 +Res0 63:48 +UnsignedEnum 47:44 SRMASK + 0b0000 NI + 0b0001 IMP +EndEnum +Res0 43:40 UnsignedEnum 39:36 E3DSE 0b0000 NI 0b0001 IMP EndEnum -Res0 35:28 +Res0 35:32 +UnsignedEnum 31:28 RMEGDI + 0b0000 NI + 0b0001 IMP +EndEnum SignedEnum 27:24 E2H0 0b0000 IMP 0b1110 NI_NV1 @@ -2255,6 +2264,7 @@ EndEnum UnsignedEnum 23:20 NV_frac 0b0000 NV_NV2 0b0001 NV2_ONLY + 0b0010 NV2P1 EndEnum UnsignedEnum 19:16 FGWTE3 0b0000 NI @@ -2274,7 +2284,10 @@ SignedEnum 7:4 EIESB 0b0010 ToELx 0b1111 ANY EndEnum -Res0 3:0 +UnsignedEnum 3:0 PoPS + 0b0000 NI + 0b0001 IMP +EndEnum EndSysreg Sysreg SCTLR_EL1 3 0 1 0 0 -- 2.43.0

From: Liao Chang <liaochang1@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9795 ---------------------------------------- Add the initial infrastructure to support the Armv9.3 FEAT_TLBID extension, which allows OSPM to target TLB invalidations to a specific domain. A TLBI domain can be contained in multiple parent domains. This patch provides: - Kconfig option CONFIG_ARM64_TLBID to enable the feature - TLBIDIDR_EL1 register definition and CPU feature detection - ACPI TLBI table parser to discover TLBI domains for processors and SMMU [1]. - Data structures to manage TLBI domain to CPU mapping. - Help functions to choose the best TLBI domain for a specific set of CPUs. The TLBI table lists all the PE and SMMU components and the TLBI domains that each component belongs to. This information is used to optimize TLB invalidation by targeting specific domains instead of broadcasting to all CPUs. [1] https://github.com/user-attachments/files/25049127/ACPI_TLBID_v3.docx [2] https://github.com/masoncl/review-prompts.git CC: Xia Qinxin <xiaqinxin@huawei.com> CC: Zheng Tian <zhengtian10@huawei.com> CC: Yang Jinqian <yangjinqian1@huawei.com> CC: Wang Kefeng <wangkefeng.wang@huawei.com> CC: Yu Jiacheng <yujiacheng3@huawei.com> Signed-off-by: Liao Chang <liaochang1@huawei.com> Signed-off-by: Xinyu Zheng <zhengxinyu6@huawei.com> Signed-off-by: Zeng Heng <zengheng4@huawei.com> ------- v2->v3: 1. Fix bug about the calculation of number of system-wide domains. 2. Fix bug about the TLBID sanitization for vCPU hotplug 3. Polish implementation based on the review feedback 4. Fix compilation error for including tlbidomain.h in tlbflush.h 5. Fix fatal bug in validate_ids() that causes TLBID does't take effect ------- v1->v2: The /kreview skill in [2] discover 4 bugs: 1. validate_ids() incorrectly validates tlbid_data.nvis and tlbid_data.nvos instead of the passed parameters. 2. pick_best_domain() out-of-bounds access: Accesses cpu_domain[nr_cpu] without checking if nr_cpu < nr_domain. 3. tlbid_chip_data_init() has NULL pointer dereference due to find_target() can return NULL. 4. tlbid_chip_data_init() accesses cpu_domain[nr_cpu] where nr_cpu can exceed nr_domain. --- arch/arm64/include/asm/cpu.h | 1 + arch/arm64/include/asm/cpufeature.h | 7 + arch/arm64/include/asm/tlbidomain.h | 34 +++ arch/arm64/kernel/cpufeature.c | 2 + arch/arm64/kernel/cpuinfo.c | 3 + arch/arm64/mm/Makefile | 1 + arch/arm64/mm/tlbidomain.c | 339 ++++++++++++++++++++++++++++ arch/arm64/tools/sysreg | 11 + drivers/acpi/arm64/Makefile | 1 + drivers/acpi/arm64/init.c | 2 + drivers/acpi/arm64/init.h | 1 + drivers/acpi/arm64/tlbid.c | 107 +++++++++ drivers/acpi/tables.c | 9 + include/acpi/actbl2.h | 29 +++ include/linux/acpi.h | 1 + 15 files changed, 548 insertions(+) create mode 100644 arch/arm64/include/asm/tlbidomain.h create mode 100644 arch/arm64/mm/tlbidomain.c create mode 100644 drivers/acpi/arm64/tlbid.c diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h index 56eb090ae48d..ede40c40d410 100644 --- a/arch/arm64/include/asm/cpu.h +++ b/arch/arm64/include/asm/cpu.h @@ -47,6 +47,7 @@ struct cpuinfo_arm64 { u64 reg_gmid; u64 reg_smidr; u64 reg_mpamidr; + u64 reg_tlbididr; u64 reg_id_aa64dfr0; u64 reg_id_aa64dfr1; diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 9ff5141518b4..8d46fff3c104 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -641,6 +641,13 @@ static inline bool id_aa64pfr1_mte(u64 pfr1) return val >= ID_AA64PFR1_EL1_MTE_MTE2; } +static inline bool id_aa64mmfr4_tlbid(u64 mmfr4) +{ + u32 val = cpuid_feature_extract_unsigned_field(mmfr4, ID_AA64MMFR4_EL1_TLBID_SHIFT); + + return val > 0; +} + void __init setup_cpu_features(void); void check_local_cpu_capabilities(void); diff --git a/arch/arm64/include/asm/tlbidomain.h b/arch/arm64/include/asm/tlbidomain.h new file mode 100644 index 000000000000..1dac3771ff57 --- /dev/null +++ b/arch/arm64/include/asm/tlbidomain.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2026 Huawei Ltd. + */ +#ifndef __ASM_TLBIDOMAIN_H +#define __ASM_TLBIDOMAIN_H + +#include <linux/types.h> + +struct tlbi_target { + u32 identifier; + s32 *domains; + s32 nr_domain; + struct list_head list; +}; + +#define TLBI_RT_DOMAIN_WIDTH (16) +#define TLBI_INV_DOMAIN (-1) + +#ifdef CONFIG_ARM64_TLBID +void add_target(int type, struct tlbi_target *target); +int pick_best_domain(const cpumask_t *active_cpus); +int __init parse_tlbid_topology(void); +int __init tlbid_data_init(void); +void __init tlbid_data_reset(void); +void __init tlbid_early_init(void); + +#else +static inline int pick_best_domain(const cpumask_t *active_cpus) { return 0; } +static inline void __init tlbid_early_init(void) { } + +#endif /* ! CONFIG_ARM64_TLBID */ +#endif + diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 63115dfa0236..0ff73f95279b 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -95,6 +95,7 @@ #include <asm/traps.h> #include <asm/vectors.h> #include <asm/virt.h> +#include <asm/tlbidomain.h> /* Kernel representation of AT_HWCAP and AT_HWCAP2 */ static DECLARE_BITMAP(elf_hwcap, MAX_CPU_FEATURES) __read_mostly; @@ -3951,6 +3952,7 @@ void __init setup_cpu_features(void) sve_setup(); sme_setup(); + tlbid_early_init(); minsigstksz_setup(); mpam_extra_caps(); diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c index 56b90127e78a..b320e9211127 100644 --- a/arch/arm64/kernel/cpuinfo.c +++ b/arch/arm64/kernel/cpuinfo.c @@ -340,6 +340,9 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) __cpuinfo_store_cpu_32bit(&info->aarch32); + if (id_aa64mmfr4_tlbid(info->reg_id_aa64mmfr4)) + info->reg_tlbididr = read_cpuid(TLBIDIDR_EL1); + if (IS_ENABLED(CONFIG_ARM64_MPAM) && mpam_detect_is_enabled() && (id_aa64pfr0_mpam(info->reg_id_aa64pfr0) || diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index c02aeb729717..4d92d42826bf 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o +obj-$(CONFIG_ARM64_TLBID) += tlbidomain.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/tlbidomain.c b/arch/arm64/mm/tlbidomain.c new file mode 100644 index 000000000000..9133167943d2 --- /dev/null +++ b/arch/arm64/mm/tlbidomain.c @@ -0,0 +1,339 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * This file implements the kernel functions rely on FEAT_TLBID + * + * Copyright (C) 2026 Huawei Ltd. + */ + +#define pr_fmt(fmt) "tlbidomain: " fmt + +#include <linux/acpi.h> +#include <linux/cpumask.h> + +#include <asm/cpu.h> +#include <asm/sysreg.h> +#include <asm/tlbidomain.h> +#include <asm/cpufeature.h> + +struct tlbi_domain { + cpumask_t cpumask; + int id; +}; + +struct tlbid_data { + struct list_head target_list[ACPI_TLBI_TYPE_RESERVED]; + unsigned long *found_domains; + u32 nr_domain; + struct tlbi_domain *cpu_domain[NR_CPUS]; + u8 nis; /* TLBIDIDR_EL1.NIS */ + u8 nvis; /* TLBIDIDR_EL1.NVIS */ + u8 nos; /* TLBIDIDR_EL1.NOS */ + u8 nvos; /* TLBIDIDR_EL1.NVOS */ +}; + +/* + * Global TLBI domain data. + * Actual supported domain bits will be reduced based on hardware capability. + */ +static struct tlbid_data tlbid_data __read_mostly = { + .nr_domain = 1, + .nis = TLBI_RT_DOMAIN_WIDTH, + .nvis = TLBI_RT_DOMAIN_WIDTH, + .nos = TLBI_RT_DOMAIN_WIDTH, + .nvos = TLBI_RT_DOMAIN_WIDTH, +}; + +/* + * validate_ids - Validate TLBID domain bit configurations + * + * Validates that NIS/NVIS and NOS/NVOS configurations follow Arm architecture + * constraints: if outer domain bits are zero, inner domain bits must also be + * zero. Outputs error and zeroes out all domain counts if validation fails. + */ +static void validate_ids(u8 nis, u8 nvis, u8 nos, u8 nvos) +{ + bool inv = false; + + switch (nos) { + case 0: + inv = tlbid_data.nvos > 0; + break; + case 1 ... 8: + inv = tlbid_data.nvos > 6; + break; + case 9 ... 16: + inv = tlbid_data.nvos > 5; + break; + } + + if (inv) { + pr_err("Illegal value of NVOS(%u) for NOS(%u)\n", + tlbid_data.nvos, tlbid_data.nos); + goto on_error; + } + + switch (nis) { + case 0: + inv = tlbid_data.nvis > 0; + break; + case 1 ... 8: + inv = tlbid_data.nvis > 6; + break; + case 9 ... 16: + inv = tlbid_data.nvis > 5; + break; + } + + if (!inv) + return; + + pr_err("Illegal value of NVIS(%u) for NIS(%u)\n", + tlbid_data.nvis, tlbid_data.nis); +on_error: + tlbid_data.nis = 0; + tlbid_data.nvis = 0; + tlbid_data.nos = 0; + tlbid_data.nvos = 0; +} + +/* + * find_target - Find TLBI target by type and identifier + * + * Looks up the tlbi_target structure associated with the given type and cpu. + * For PROCESSOR type, converts CPU logical ID to ACPI Processor UID. + */ +static struct tlbi_target *find_target(int type, int cpu) +{ + struct tlbi_target *target = NULL; + struct list_head *head; + int uid; + + switch (type) { + case ACPI_TLBI_TYPE_PROCESSOR: + uid = get_acpi_id_for_cpu(cpu); + break; + default: + pr_err("Only support Processor as TLBI broadcast target\n"); + return NULL; + } + + head = &tlbid_data.target_list[type]; + list_for_each_entry(target, head, list) { + if (target->identifier == uid) + break; + } + + return target; +} + +void add_target(int type, struct tlbi_target *target) +{ + s32 i; + + for (i = 0; i < target->nr_domain; i++) + set_bit(target->domains[i], tlbid_data.found_domains); + + INIT_LIST_HEAD(&target->list); + list_add_tail(&target->list, &tlbid_data.target_list[type]); +} + +/* + * pick_best_domain - Select optimal TLBI domain for given CPUs + * + * Iterates through cpu_domain table to find a domain whose cpumask exactly + * matches the given active CPUs. This allows targeted TLBI instead of + * broadcasting to all CPUs, improving TLB invalidation efficiency. + * Returns domain ID on success, 0 on failure. + */ +int pick_best_domain(const cpumask_t *active_cpus) +{ + struct tlbi_domain *domain; + cpumask_t tmp; + int nr_cpu; + + /* Domain 0 is the only supported for Innern Shareable TLBI */ + if (unlikely(tlbid_data.nr_domain == 1)) + return 0; + + nr_cpu = cpumask_weight(active_cpus); + if (unlikely(nr_cpu == 0)) + return 0; + + do { + domain = &tlbid_data.cpu_domain[nr_cpu][0]; + while (domain->id != TLBI_INV_DOMAIN) { + if (cpumask_and(&tmp, active_cpus, &domain->cpumask) && + cpumask_equal(&tmp, active_cpus)) + return domain->id; + domain++; + } + } while (++nr_cpu < num_possible_cpus()); + + return 0; +} + +int __init tlbid_data_init(void) +{ + s32 max_domains = (1 << max(tlbid_data.nis, tlbid_data.nos)); + int type; + + if (!tlbid_data.nis || !tlbid_data.nos) + return -EINVAL; + + tlbid_data.found_domains = kcalloc(BITS_TO_LONGS(max_domains), + sizeof(long), GFP_KERNEL); + if (!tlbid_data.found_domains) + return -ENOMEM; + + type = ACPI_TLBI_TYPE_PROCESSOR; + while (type != ACPI_TLBI_TYPE_RESERVED) + INIT_LIST_HEAD(&tlbid_data.target_list[type++]); + + return 0; +} + +void __init tlbid_data_reset(void) +{ + kfree(tlbid_data.found_domains); + tlbid_data.found_domains = NULL; + tlbid_data.nr_domain = 1; + tlbid_data.nis = 0; + tlbid_data.nos = 0; + tlbid_data.nvis = 0; + tlbid_data.nvos = 0; +} + +/* + * parse_tlbid_topology - Initialize TLBI domain mapping tables + * + * Allocates and populates the cpu_domain table: + * 1. Build CPU-to-domain mapping from ACPI-parsed target data + * 2. Reorganize data for efficient lookup (domain x CPU matrix) + * + * Returns 0 on success, -ENOMEM on allocation failure, -EVINAL on topology + * parse failure + */ +int __init parse_tlbid_topology(void) +{ + struct tlbi_domain *src, *dst; + struct tlbi_target *target; + int cpu, nr_cpu, err; + s32 i, j; + + tlbid_data.nr_domain = bitmap_weight(tlbid_data.found_domains, + ((1<<TLBI_RT_DOMAIN_WIDTH)-1)); + if (tlbid_data.nr_domain >= (1 << max(tlbid_data.nis, tlbid_data.nos))) { + pr_err("TLBI Domains ACPI reports are more than CPU support\n"); + err = -EINVAL; + goto on_error; + } + pr_info("found %d TLBI Domain(s).\n", tlbid_data.nr_domain); + + /* + * Allocate a sparse table to map the CPU count to domain list sharing + * the same CPU count. + * + * NOTICE: Each domain is assumed to contain all system-wide SMMUs, + * Any broadcast in a specific domain must wait replies from all SMMUs + * before it complete. + */ + for (nr_cpu = 0; nr_cpu < num_possible_cpus(); nr_cpu++) { + tlbid_data.cpu_domain[nr_cpu] = + kcalloc(tlbid_data.nr_domain, + sizeof(struct tlbi_domain), GFP_KERNEL); + if (!tlbid_data.cpu_domain[nr_cpu]) { + err = -ENOMEM; + goto on_error; + } + } + + for (i = 0; i < tlbid_data.nr_domain; i++) { + cpumask_clear(&tlbid_data.cpu_domain[0][i].cpumask); + tlbid_data.cpu_domain[0][i].id = i; + } + + /* Initialize the cpumask associated with each tlbi domain */ + for_each_possible_cpu(cpu) { + target = find_target(ACPI_TLBI_TYPE_PROCESSOR, cpu); + if (!target) { + err = -EINVAL; + goto on_error; + } + i = 0; + while ((j = target->domains[i++]) != TLBI_INV_DOMAIN) { + src = &tlbid_data.cpu_domain[0][j]; + /* Record all CPUs belong to the same domain */ + cpumask_set_cpu(cpu, &(src->cpumask)); + } + } + + /* Move each domain data onto the right position in sparse table */ + for (i = 0; i < tlbid_data.nr_domain; i++) { + src = &tlbid_data.cpu_domain[0][i]; + nr_cpu = cpumask_weight(&(src->cpumask)); + j = 0; + do { + dst = &tlbid_data.cpu_domain[nr_cpu][j++]; + } while (dst->id != TLBI_INV_DOMAIN); + memcpy(dst, src, sizeof(*dst)); + cpumask_clear(&src->cpumask); + src->id = TLBI_INV_DOMAIN; + } + + return 0; + +on_error: + for (nr_cpu = 0; nr_cpu < num_possible_cpus(); nr_cpu++) + kfree(tlbid_data.cpu_domain[nr_cpu]); + return err; +} + +/* + * tlbid_early_init - Early initialization of TLBID hardware capability after + * all CPUs activate. + * + * Called early during boot from setup_system_features() before ACPI is parsed. + * Reads TLBIDIDR_EL1 register on each CPU to discover hardware capabilities + * (NIS, NVIS, NOS, NVOS bits), and takes the minimum across all CPUs. + * Validates the combined configuration via validate_ids(). + */ +void __init tlbid_early_init(void) +{ + u16 nis, nos, nvis, nvos; + u64 tlbididr; + int cpu; + + if (!system_supports_tlbid()) { + tlbid_data.nis = 0; + tlbid_data.nos = 0; + tlbid_data.nvis = 0; + tlbid_data.nvos = 0; + return; + } + + for_each_online_cpu(cpu) { + tlbididr = per_cpu_ptr(&cpu_data, cpu)->reg_tlbididr; + nis = FIELD_GET(TLBIDIDR_EL1_NIS_MASK, tlbididr); + nos = FIELD_GET(TLBIDIDR_EL1_NOS_MASK, tlbididr); + nvis = FIELD_GET(TLBIDIDR_EL1_NVIS_MASK, tlbididr); + nvos = FIELD_GET(TLBIDIDR_EL1_NVOS_MASK, tlbididr); + + if (tlbid_data.nis > nis) + tlbid_data.nis = nis; + + if (tlbid_data.nos > nos) + tlbid_data.nos = nos; + + if (tlbid_data.nvis > nvis) + tlbid_data.nvis = nvis; + + if (tlbid_data.nvos > nvos) + tlbid_data.nvos = nvos; + } + + validate_ids(tlbid_data.nis, tlbid_data.nvis, + tlbid_data.nos, tlbid_data.nvos); + + pr_info("TLBI Domain bits: nis %d nos %d nvis %d nvos %d\n", + tlbid_data.nis, tlbid_data.nos, tlbid_data.nvis, tlbid_data.nvos); +} diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 470b07be5d2c..5a76321691c5 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -3524,3 +3524,14 @@ Field 5 F Field 4 P Field 3:0 Align EndSysreg + +Sysreg TLBIDIDR_EL1 3 0 10 4 6 +Res0 63:45 +Field 44:40 NVOS +Res0 39:37 +Field 36:32 NOS +Res0 31:13 +Field 12:8 NVIS +Res0 7:5 +Field 4:0 NIS +EndSysreg diff --git a/drivers/acpi/arm64/Makefile b/drivers/acpi/arm64/Makefile index 9497a7777729..2c068491f16e 100644 --- a/drivers/acpi/arm64/Makefile +++ b/drivers/acpi/arm64/Makefile @@ -5,5 +5,6 @@ obj-$(CONFIG_ACPI_GTDT) += gtdt.o obj-$(CONFIG_ACPI_APMT) += apmt.o obj-$(CONFIG_ARM_AMBA) += amba.o obj-$(CONFIG_ACPI_MPAM) += mpam.o +obj-$(CONFIG_ARM64_TLBID) += tlbid.o obj-y += dma.o init.o diff --git a/drivers/acpi/arm64/init.c b/drivers/acpi/arm64/init.c index d0c8aed90fd1..309f29b63f5d 100644 --- a/drivers/acpi/arm64/init.c +++ b/drivers/acpi/arm64/init.c @@ -12,4 +12,6 @@ void __init acpi_arm_init(void) acpi_iort_init(); if (IS_ENABLED(CONFIG_ARM_AMBA)) acpi_amba_init(); + if (IS_ENABLED(CONFIG_ARM64_TLBID)) + acpi_tlbi_init(); } diff --git a/drivers/acpi/arm64/init.h b/drivers/acpi/arm64/init.h index dcc277977194..c036e2622a2f 100644 --- a/drivers/acpi/arm64/init.h +++ b/drivers/acpi/arm64/init.h @@ -5,3 +5,4 @@ void __init acpi_agdi_init(void); void __init acpi_apmt_init(void); void __init acpi_iort_init(void); void __init acpi_amba_init(void); +void __init acpi_tlbi_init(void); diff --git a/drivers/acpi/arm64/tlbid.c b/drivers/acpi/arm64/tlbid.c new file mode 100644 index 000000000000..3268c61982ba --- /dev/null +++ b/drivers/acpi/arm64/tlbid.c @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * This file implements handling of + * Arm Translation Lookaside Buffer Invalidate Domain table (TLBI) + * + * Copyright (C) 2026, Huawei Ltd. + */ +#define pr_fmt(fmt) "ACPI: TLBID: " fmt + +#include <linux/acpi.h> +#include <linux/kernel.h> +#include <linux/list.h> +#include <linux/slab.h> + +#include "init.h" + +#include <asm/tlbidomain.h> + +static struct acpi_table_header *acpi_tlbid_table; + +/* + * tlbid_parse_subtable - Parse ACPI TLBI subtable entry + * + * Called for each entry in the ACPI TLBI table. Allocates a tlbi_target + * structure and copies domain IDs from the ACPI table. Builds the + * found_domains bitmap for later domain count calculation. + */ +static int __init tlbid_parse_subtable(union acpi_subtable_headers *header, + unsigned long end) +{ + struct acpi_tlbi_subtable *hdr = (struct acpi_tlbi_subtable *)header; + struct tlbi_target *target; + + if (!hdr) + return -EINVAL; + + target = kzalloc(sizeof(*target), GFP_KERNEL); + if (!target) + return -ENOMEM; + + /* The ACPI Processor UID of the corresponding GICC MADT entry */ + target->nr_domain = hdr->nr_domain; + target->identifier = hdr->identifier; + target->domains = kcalloc(hdr->nr_domain + 1, sizeof(*target->domains), + GFP_KERNEL); + if (!target->domains) { + kfree(target); + return -ENOMEM; + } + + memcpy(target->domains, + (void *)acpi_tlbid_table + hdr->offset, + hdr->nr_domain * sizeof(*target->domains)); + target->domains[hdr->nr_domain] = TLBI_INV_DOMAIN; + + add_target(header->tlbi.type, target); + + return 0; +} + +/* + * acpi_tlbi_init - Initialize TLBID from ACPI table + * + * Main entry point for TLBI table parsing. Gets the ACPI TLBI table, + * parses Processor and SMMU subtable entries, calculates total domain + * count, validates against hardware capability, and calls + * tlbid_chip_data_init() to build domain mapping tables. + * Outputs appropriate error messages on failure. + */ +void __init acpi_tlbi_init(void) +{ + enum acpi_tlbi_target_type type; + acpi_status status; + int ret; + + if (tlbid_data_init() < 0) + goto on_error; + + status = acpi_get_table(ACPI_SIG_TLBI, 0, &acpi_tlbid_table); + if (ACPI_FAILURE(status)) + goto on_error; + + type = ACPI_TLBI_TYPE_PROCESSOR; + while (type != ACPI_TLBI_TYPE_RESERVED) { + ret = acpi_table_parse_entries(ACPI_SIG_TLBI, + sizeof(struct acpi_table_tlbi), + type, tlbid_parse_subtable, 0); + if (ret) { + pr_err("Failed to get TLBID subtable for %s\n", + type == ACPI_TLBI_TYPE_PROCESSOR ? "Processor":"SMMU"); + goto on_error; + } + type++; + } + + if (parse_tlbid_topology()) { + pr_warn("Failed to initialize TLBI Domain data\n"); + goto on_error; + } + + acpi_put_table(acpi_tlbid_table); + return; + +on_error: + tlbid_data_reset(); + acpi_put_table(acpi_tlbid_table); +} diff --git a/drivers/acpi/tables.c b/drivers/acpi/tables.c index 4fca04c8dd2d..62d786110ebf 100644 --- a/drivers/acpi/tables.c +++ b/drivers/acpi/tables.c @@ -42,6 +42,7 @@ enum acpi_subtable_type { ACPI_SUBTABLE_HMAT, ACPI_SUBTABLE_PRMT, ACPI_SUBTABLE_CEDT, + ACPI_SUBTABLE_TLBI, }; struct acpi_subtable_entry { @@ -287,6 +288,8 @@ acpi_get_entry_type(struct acpi_subtable_entry *entry) return 0; case ACPI_SUBTABLE_CEDT: return entry->hdr->cedt.type; + case ACPI_SUBTABLE_TLBI: + return entry->hdr->tlbi.type; } return 0; } @@ -303,6 +306,8 @@ acpi_get_entry_length(struct acpi_subtable_entry *entry) return entry->hdr->prmt.length; case ACPI_SUBTABLE_CEDT: return entry->hdr->cedt.length; + case ACPI_SUBTABLE_TLBI: + return entry->hdr->tlbi.length; } return 0; } @@ -319,6 +324,8 @@ acpi_get_subtable_header_length(struct acpi_subtable_entry *entry) return sizeof(entry->hdr->prmt); case ACPI_SUBTABLE_CEDT: return sizeof(entry->hdr->cedt); + case ACPI_SUBTABLE_TLBI: + return sizeof(entry->hdr->tlbi); } return 0; } @@ -332,6 +339,8 @@ acpi_get_subtable_type(char *id) return ACPI_SUBTABLE_PRMT; if (strncmp(id, ACPI_SIG_CEDT, 4) == 0) return ACPI_SUBTABLE_CEDT; + if (strncmp(id, ACPI_SIG_TLBI, 4) == 0) + return ACPI_SUBTABLE_TLBI; return ACPI_SUBTABLE_COMMON; } diff --git a/include/acpi/actbl2.h b/include/acpi/actbl2.h index 97b05bd334a5..29dda0318c0f 100644 --- a/include/acpi/actbl2.h +++ b/include/acpi/actbl2.h @@ -26,6 +26,7 @@ */ #define ACPI_SIG_AGDI "AGDI" /* Arm Generic Diagnostic Dump and Reset Device Interface */ #define ACPI_SIG_APMT "APMT" /* Arm Performance Monitoring Unit table */ +#define ACPI_SIG_TLBI "TLBI" /* Translation Lookaside Buffer Invalidate Domain Table */ #define ACPI_SIG_BDAT "BDAT" /* BIOS Data ACPI Table */ #define ACPI_SIG_CCEL "CCEL" /* CC Event Log Table */ #define ACPI_SIG_CDAT "CDAT" /* Coherent Device Attribute Table */ @@ -341,6 +342,34 @@ enum acpi_apmt_node_type { #define ACPI_APMT_OVFLW_IRQ_FLAGS_TYPE_WIRED (0<<1) +/******************************************************************************* + * TLBI - Arm Translation Lookaside Buffer Invalidation Domain Table (TLBI) + * + ******************************************************************************/ +struct acpi_table_tlbi { + struct acpi_table_header header; /* Common ACPI table header */ + u32 nr_subtable; +}; + +struct acpi_tlbi_header { + u16 type; + u16 reserved; + u32 length; +}; + +struct acpi_tlbi_subtable { + struct acpi_tlbi_header header; + u32 identifier; + u32 nr_domain; + u32 offset; +}; + +enum acpi_tlbi_target_type { + ACPI_TLBI_TYPE_PROCESSOR = 1, + ACPI_TLBI_TYPE_SMMU, + ACPI_TLBI_TYPE_RESERVED, +}; + /******************************************************************************* * * BDAT - BIOS Data ACPI Table diff --git a/include/linux/acpi.h b/include/linux/acpi.h index e59a74273e6e..acd1a9241829 100644 --- a/include/linux/acpi.h +++ b/include/linux/acpi.h @@ -127,6 +127,7 @@ union acpi_subtable_headers { struct acpi_hmat_structure hmat; struct acpi_prmt_module_header prmt; struct acpi_cedt_header cedt; + struct acpi_tlbi_header tlbi; }; typedef int (*acpi_tbl_table_handler)(struct acpi_table_header *table); -- 2.43.0

From: Jinjiang Tu <tujinjiang@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9795 ---------------------------------------- This patch optimizes flush_tlb_mm() by using TLBID when available: - Local invalidation: When the mm is only active on the current CPU, use non-shareable TLB invalidation (aside1) for better performance. - Domain-based invalidation: When the mm is active on multiple CPUs but not all, use TLBID to target only the relevant CPUs, reducing cache coherency traffic compared to full broadcast. - Broadcast fallback: When TLBID is not supported or when targeting all CPUs, maintain the existing behavior (aside1is). The scope detection logic (flush_tlb_user_scope()) determines the optimal invalidation strategy based on mm_cpumask() and TLBID capabilities. This optimization is particularly beneficial for: - Workloads with many short-lived processes - Systems with high CPU counts where broadcast TLB shootdowns are costly - Scenarios where process memory is mostly local to a subset of CPUs Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com> Signed-off-by: Zeng Heng <zengheng4@huawei.com> --- arch/arm64/include/asm/tlbflush.h | 66 +++++++++++++++++++++++++++++-- 1 file changed, 63 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index c082cace5683..a0555b81c1e5 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -16,6 +16,7 @@ #include <linux/mmu_notifier.h> #include <asm/cputype.h> #include <asm/mmu.h> +#include <asm/tlbidomain.h> /* * Raw TLBI operations. @@ -165,6 +166,51 @@ static inline unsigned long get_trans_granule(void) (__pages >> (5 * (scale) + 1)) - 1; \ }) +enum tlb_flush_scope { + TLB_FLUSH_SCOPE_LOCAL, + TLB_FLUSH_SCOPE_DOMAIN, + TLB_FLUSH_SCOPE_BROADCAST, +}; + +#ifdef CONFIG_ARM64_TLBID +/* This macro creates a properly formatted VA operand for the TLBID */ +#define __TLBI_DOMAIN(asid, domain) \ + ({ \ + unsigned long __ta = 0; \ + __ta |= FIELD_PREP(GENMASK_ULL(15, 0), domain); \ + __ta |= FIELD_PREP(GENMASK_ULL(63, 48), asid); \ + __ta; \ + }) + +/* + * Determines whether the user tlbi invalidation can be performed only on the + * local CPU or whether it needs to be multicast or broadcast. + */ +static inline enum tlb_flush_scope flush_tlb_user_scope(struct mm_struct *mm) +{ + if (!system_supports_tlbid()) + return TLB_FLUSH_SCOPE_BROADCAST; + + /* check if the tlbflush needs to be sent to other CPUs */ + if (cpumask_any_but(mm_cpumask(mm), smp_processor_id()) >= + nr_cpu_ids) + return TLB_FLUSH_SCOPE_LOCAL; + + return TLB_FLUSH_SCOPE_DOMAIN; +} +#else +#define __TLBI_DOMAIN(asid, domain) \ + ( \ + (void)(domain), \ + __TLBI_VADDR(0, (asid)) \ + ) + +static inline enum tlb_flush_scope flush_tlb_user_scope(struct mm_struct *mm) +{ + return TLB_FLUSH_SCOPE_BROADCAST; +} +#endif + /* * TLB Invalidation * ================ @@ -252,12 +298,26 @@ static inline void flush_tlb_all(void) static inline void flush_tlb_mm(struct mm_struct *mm) { + enum tlb_flush_scope scope; unsigned long asid; + int domain; dsb(ishst); - asid = __TLBI_VADDR(0, ASID(mm)); - __tlbi(aside1is, asid); - __tlbi_user(aside1is, asid); + scope = flush_tlb_user_scope(mm); + if (scope == TLB_FLUSH_SCOPE_LOCAL) { + asid = __TLBI_VADDR(0, ASID(mm)); + __tlbi(aside1, asid); + __tlbi_user(aside1, asid); + } else if (scope == TLB_FLUSH_SCOPE_BROADCAST) { + asid = __TLBI_VADDR(0, ASID(mm)); + __tlbi(aside1is, asid); + __tlbi_user(aside1is, asid); + } else { + domain = pick_best_domain(mm_cpumask(mm)); + asid = __TLBI_DOMAIN(ASID(mm), domain); + __tlbi(aside1is, asid); + __tlbi_user(aside1is, asid); + } dsb(ish); mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); } -- 2.43.0