From: James Morse <james.morse@arm.com> maillist inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 Reference: https://lore.kernel.org/all/20211015161416.2196-1-james.morse@arm.com/t/#u -------------------------------- The cpus.yaml file describes the cpu nodes, not the cpus node. Rename it to allow integration properties of all the cpus to be described in the cpus node. Also update the in-tree references to the renamed file. They describe CPU node properties, so they must point at cpu.yaml; left alone they would resolve to the cpus container node added later in this series. Signed-off-by: James Morse <james.morse@arm.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- Changes in v3: - update the six references to the CPU node binding, from arm/cpus.yaml to arm/cpu.yaml --- Documentation/devicetree/bindings/arm/{cpus.yaml => cpu.yaml} | 2 +- Documentation/devicetree/bindings/cpu/cpu-capacity.txt | 2 +- Documentation/devicetree/bindings/cpu/cpu-topology.txt | 2 +- Documentation/devicetree/bindings/cpu/idle-states.yaml | 2 +- .../devicetree/bindings/interrupt-controller/apple,aic.yaml | 2 +- .../devicetree/bindings/interrupt-controller/apple,aic2.yaml | 2 +- .../devicetree/bindings/interrupt-controller/arm,gic-v3.yaml | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) rename Documentation/devicetree/bindings/arm/{cpus.yaml => cpu.yaml} (99%) diff --git a/Documentation/devicetree/bindings/arm/cpus.yaml b/Documentation/devicetree/bindings/arm/cpu.yaml similarity index 99% rename from Documentation/devicetree/bindings/arm/cpus.yaml rename to Documentation/devicetree/bindings/arm/cpu.yaml index 0d17cbad9778..95d3c2305676 100644 --- a/Documentation/devicetree/bindings/arm/cpus.yaml +++ b/Documentation/devicetree/bindings/arm/cpu.yaml @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 %YAML 1.2 --- -$id: http://devicetree.org/schemas/arm/cpus.yaml# +$id: http://devicetree.org/schemas/arm/cpu.yaml# $schema: http://devicetree.org/meta-schemas/core.yaml# title: ARM CPUs diff --git a/Documentation/devicetree/bindings/cpu/cpu-capacity.txt b/Documentation/devicetree/bindings/cpu/cpu-capacity.txt index f28e1adad428..e910e045605a 100644 --- a/Documentation/devicetree/bindings/cpu/cpu-capacity.txt +++ b/Documentation/devicetree/bindings/cpu/cpu-capacity.txt @@ -235,4 +235,4 @@ cpus { =========================================== [1] ARM Linux Kernel documentation - CPUs bindings - Documentation/devicetree/bindings/arm/cpus.yaml + Documentation/devicetree/bindings/arm/cpu.yaml diff --git a/Documentation/devicetree/bindings/cpu/cpu-topology.txt b/Documentation/devicetree/bindings/cpu/cpu-topology.txt index 9bd530a35d14..72a59c42b14b 100644 --- a/Documentation/devicetree/bindings/cpu/cpu-topology.txt +++ b/Documentation/devicetree/bindings/cpu/cpu-topology.txt @@ -545,7 +545,7 @@ Example 3: HiFive Unleashed (RISC-V 64 bit, 4 core system) }; =============================================================================== [1] ARM Linux kernel documentation - Documentation/devicetree/bindings/arm/cpus.yaml + Documentation/devicetree/bindings/arm/cpu.yaml [2] Devicetree NUMA binding description Documentation/devicetree/bindings/numa.txt [3] RISC-V Linux kernel documentation diff --git a/Documentation/devicetree/bindings/cpu/idle-states.yaml b/Documentation/devicetree/bindings/cpu/idle-states.yaml index b3a5356f9916..a4ec0fa6ab52 100644 --- a/Documentation/devicetree/bindings/cpu/idle-states.yaml +++ b/Documentation/devicetree/bindings/cpu/idle-states.yaml @@ -247,7 +247,7 @@ description: |+ =========================================== [1] ARM Linux Kernel documentation - CPUs bindings - Documentation/devicetree/bindings/arm/cpus.yaml + Documentation/devicetree/bindings/arm/cpu.yaml [2] ARM Linux Kernel documentation - PSCI bindings Documentation/devicetree/bindings/arm/psci.yaml diff --git a/Documentation/devicetree/bindings/interrupt-controller/apple,aic.yaml b/Documentation/devicetree/bindings/interrupt-controller/apple,aic.yaml index 698588e9aa86..21e322feb23b 100644 --- a/Documentation/devicetree/bindings/interrupt-controller/apple,aic.yaml +++ b/Documentation/devicetree/bindings/interrupt-controller/apple,aic.yaml @@ -92,7 +92,7 @@ properties: cpus: description: Should be a list of phandles to CPU nodes (as described in - Documentation/devicetree/bindings/arm/cpus.yaml). + Documentation/devicetree/bindings/arm/cpu.yaml). required: - apple,fiq-index diff --git a/Documentation/devicetree/bindings/interrupt-controller/apple,aic2.yaml b/Documentation/devicetree/bindings/interrupt-controller/apple,aic2.yaml index 2bde6cc6fe0a..8059d5231f7f 100644 --- a/Documentation/devicetree/bindings/interrupt-controller/apple,aic2.yaml +++ b/Documentation/devicetree/bindings/interrupt-controller/apple,aic2.yaml @@ -95,7 +95,7 @@ properties: $ref: /schemas/types.yaml#/definitions/phandle-array description: Should be a list of phandles to CPU nodes (as described in - Documentation/devicetree/bindings/arm/cpus.yaml). + Documentation/devicetree/bindings/arm/cpu.yaml). required: - apple,fiq-index diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.yaml b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.yaml index 72e70f9d4b4a..9540cfc705eb 100644 --- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.yaml +++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.yaml @@ -154,7 +154,7 @@ properties: maxItems: 1 description: Should be a list of phandles to CPU nodes (as described in - Documentation/devicetree/bindings/arm/cpus.yaml). + Documentation/devicetree/bindings/arm/cpu.yaml). required: - affinity

From: James Morse <james.morse@arm.com> maillist inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 Reference: https://lore.kernel.org/all/20211015161416.2196-1-james.morse@arm.com/t/#u -------------------------------- If the CPUs support HPDS2, and there is a DT description of PBHA values that only affect performance, enable those bits for both TTBR0 and TTBR1. Enabling PBHA requires the hierarchical-permissions to be disabled. Commit 87143f404f33 ("arm64: mm: use XN table mapping attributes for the linear region") used these, but only as an optimisation. Only the necessary PBHA bits are enabled to reduce the risk of an unsafe bit/value being used by accident. Signed-off-by: James Morse <james.morse@arm.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- arch/arm64/Kconfig | 13 ++++ arch/arm64/include/asm/pgtable-hwdef.h | 4 ++ arch/arm64/kernel/cpufeature.c | 83 ++++++++++++++++++++++++-- arch/arm64/tools/cpucaps | 2 +- 4 files changed, 97 insertions(+), 5 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 182a6a06b299..39170028fd0f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2196,6 +2196,19 @@ config ARM64_CNP at runtime, and does not affect PEs that do not implement this feature. +config ARM64_PBHA + bool "Enable support for Page Based Hardware Attributes (PBHA)" + default n + help + Page Based Hardware Attributes (PBHA) allow the SoC hardware to + change behaviour depending on which mapping was used to access + a page of memory. e.g. access via one mapping may always result + in the data being cached, whereas using another mapping of the same + physical memory. + + The behaviour of each PBHA bit is not defined. Say no unless you + are very sure you want this + endmenu # "ARMv8.2 architectural features" menu "ARMv8.3 architectural features" diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index f736a4222190..30f28b51b47e 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -280,6 +280,10 @@ #define TCR_TBI1 (UL(1) << 38) #define TCR_HA (UL(1) << 39) #define TCR_HD (UL(1) << 40) +#define TCR_HPD0 (UL(1) << 41) +#define TCR_HPD1 (UL(1) << 42) +#define TCR_HWU0nn_MASK (UL(0xf) << 43) +#define TCR_HWU1nn_MASK (UL(0xf) << 47) #define TCR_TBID1 (UL(1) << 52) #define TCR_NFD0 (UL(1) << 53) #define TCR_NFD1 (UL(1) << 54) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d3a45df97ff4..bcfae671ca24 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -72,6 +72,7 @@ #include <linux/types.h> #include <linux/minmax.h> #include <linux/mm.h> +#include <linux/of.h> #include <linux/cpu.h> #include <linux/kasan.h> #include <linux/percpu.h> @@ -123,6 +124,8 @@ EXPORT_SYMBOL(arm64_use_ng_mappings); DEFINE_PER_CPU_READ_MOSTLY(const char *, this_cpu_vector) = vectors; +unsigned long __ro_after_init arm64_pbha_perf_only_values; + /* * Permit PER_LINUX32 and execve() of 32-bit binaries even if not all CPUs * support it? @@ -2065,6 +2068,71 @@ static bool has_hw_dbm(const struct arm64_cpu_capabilities *cap, #endif +#ifdef CONFIG_ARM64_PBHA +static u8 pbha_stage1_enable_bits; + +static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, + int scope) +{ + u8 val; + struct device_node *cpus; + const u8 *perf_only_vals; + int num_perf_only_vals, i; + + if (!has_cpuid_feature(cap, scope)) + return false; + + /* + * Calls with scope == SCOPE_LOCAL_CPU need only testing whether this + * cpu has the feature. A later 'system' scope call will check for a + * firmware description. + */ + if (scope == SCOPE_LOCAL_CPU) + return true; + + cpus = of_find_node_by_path("/cpus"); + if (!cpus) + goto done; + + perf_only_vals = of_get_property(cpus, "arm,pbha-performance-only", + &num_perf_only_vals); + if (!perf_only_vals) + goto done; + + /* any listed value is usable at stage 1 */ + for (i = 0 ; i < num_perf_only_vals; i++) { + val = perf_only_vals[i]; + if (val > 0xf) + continue; + + pbha_stage1_enable_bits |= val; + set_bit(val, &arm64_pbha_perf_only_values); + } + +done: + of_node_put(cpus); + + return !!pbha_stage1_enable_bits; +} + +static void cpu_enable_pbha(struct arm64_cpu_capabilities const *cap) +{ + u64 tcr; + + if (!pbha_stage1_enable_bits) + return; + + tcr = read_sysreg(tcr_el1); + tcr |= FIELD_PREP(TCR_HWU0nn_MASK, pbha_stage1_enable_bits); + tcr |= FIELD_PREP(TCR_HWU1nn_MASK, pbha_stage1_enable_bits); + tcr |= FIELD_PREP(TCR_HPD0, 1) | FIELD_PREP(TCR_HPD1, 1); + + write_sysreg(tcr, tcr_el1); + isb(); + local_flush_tlb_all(); +} +#endif /* CONFIG_ARM64_PBHA */ + #ifdef CONFIG_ARM64_AMU_EXTN /* @@ -3192,12 +3260,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { { .capability = ARM64_HAS_PBHA, .type = ARM64_CPUCAP_SYSTEM_FEATURE, - .sys_reg = SYS_ID_AA64MMFR1_EL1, - .sign = FTR_UNSIGNED, - .field_pos = ID_AA64MMFR1_EL1_HPDS_SHIFT, .matches = has_cpuid_feature, - .min_field_value = 2, + ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HPDS, HPDS2) }, +#ifdef CONFIG_ARM64_PBHA + { + .desc = "Page Based Hardware Attributes (PBHA)", + .capability = ARM64_HAS_PBHA_STAGE1, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = plat_can_use_pbha_stage1, + .cpu_enable = cpu_enable_pbha, + ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HPDS, HPDS2) + }, +#endif #ifdef CONFIG_ARM64_SME { .desc = "Scalable Matrix Extension", diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index daac6c5703df..3564b98915c2 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -122,7 +122,7 @@ HAS_BBML3 HAS_SYSINSTR128 HAS_TLBID HAS_PBHA -KABI_RESERVE_15 +HAS_PBHA_STAGE1 KABI_RESERVE_16 KABI_RESERVE_17 KABI_RESERVE_18

From: James Morse <james.morse@arm.com> maillist inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 Reference: https://lore.kernel.org/all/20211015161416.2196-1-james.morse@arm.com/t/#u -------------------------------- There are two conflicting use-cases for PBHA at stage2. We could copy the stage1 PBHA bits to stage2, this would ensure the VMMs memory is exactly reproduced for the guest, including the PBHA bits. The problem here is how the VMM's memory is allocated with the PBHA bits set. The other is allowing the guest to configure PBHA directly. This would allow guest device drivers to map memory with the appropriate PBHA bits. This would only be safe if the guest can be trusted to only generate PBHA values that only affect performance. The arm-arm doesn't describe how the stage1 and stage2 bits are combined. Arm's implementations appear to all have the same behaviour, according to the TRM: stage2 wins. For these CPUs, we can allow a guest to use a PBHA bit by disabling it in VTCR_EL2. We just need to know which bits... The DT describes the values that only affect performance, but if value-5 is safe for use, we can't prevent the guest from using value-1 and value-4. These 'decomposed' values would also need to be listed as only affecting performance. Add a cpufeature for CPUs that have this 'stage2 wins' behaviour. Decompose each performance-only value (5 -> 5, 4, 1), and check each of these values is listed as only affecting performance. If so, the bits of the original value (5) can be used by the guest at stage1. (by clearing the bits from VTCR_EL2) Signed-off-by: James Morse <james.morse@arm.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- Changes in v3: - fix the for-loop style in plat_can_use_pbha_stage1() --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/cpufeature.c | 107 +++++++++++++++++++++++++++- arch/arm64/kernel/image-vars.h | 2 + arch/arm64/kvm/hyp/pgtable.c | 9 ++- arch/arm64/tools/cpucaps | 2 +- 5 files changed, 117 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 7a0c8981120b..e151c688606b 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -106,6 +106,7 @@ struct arm64_ftr_reg { }; extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; +extern unsigned long arm64_pbha_stage2_safe_bits; /* * CPU capabilities: diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 5c6d85c684a8..b218985a3edc 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -81,6 +81,7 @@ #include <asm/cpu.h> #include <asm/cpufeature.h> #include <asm/cpu_ops.h> +#include <asm/cputype.h> #include <asm/fpsimd.h> #include <asm/hwcap.h> #include <asm/insn.h> @@ -126,6 +127,7 @@ DEFINE_PER_CPU_READ_MOSTLY(const char *, this_cpu_vector) = vectors; unsigned long __ro_after_init arm64_pbha_perf_only_values; EXPORT_SYMBOL(arm64_pbha_perf_only_values); +unsigned long __ro_after_init arm64_pbha_stage2_safe_bits; /* * Permit PER_LINUX32 and execve() of 32-bit binaries even if not all CPUs @@ -2069,13 +2071,50 @@ static bool has_hw_dbm(const struct arm64_cpu_capabilities *cap, #endif + #ifdef CONFIG_ARM64_PBHA static u8 pbha_stage1_enable_bits; +static DEFINE_SPINLOCK(pbha_dt_lock); + +/* For the value 5, return a bitmap with bits 5, 4, and 1 set. */ +static unsigned long decompose_pbha_values(u8 val) +{ + int i; + unsigned long mask = 0; + + for (i = 1; i <= 15; i++) { + if ((i & val) == i) + set_bit(i, &mask); + } + + return mask; +} + +/* + * The bits of a value are safe if all values that can be built from those + * enabled bits are listed as only affecting performance. + * e.g. 5 would also need 1 and 4 to be listed. + * + * When there is a conflict with the bits already enabled, the new value is + * skipped. + * e.g. if 5 already caused bit-0 and bit-2 to be enabled, adding 3 to the list + * would need to test 7 as bit-2 is already enabled. If 7 is not listed, 3 is + * skipped and bit-1 is not enabled. + */ +static void stage2_test_pbha_value(u8 val) +{ + unsigned long mask; + + mask = decompose_pbha_values(val | arm64_pbha_stage2_safe_bits); + if ((arm64_pbha_perf_only_values & mask) == mask) + arm64_pbha_stage2_safe_bits |= val; +} static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, int scope) { u8 val; + static bool dt_check_done; struct device_node *cpus; const u8 *perf_only_vals; int num_perf_only_vals, i; @@ -2091,6 +2130,10 @@ static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, if (scope == SCOPE_LOCAL_CPU) return true; + spin_lock(&pbha_dt_lock); + if (dt_check_done) + goto out_unlock; + cpus = of_find_node_by_path("/cpus"); if (!cpus) goto done; @@ -2101,7 +2144,7 @@ static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, goto done; /* any listed value is usable at stage 1 */ - for (i = 0 ; i < num_perf_only_vals; i++) { + for (i = 0; i < num_perf_only_vals; i++) { val = perf_only_vals[i]; if (val > 0xf) continue; @@ -2110,9 +2153,24 @@ static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, set_bit(val, &arm64_pbha_perf_only_values); } + /* + * for stage2 the values are collapsed back to 4 bits that can only + * enable values in the arm64_pbha_perf_only_values mask. + */ + for (i = 0; i < num_perf_only_vals; i++) { + val = perf_only_vals[i]; + if (val > 0xf) + continue; + + stage2_test_pbha_value(val); + } + done: of_node_put(cpus); + dt_check_done = true; +out_unlock: + spin_unlock(&pbha_dt_lock); return !!pbha_stage1_enable_bits; } @@ -2132,6 +2190,47 @@ static void cpu_enable_pbha(struct arm64_cpu_capabilities const *cap) isb(); local_flush_tlb_all(); } + +/* + * PBHA's behaviour is implementation defined, as is the way it combines + * stage1 and stage2 attributes. If the kernel has KVM supported, and booted + * at EL2, only these CPUs can allow PBHA in a guest, as KVM knows how the PBHA + * bits are combined. This prevents the host being affected by some + * implementation defined behaviour from the guest. + * + * The TRM for these CPUs describe stage2 as overriding stage1. + */ +static const struct midr_range pbha_stage2_wins[] = { + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A76), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A77), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78), + {}, +}; + +static bool plat_can_use_pbha_stage2(const struct arm64_cpu_capabilities *cap, + int scope) +{ + /* Booted at EL2? */ + if (!is_hyp_mode_available() && !is_kernel_in_hyp_mode()) + return false; + + if (!is_midr_in_range_list(cap->midr_range_list)) + return false; + + /* + * Calls with scope == SCOPE_LOCAL_CPU need only testing whether this + * cpu has the feature. A later 'system' scope call will check for a + * firmware description. + */ + if (scope == SCOPE_LOCAL_CPU) + return true; + + if (!__system_matches_cap(ARM64_HAS_PBHA_STAGE1)) + return false; + + return !!arm64_pbha_stage2_safe_bits; +} #endif /* CONFIG_ARM64_PBHA */ #ifdef CONFIG_ARM64_AMU_EXTN @@ -3273,6 +3372,12 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .cpu_enable = cpu_enable_pbha, ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HPDS, HPDS2) }, + { + .capability = ARM64_HAS_PBHA_STAGE2, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = plat_can_use_pbha_stage2, + .midr_range_list = pbha_stage2_wins, + }, #endif #ifdef CONFIG_ARM64_SME { diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index ef942592d767..8027ee7867d4 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -113,6 +113,8 @@ KVM_NVHE_ALIAS(kvm_protected_mode_initialized); /* Capability of non-cacheable snooping */ KVM_NVHE_ALIAS(kvm_ncsnp_support); +/* PBHA bits for stage2 */ +KVM_NVHE_ALIAS(arm64_pbha_stage2_safe_bits); #endif /* CONFIG_KVM */ diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index c1d66fa7c054..48a010e4fe96 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -8,6 +8,7 @@ */ #include <linux/bitfield.h> +#include <asm/cpufeature.h> #include <asm/kvm_pgtable.h> #include <asm/stage2_pgtable.h> @@ -626,7 +627,7 @@ struct stage2_map_data { u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) { u64 vtcr = VTCR_EL2_FLAGS; - u8 lvls; + u8 lvls, pbha = 0xf; vtcr |= kvm_get_parange(mmfr0) << VTCR_EL2_PS_SHIFT; vtcr |= VTCR_EL2_T0SZ(phys_shift); @@ -661,9 +662,13 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) * value will always be 0, which is defined as the safe default * setting. On Cortex cores, enabling PBHA for stage2 effectively * disables it for stage1. + * When the HAS_PBHA_STAGE2 feature is supported, clear the 'safe' + * bits to allow the guest's stage1 to use these bits. */ + if (cpus_have_final_cap(ARM64_HAS_PBHA_STAGE2)) + pbha = pbha ^ arm64_pbha_stage2_safe_bits; if (cpus_have_final_cap(ARM64_HAS_PBHA)) - vtcr |= FIELD_PREP(VTCR_EL2_PBHA_MASK, 0xf); + vtcr |= FIELD_PREP(VTCR_EL2_PBHA_MASK, pbha); /* Set the vmid bits */ vtcr |= (get_vmid_bits(mmfr1) == 16) ? diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 3564b98915c2..48553be3421c 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -123,7 +123,7 @@ HAS_SYSINSTR128 HAS_TLBID HAS_PBHA HAS_PBHA_STAGE1 -KABI_RESERVE_16 +HAS_PBHA_STAGE2 KABI_RESERVE_17 KABI_RESERVE_18 KABI_RESERVE_19

From: Ma Wupeng <mawupeng1@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 -------------------------------- Now PBHA can be enabled via FDT table at early startup. Here is an example for item in FDT table: cpus { arm,pbha-performance-only = <0x01000000 0x00000000 0x00000000 0x00000000>; }; Signed-off-by: Ma Wupeng <mawupeng1@huawei.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- Changes in v3: - fix the for-loop style in update_pbha_perf_only_bit() --- arch/arm64/include/asm/cpufeature.h | 14 +++++ arch/arm64/kernel/cpufeature.c | 89 +++++++++++++++++++++++++---- arch/arm64/kernel/head.S | 3 + 3 files changed, 94 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index e151c688606b..46c5a0afe32a 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -967,6 +967,20 @@ static inline bool system_supports_bbml3(void) int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); +#ifdef CONFIG_ARM64_PBHA +extern bool pbha_enabled; + +static inline bool system_supports_pbha(void) +{ + return pbha_enabled; +} +#else +static inline bool system_supports_pbha(void) +{ + return false; +} +#endif + static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange) { switch (parange) { diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index b218985a3edc..5d4e259ac390 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -76,6 +76,8 @@ #include <linux/cpu.h> #include <linux/kasan.h> #include <linux/percpu.h> +#include <linux/init.h> +#include <linux/libfdt.h> #include <asm/actlr.h> #include <asm/cpu.h> @@ -99,6 +101,7 @@ #include <asm/vectors.h> #include <asm/virt.h> #include <asm/vip_smt.h> +#include <asm/setup.h> /* Kernel representation of AT_HWCAP and AT_HWCAP2 */ static DECLARE_BITMAP(elf_hwcap, MAX_CPU_FEATURES) __read_mostly; @@ -2076,6 +2079,8 @@ static bool has_hw_dbm(const struct arm64_cpu_capabilities *cap, static u8 pbha_stage1_enable_bits; static DEFINE_SPINLOCK(pbha_dt_lock); +bool pbha_enabled; + /* For the value 5, return a bitmap with bits 5, 4, and 1 set. */ static unsigned long decompose_pbha_values(u8 val) { @@ -2110,11 +2115,26 @@ static void stage2_test_pbha_value(u8 val) arm64_pbha_stage2_safe_bits |= val; } +static void update_pbha_perf_only_bit(const u8 *bits, int cnt) +{ + u8 val; + int i; + + /* any listed value is usable at stage 1 */ + for (i = 0; i < cnt; i++) { + val = bits[i]; + if (val > 0xf) + continue; + + pbha_stage1_enable_bits |= val; + set_bit(val, &arm64_pbha_perf_only_values); + } +} + static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, int scope) { u8 val; - static bool dt_check_done; struct device_node *cpus; const u8 *perf_only_vals; int num_perf_only_vals, i; @@ -2131,7 +2151,7 @@ static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, return true; spin_lock(&pbha_dt_lock); - if (dt_check_done) + if (pbha_enabled) goto out_unlock; cpus = of_find_node_by_path("/cpus"); @@ -2144,14 +2164,7 @@ static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, goto done; /* any listed value is usable at stage 1 */ - for (i = 0; i < num_perf_only_vals; i++) { - val = perf_only_vals[i]; - if (val > 0xf) - continue; - - pbha_stage1_enable_bits |= val; - set_bit(val, &arm64_pbha_perf_only_values); - } + update_pbha_perf_only_bit(perf_only_vals, num_perf_only_vals); /* * for stage2 the values are collapsed back to 4 bits that can only @@ -2167,14 +2180,14 @@ static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, done: of_node_put(cpus); - dt_check_done = true; + pbha_enabled = true; out_unlock: spin_unlock(&pbha_dt_lock); return !!pbha_stage1_enable_bits; } -static void cpu_enable_pbha(struct arm64_cpu_capabilities const *cap) +static void enable_pbha_inner(void) { u64 tcr; @@ -2191,6 +2204,58 @@ static void cpu_enable_pbha(struct arm64_cpu_capabilities const *cap) local_flush_tlb_all(); } +static void cpu_enable_pbha(struct arm64_cpu_capabilities const *cap) +{ + enable_pbha_inner(); +} + +static inline bool cpu_has_pbha(void) +{ + u64 mmfr1 = read_cpuid(ID_AA64MMFR1_EL1); + int val = cpuid_feature_extract_unsigned_field(mmfr1, + ID_AA64MMFR1_EL1_HPDS_SHIFT); + + return val == 2; +} + +void __init early_pbha_init(void) +{ + void *fdt; + int node; + const u8 *prop; + int size; + + fdt = get_early_fdt_ptr(); + if (!fdt) + return; + + /* arm,pbha-performance-only may exist in both chosen and cpus node */ + node = fdt_path_offset(fdt, "/chosen"); + if (node < 0) + return; + + prop = fdt_getprop(fdt, node, "arm,pbha-performance-only", &size); + if (prop) + goto found; + + node = fdt_path_offset(fdt, "/cpus"); + if (node < 0) + return; + + prop = fdt_getprop(fdt, node, "arm,pbha-performance-only", &size); + if (!prop) + return; + +found: + if (!cpu_has_pbha()) + return; + + update_pbha_perf_only_bit(prop, size); + enable_pbha_inner(); + + pbha_enabled = true; +} + /* * PBHA's behaviour is implementation defined, as is the way it combines * stage1 and stage2 attributes. If the kernel has KVM supported, and booted diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 6517bf2644a0..ec972cc4508f 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -512,6 +512,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif mov x0, x21 // pass FDT address in x0 bl early_fdt_map // Try mapping the FDT early +#ifdef CONFIG_ARM64_PBHA + bl early_pbha_init // Init PBHA early via FDT +#endif mov x0, x20 // pass the full boot status bl init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS

From: Ma Wupeng <mawupeng1@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 -------------------------------- Update kernel pte entries if pbha bit0 enabled. This can be used as a hint for kernel page entry for MMU. Signed-off-by: Ma Wupeng <mawupeng1@huawei.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- Changes in v3: - fix a typo in the commit message ("hit" -> "hint") --- arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/mm/fixmap.c | 2 ++ arch/arm64/mm/mmu.c | 13 ++++++++++++- include/linux/pbha.h | 13 ++++++++++++- mm/vmalloc.c | 4 ++++ 5 files changed, 31 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 6b2e95af2971..25b4a70ae8e2 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -158,6 +158,7 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ #define PTE_PBHA_MASK (_AT(pteval_t, 0xf) << 59) /* Page Base Hardware Attributes */ +#define PTE_PBHA0 (_AT(pteval_t, 1) << 59) /* PBHA 59 bit */ #define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index a55b36d04590..9bb565f1cc20 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -10,6 +10,7 @@ #include <linux/memory.h> #include <linux/mm.h> #include <linux/sizes.h> +#include <linux/pbha.h> #include <asm/fixmap.h> #include <asm/kernel-pgtable.h> @@ -124,6 +125,7 @@ void __set_fixmap(enum fixed_addresses idx, ptep = fixmap_pte(addr); if (pgprot_val(flags)) { + flags = pgprot_pbha_bit0(flags); __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { __pte_clear(&init_mm, addr, ptep); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6ffd7305ee80..47da7fb76903 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -27,6 +27,7 @@ #include <linux/kfence.h> #include <linux/pagewalk.h> #include <linux/stop_machine.h> +#include <linux/pbha.h> #include <linux/numa_remote.h> #include "internal.h" @@ -151,6 +152,10 @@ bool pgattr_change_is_safe(u64 old, u64 new) */ pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; +#ifdef CONFIG_ARM64_PBHA + mask |= PTE_PBHA0; +#endif + /* creating or taking down mappings is always safe */ if (!pte_valid(__pte(old)) || !pte_valid(__pte(new))) return true; @@ -395,6 +400,8 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) return; + prot = pgprot_pbha_bit0(prot); + phys &= PAGE_MASK; addr = virt & PAGE_MASK; end = PAGE_ALIGN(virt + size); @@ -1630,7 +1637,11 @@ void __init pmd_mapping_reserved_remap(phys_addr_t start, phys_addr_t end) void __meminit vmemmap_set_pmd(pmd_t *pmdp, void *p, int node, unsigned long addr, unsigned long next) { - pmd_set_huge(pmdp, __pa(p), __pgprot(PROT_SECT_NORMAL)); + pgprot_t prot; + + prot = __pgprot(PROT_SECT_NORMAL); + prot = pgprot_pbha_bit0(prot); + pmd_set_huge(pmdp, __pa(p), prot); } int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, diff --git a/include/linux/pbha.h b/include/linux/pbha.h index 87cfda8041bf..35af1571091d 100644 --- a/include/linux/pbha.h +++ b/include/linux/pbha.h @@ -5,8 +5,10 @@ #ifndef __LINUX_PBHA_H #define __LINUX_PBHA_H -#include <linux/efi.h> #include <linux/libfdt.h> +#include <linux/pgtable.h> + +#define PBHA_VAL_BIT0 1UL #define EFI_OEMCONFIG_VARIABLE_GUID \ EFI_GUID(0x21f3b3c5, 0x946d, 0x41c1, 0x83, 0x8c, 0x19, 0x4e, 0x48, \ @@ -23,8 +25,17 @@ static inline bool system_support_pbha_bit0(void) { return pbha_bit0_enabled; } + +static inline pgprot_t pgprot_pbha_bit0(pgprot_t prot) +{ + if (!system_support_pbha_bit0()) + return prot; + + return pgprot_pbha(prot, PBHA_VAL_BIT0); +} #else static inline bool system_support_pbha_bit0(void) { return false; } +static inline pgprot_t pgprot_pbha_bit0(pgprot_t prot) { return prot; } #endif #endif diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 80f91878d150..455287090795 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -40,6 +40,7 @@ #include <linux/pgtable.h> #include <linux/hugetlb.h> #include <linux/sched/mm.h> +#include <linux/pbha.h> #include <asm/tlbflush.h> #include <asm/shmparam.h> #include <linux/cleanup.h> @@ -340,6 +341,7 @@ int vmap_page_range(unsigned long addr, unsigned long end, { int err; + prot = pgprot_pbha_bit0(prot); err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); @@ -642,6 +644,8 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, WARN_ON(page_shift < PAGE_SHIFT); + prot = pgprot_pbha_bit0(prot); + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || page_shift == PAGE_SHIFT) return vmap_small_pages_range_noflush(addr, end, prot, pages);

From: Ma Wupeng <mawupeng1@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 -------------------------------- Introduce VM_PBHA_BIT0 to enable pbha bit0 for single vma. Signed-off-by: Ma Wupeng <mawupeng1@huawei.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- Changes in v3: - use VM_HIGH_ARCH_5 (bit 37) instead of bit 39 - document the p0 mnemonic in Documentation/filesystems/proc.rst - drop the unused <uapi/asm/mman.h> include - fix the VM_PBHA_BIT0 comment --- Documentation/filesystems/proc.rst | 1 + arch/arm64/Kconfig | 1 + arch/arm64/include/asm/mman.h | 7 +++++++ arch/arm64/include/uapi/asm/mman.h | 1 + arch/arm64/mm/mmap.c | 3 +++ fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 6 ++++++ include/uapi/asm-generic/mman-common.h | 1 + tools/include/uapi/asm-generic/mman-common.h | 1 + 9 files changed, 24 insertions(+) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index a0062574e2ce..17c2c79d70c4 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -572,6 +572,7 @@ encoded manner. The codes are the following: mg mergeable advise flag bt arm64 BTI guarded page mt arm64 MTE allocation tags are enabled + p0 arm64 PBHA bit 0 is enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking ss shadow stack page diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 56f05c78c9d4..18ad070d5b8e 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2199,6 +2199,7 @@ config ARM64_CNP config ARM64_PBHA bool "Enable support for Page Based Hardware Attributes (PBHA)" default n + select ARCH_USES_HIGH_VMA_FLAGS help Page Based Hardware Attributes (PBHA) allow the SoC hardware to change behaviour depending on which mapping was used to access diff --git a/arch/arm64/include/asm/mman.h b/arch/arm64/include/asm/mman.h index 5966ee4a6154..ff95d6c65fb7 100644 --- a/arch/arm64/include/asm/mman.h +++ b/arch/arm64/include/asm/mman.h @@ -5,6 +5,7 @@ #include <linux/compiler.h> #include <linux/types.h> #include <uapi/asm/mman.h> +#include <linux/pbha.h> static inline unsigned long arch_calc_vm_prot_bits(unsigned long prot, unsigned long pkey __always_unused) @@ -17,6 +18,9 @@ static inline unsigned long arch_calc_vm_prot_bits(unsigned long prot, if (system_supports_mte() && (prot & PROT_MTE)) ret |= VM_MTE; + if (system_support_pbha_bit0() && (prot & PROT_PBHA_BIT0)) + ret |= VM_PBHA_BIT0; + return ret; } #define arch_calc_vm_prot_bits(prot, pkey) arch_calc_vm_prot_bits(prot, pkey) @@ -46,6 +50,9 @@ static inline bool arch_validate_prot(unsigned long prot, if (system_supports_mte()) supported |= PROT_MTE; + if (system_support_pbha_bit0()) + supported |= PROT_PBHA_BIT0; + return (prot & ~supported) == 0; } #define arch_validate_prot(prot, addr) arch_validate_prot(prot, addr) diff --git a/arch/arm64/include/uapi/asm/mman.h b/arch/arm64/include/uapi/asm/mman.h index 1e6482a838e1..07e6f340a2f3 100644 --- a/arch/arm64/include/uapi/asm/mman.h +++ b/arch/arm64/include/uapi/asm/mman.h @@ -6,5 +6,6 @@ #define PROT_BTI 0x10 /* BTI guarded page */ #define PROT_MTE 0x20 /* Normal Tagged mapping */ +#define PROT_PBHA_BIT0 0x1000 /* PBHA 59 bit */ #endif /* ! _UAPI__ASM_MMAN_H */ diff --git a/arch/arm64/mm/mmap.c b/arch/arm64/mm/mmap.c index 8f5b7ce857ed..75d099699c8c 100644 --- a/arch/arm64/mm/mmap.c +++ b/arch/arm64/mm/mmap.c @@ -98,6 +98,9 @@ pgprot_t vm_get_page_prot(unsigned long vm_flags) if (vm_flags & VM_MTE) prot |= PTE_ATTRINDX(MT_NORMAL_TAGGED); + if (vm_flags & VM_PBHA_BIT0) + prot |= PTE_PBHA0; + return __pgprot(prot); } EXPORT_SYMBOL(vm_get_page_prot); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 35c6a736e11c..09bacd826511 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -684,6 +684,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_MTE)] = "mt", [ilog2(VM_MTE_ALLOWED)] = "", #endif +#ifdef CONFIG_ARM64_PBHA + [ilog2(VM_PBHA_BIT0)] = "p0", +#endif #ifdef CONFIG_ARCH_HAS_PKEYS /* These come out via ProtectionKey: */ [ilog2(VM_PKEY_BIT0)] = "", diff --git a/include/linux/mm.h b/include/linux/mm.h index dfe362127354..c94c8f43f8a0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -411,6 +411,12 @@ extern unsigned int kobjsize(const void *objp); # define VM_SHARE_POOL VM_NONE #endif +#if defined(CONFIG_ARM64_PBHA) +# define VM_PBHA_BIT0 VM_HIGH_ARCH_5 /* Page Based Hardware Attributes bit 0 */ +#else +# define VM_PBHA_BIT0 VM_NONE +#endif + #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE #endif diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h index 559ad7c50b60..25c5170d6c26 100644 --- a/include/uapi/asm-generic/mman-common.h +++ b/include/uapi/asm-generic/mman-common.h @@ -13,6 +13,7 @@ #define PROT_SEM 0x8 /* page may be used for atomic ops */ /* 0x10 reserved for arch-specific use */ /* 0x20 reserved for arch-specific use */ +/* 0x1000 reserved for arch-specific use */ #define PROT_NONE 0x0 /* page can not be accessed */ #define PROT_GROWSDOWN 0x01000000 /* mprotect flag: extend change to start of growsdown vma */ #define PROT_GROWSUP 0x02000000 /* mprotect flag: extend change to end of growsup vma */ diff --git a/tools/include/uapi/asm-generic/mman-common.h b/tools/include/uapi/asm-generic/mman-common.h index 6ce1f1ceb432..e397d1224747 100644 --- a/tools/include/uapi/asm-generic/mman-common.h +++ b/tools/include/uapi/asm-generic/mman-common.h @@ -13,6 +13,7 @@ #define PROT_SEM 0x8 /* page may be used for atomic ops */ /* 0x10 reserved for arch-specific use */ /* 0x20 reserved for arch-specific use */ +/* 0x1000 reserved for arch-specific use */ #define PROT_NONE 0x0 /* page can not be accessed */ #define PROT_GROWSDOWN 0x01000000 /* mprotect flag: extend change to start of growsdown vma */ #define PROT_GROWSUP 0x02000000 /* mprotect flag: extend change to end of growsup vma */

From: Fanhua Li <lifanhua5@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 -------------------------------- Introduce /proc/<pid>/pbha_bit0 to update PBHA bit0. Value 0/1 can be accepted by this procfs: - 0: iter all vmas of this task, clear VM_PBHA_BIT0 for all vmas. New vma will stop bring VM_PBHA_BIT0 if exists. clear PTE 59 bit for all pte entries. - 1: iter all vmas of this task, set VM_PBHA_BIT0 for all vmas. New vma will bring VM_PBHA_BIT0 by default. set PTE 59 bit for all pte entries. Signed-off-by: Ma Wupeng <mawupeng1@huawei.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- Changes in v3: - use pmdp_invalidate_ad() when rewriting a live THP pmd - use get_task_mm()/mmput() in pbha_bit0_read() - add a .hugetlb_entry callback so hugetlb mappings are updated too - drop the mm->def_flags early return and track whether anything changed --- Documentation/admin-guide/mm/pagemap.rst | 4 +- arch/arm64/include/asm/pgtable.h | 9 ++ drivers/soc/hisilicon/pbha.c | 197 +++++++++++++++++++++++ fs/proc/base.c | 124 ++++++++++++++ fs/proc/task_mmu.c | 15 ++ include/linux/mm.h | 2 +- include/linux/pbha.h | 27 ++++ 7 files changed, 376 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index c8f380271cad..f81a4045d0e6 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -21,7 +21,9 @@ There are four components to pagemap: * Bit 56 page exclusively mapped (since 4.2) * Bit 57 pte is uffd-wp write-protected (since 5.13) (see Documentation/admin-guide/mm/userfaultfd.rst) - * Bits 58-60 zero + * Bits 58 zero + * Bit 59 PBHA bit0, used only CONFIG_ARM64_PBHA is enabled + * Bits 60 zero * Bit 61 page is file-page or shared-anon (since 3.5) * Bit 62 page swapped * Bit 63 page present diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 4514cb0ea736..6326bba0feb4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -654,6 +654,15 @@ static inline pte_t pte_mkpbha(pte_t pte, unsigned long pbha_val) return set_pte_bit(pte, __pgprot(__pbha_check_perf_only(pbha_val))); } +#define pmd_mkpbha(pmd, pbha_val) pte_pmd(pte_mkpbha(pmd_pte(pmd), pbha_val)) + +static inline pte_t pte_rmpbha(pte_t pte, unsigned long pbha_val) +{ + return clear_pte_bit(pte, __pgprot(__pbha_check_perf_only(pbha_val))); +} + +#define pmd_rmpbha(pmd, pbha_val) pte_pmd(pte_rmpbha(pmd_pte(pmd), pbha_val)) + /* * DMA allocations for non-coherent devices use what the Arm architecture calls * "Normal non-cacheable" memory, which permits speculation, unaligned accesses diff --git a/drivers/soc/hisilicon/pbha.c b/drivers/soc/hisilicon/pbha.c index 7e80dcf77607..5de5dbb89b6d 100644 --- a/drivers/soc/hisilicon/pbha.c +++ b/drivers/soc/hisilicon/pbha.c @@ -9,8 +9,15 @@ #include <linux/libfdt.h> #include <linux/printk.h> #include <linux/cpufeature.h> +#include <linux/mmu_notifier.h> +#include <linux/hugetlb.h> +#include <linux/pagewalk.h> +#include <linux/pbha.h> +#include <linux/mm_inline.h> #include <linux/cache.h> + #include <asm/setup.h> +#include <linux/processor.h> #define MOC_MODE_CACHE 1 @@ -40,3 +47,193 @@ void __init early_pbha_bit0_init(void) if (*prop == MOC_MODE_CACHE) pbha_bit0_enabled = true; } + +#define pte_pbha_bit0(pte) \ + (!!(pte_val(pte) & (PBHA_VAL_BIT0 << PBHA_BITS_SHIFT))) + +enum { + CLEAR_PBHA_BIT0_FLAG, + SET_PBHA_BIT0_FLAG, +}; + +static inline void pbha_bit0_update_pte_bits(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte, bool set) +{ + pte_t ptent = *pte; + + if (pte_present(ptent)) { + pte_t old_pte; + + old_pte = ptep_modify_prot_start(vma, addr, pte); + if (set) + ptent = pte_mkpbha(old_pte, PBHA_VAL_BIT0); + else + ptent = pte_rmpbha(old_pte, PBHA_VAL_BIT0); + ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); + } +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline void pbha_bit0_update_pmd_bits(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp, bool set) +{ + if (pmd_present(*pmdp)) { + pmd_t pmd = pmdp_invalidate_ad(vma, addr, pmdp); + + if (set) + pmd = pmd_mkpbha(pmd, PBHA_VAL_BIT0); + else + pmd = pmd_rmpbha(pmd, PBHA_VAL_BIT0); + + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } +} +#else +static inline void pbha_bit0_update_pmd_bits(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp, + bool set) +{ +} +#endif + +static int pbha_bit0_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) +{ + int *op = (int *)walk->private; + struct vm_area_struct *vma = walk->vma; + pte_t *pte; + spinlock_t *ptl; + bool set = (*op == SET_PBHA_BIT0_FLAG); + + ptl = pmd_trans_huge_lock(pmd, vma); + if (ptl) { + pbha_bit0_update_pmd_bits(vma, addr, pmd, set); + + spin_unlock(ptl); + return 0; + } + + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } + for (; addr != end; pte++, addr += PAGE_SIZE) + pbha_bit0_update_pte_bits(vma, addr, pte, set); + pte_unmap_unlock(pte - 1, ptl); + cond_resched(); + return 0; +} + +static int pbha_bit0_test_walk(unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + struct vm_area_struct *vma = walk->vma; + + if (vma->vm_flags & VM_PFNMAP) + return 1; + + return 0; +} + +#ifdef CONFIG_HUGETLB_PAGE +static int pbha_bit0_hugetlb_entry(pte_t *ptep, unsigned long hmask, + unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + struct vm_area_struct *vma = walk->vma; + struct hstate *h = hstate_vma(vma); + unsigned long sz = huge_page_size(h); + bool set = (*(int *)walk->private == SET_PBHA_BIT0_FLAG); + struct mm_struct *mm = walk->mm; + spinlock_t *ptl; + pte_t pte; + + /* + * Page tables shared between mms are visible to every sharer, leave + * them alone instead of changing the hint behind their back. + */ + if (hugetlb_pmd_shared(ptep)) + return 0; + + ptl = huge_pte_lock(h, mm, ptep); + pte = huge_ptep_get(ptep); + if (pte_present(pte)) { + if (set) + pte = pte_mkpbha(pte, PBHA_VAL_BIT0); + else + pte = pte_rmpbha(pte, PBHA_VAL_BIT0); + + set_huge_pte_at(mm, addr & hmask, ptep, pte, sz); + } + spin_unlock(ptl); + + return 0; +} +#endif + +struct mm_walk_ops pbha_bit0_walk_ops = { + .pmd_entry = pbha_bit0_pte_range, +#ifdef CONFIG_HUGETLB_PAGE + .hugetlb_entry = pbha_bit0_hugetlb_entry, +#endif + .test_walk = pbha_bit0_test_walk, +}; + +int pbha_bit0_update_vma(struct mm_struct *mm, int val) +{ + struct mmu_notifier_range range; + struct vm_area_struct *vma; + struct vma_iterator vmi; + bool need_update = false; + + if (!system_support_pbha_bit0()) + return -EINVAL; + + if (mmap_write_lock_killable(mm)) + return -EINTR; + + if (val == SET_PBHA_BIT0_FLAG) { + if (!(mm->def_flags & VM_PBHA_BIT0)) { + mm->def_flags |= VM_PBHA_BIT0; + need_update = true; + } + vma_iter_init(&vmi, mm, 0); + for_each_vma(vmi, vma) { + if (vma->vm_flags & VM_PBHA_BIT0) + continue; + vm_flags_set(vma, VM_PBHA_BIT0); + vma_set_page_prot(vma); + need_update = true; + } + } else { + if (mm->def_flags & VM_PBHA_BIT0) { + mm->def_flags &= ~VM_PBHA_BIT0; + need_update = true; + } + vma_iter_init(&vmi, mm, 0); + for_each_vma(vmi, vma) { + if (!(vma->vm_flags & VM_PBHA_BIT0)) + continue; + vm_flags_clear(vma, VM_PBHA_BIT0); + vma_set_page_prot(vma); + need_update = true; + } + } + + if (!need_update) { + mmap_write_unlock(mm); + return 0; + } + + inc_tlb_flush_pending(mm); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, 0, -1UL); + mmu_notifier_invalidate_range_start(&range); + walk_page_range(mm, 0, -1, &pbha_bit0_walk_ops, &val); + mmu_notifier_invalidate_range_end(&range); + flush_tlb_mm(mm); + dec_tlb_flush_pending(mm); + + mmap_write_unlock(mm); + return 0; +} diff --git a/fs/proc/base.c b/fs/proc/base.c index f34e357e9f97..239f2f3f95ec 100644 --- a/fs/proc/base.c +++ b/fs/proc/base.c @@ -102,6 +102,7 @@ #include <linux/resctrl.h> #include <linux/cn_proc.h> #include <linux/ksm.h> +#include <linux/pbha.h> #include <trace/events/oom.h> #include "internal.h" #include "fd.h" @@ -1290,6 +1291,120 @@ static const struct file_operations proc_oom_score_adj_operations = { .llseek = default_llseek, }; +#ifdef CONFIG_ARM64_PBHA +static inline int pbha_bit0_check(struct task_struct *task, struct pid *pid) +{ + if (!system_support_pbha_bit0()) + return -EACCES; + + if (is_global_init(task)) + return -EACCES; + + if (!task->mm || (task->flags & PF_KTHREAD) || + (task->flags & PF_EXITING)) + return -EACCES; + + return 0; +} + +static ssize_t pbha_bit0_read(struct file *file, char __user *buf, + size_t count, loff_t *ppos) +{ + struct task_struct *task = get_proc_task(file_inode(file)); + struct pid *pid = proc_pid(file_inode(file)); + char buffer[PROC_NUMBUF]; + struct mm_struct *mm; + size_t len; + short val; + int err; + + if (!task) + return -ESRCH; + + err = pbha_bit0_check(task, pid); + if (err) { + put_task_struct(task); + return err; + } + + mm = get_task_mm(task); + if (!mm) { + put_task_struct(task); + return -ESRCH; + } + + val = mm->def_flags & VM_PBHA_BIT0 ? 1 : 0; + mmput(mm); + put_task_struct(task); + len = snprintf(buffer, sizeof(buffer), "%hd\n", val); + return simple_read_from_buffer(buf, count, ppos, buffer, len); +} + +static ssize_t pbha_bit0_write(struct file *file, const char __user *buf, + size_t count, loff_t *ppos) +{ + struct task_struct *task = get_proc_task(file_inode(file)); + struct pid *pid = proc_pid(file_inode(file)); + char buffer[PROC_NUMBUF]; + struct mm_struct *mm; + int val, err; + + if (!task) + return -ESRCH; + + err = pbha_bit0_check(task, pid); + if (err) + goto out; + + memset(buffer, 0, sizeof(buffer)); + if (count > sizeof(buffer) - 1) + count = sizeof(buffer) - 1; + if (copy_from_user(buffer, buf, count)) { + err = -EFAULT; + goto out; + } + + err = kstrtoint(strstrip(buffer), 0, &val); + if (err) + goto out; + if (val != 0 && val != 1) { + err = -EINVAL; + goto out; + } + + mm = get_task_mm(task); + if (!mm) { + err = -ENOENT; + goto out; + } + + err = pbha_bit0_update_vma(mm, val); + if (err) + count = -EINTR; + + mmput(mm); +out: + put_task_struct(task); + return err < 0 ? err : count; +} + +static const struct file_operations proc_pbha_bit0_ops = { + .read = pbha_bit0_read, + .write = pbha_bit0_write, + .llseek = generic_file_llseek, +}; + +bool pbha_bit0_hide_file(const char *name) +{ + if (!system_support_pbha_bit0() && !strncmp("pbha_bit0", name, 9)) + return true; + + return false; +} +#else +static bool pbha_bit0_hide_file(const char *name) { return false; } +#endif + #ifdef CONFIG_AUDIT #define TMPBUFLEN 11 static ssize_t proc_loginuid_read(struct file * file, char __user * buf, @@ -2701,6 +2816,9 @@ static struct dentry *proc_pident_instantiate(struct dentry *dentry, static bool proc_hide_pidents(const struct pid_entry *p) { + if (pbha_bit0_hide_file(p->name)) + return true; + if (mem_reliable_hide_file(p->name)) return true; @@ -3594,6 +3712,9 @@ static const struct pid_entry tgid_base_stuff[] = { #ifdef CONFIG_MEMORY_RELIABLE REG("reliable", S_IRUGO|S_IWUSR, proc_reliable_operations), #endif +#ifdef CONFIG_ARM64_PBHA + REG("pbha_bit0", 0644, proc_pbha_bit0_ops), +#endif #ifdef CONFIG_AUDIT REG("loginuid", S_IWUSR|S_IRUGO, proc_loginuid_operations), REG("sessionid", S_IRUGO, proc_sessionid_operations), @@ -4017,6 +4138,9 @@ static const struct pid_entry tid_base_stuff[] = { #ifdef CONFIG_MEMORY_RELIABLE REG("reliable", S_IRUGO|S_IWUSR, proc_reliable_operations), #endif +#ifdef CONFIG_ARM64_PBHA + REG("pbha_bit0", 0644, proc_pbha_bit0_ops), +#endif #ifdef CONFIG_AUDIT REG("loginuid", S_IWUSR|S_IRUGO, proc_loginuid_operations), REG("sessionid", S_IRUGO, proc_sessionid_operations), diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 09bacd826511..fecf97c3507a 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -20,6 +20,7 @@ #include <linux/shmem_fs.h> #include <linux/uaccess.h> #include <linux/pkeys.h> +#include <linux/pbha.h> #include <asm/elf.h> #include <asm/tlb.h> @@ -1362,6 +1363,11 @@ struct pagemapread { #define PM_SOFT_DIRTY BIT_ULL(55) #define PM_MMAP_EXCLUSIVE BIT_ULL(56) #define PM_UFFD_WP BIT_ULL(57) +#ifdef CONFIG_ARM64_PBHA +#define PM_PBHA_BIT0 BIT_ULL(59) +#else +#define PM_PBHA_BIT0 0 +#endif #define PM_FILE BIT_ULL(61) #define PM_SWAP BIT_ULL(62) #define PM_PRESENT BIT_ULL(63) @@ -1463,6 +1469,9 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, flags |= PM_UFFD_WP; } + if (pte_pbha(pte)) + flags |= PM_PBHA_BIT0; + if (page && !PageAnon(page)) flags |= PM_FILE; if (page && (flags & PM_PRESENT) && page_mapcount(page) == 1) @@ -1527,6 +1536,9 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, } #endif + if (pmd_pbha(pmd)) + flags |= PM_PBHA_BIT0; + if (page && !PageAnon(page)) flags |= PM_FILE; @@ -1606,6 +1618,9 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, if (huge_pte_uffd_wp(pte)) flags |= PM_UFFD_WP; + if (pte_pbha(pte)) + flags |= PM_PBHA_BIT0; + flags |= PM_PRESENT; if (pm->show_pfn) frame = pte_pfn(pte) + diff --git a/include/linux/mm.h b/include/linux/mm.h index c94c8f43f8a0..765319d3a71c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -488,7 +488,7 @@ extern unsigned int kobjsize(const void *objp); #define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB) /* This mask defines which mm->def_flags a process can inherit its parent */ -#define VM_INIT_DEF_MASK VM_NOHUGEPAGE +#define VM_INIT_DEF_MASK (VM_NOHUGEPAGE | VM_PBHA_BIT0) /* This mask represents all the VMA flag bits used by mlock */ #define VM_LOCKED_MASK (VM_LOCKED | VM_LOCKONFAULT) diff --git a/include/linux/pbha.h b/include/linux/pbha.h index ae4f29331ac5..9d004c833745 100644 --- a/include/linux/pbha.h +++ b/include/linux/pbha.h @@ -10,6 +10,7 @@ #include <linux/pgtable.h> #define PBHA_VAL_BIT0 1UL +#define PBHA_BITS_SHIFT 59 #define EFI_OEMCONFIG_VARIABLE_GUID \ EFI_GUID(0x21f3b3c5, 0x946d, 0x41c1, 0x83, 0x8c, 0x19, 0x4e, 0x48, \ @@ -20,7 +21,9 @@ #ifdef CONFIG_ARM64_PBHA extern bool __ro_after_init pbha_bit0_enabled; +extern struct mm_walk_ops pbha_bit0_walk_ops; extern void __init early_pbha_bit0_init(void); +extern int pbha_bit0_update_vma(struct mm_struct *mm, int val); static inline bool system_support_pbha_bit0(void) { @@ -45,6 +48,22 @@ static inline pte_t maybe_mk_pbha_bit0(pte_t pte, struct vm_area_struct *vma) return pte; } + +static inline bool pte_pbha(pte_t pte) +{ + if (!system_support_pbha_bit0()) + return false; + + return !!(pte_val(pte) & (PBHA_VAL_BIT0 << PBHA_BITS_SHIFT)); +} + +static inline bool pmd_pbha(pmd_t pmd) +{ + if (!system_support_pbha_bit0()) + return false; + + return !!(pmd_val(pmd) & (PBHA_VAL_BIT0 << PBHA_BITS_SHIFT)); +} #else static inline bool system_support_pbha_bit0(void) { return false; } static inline pgprot_t pgprot_pbha_bit0(pgprot_t prot) { return prot; } @@ -52,6 +71,14 @@ static inline pte_t maybe_mk_pbha_bit0(pte_t pte, struct vm_area_struct *vma) { return pte; } +static inline bool pte_pbha(pte_t pte) +{ + return false; +} +static inline bool pmd_pbha(pmd_t pte) +{ + return false; +} #endif #endif

From: Ma Wupeng <mawupeng1@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 -------------------------------- Introduce prctl to control pbha behavior. Signed-off-by: Ma Wupeng <mawupeng1@huawei.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- include/uapi/linux/prctl.h | 1 + kernel/sys.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 370ed14b1ae0..02229dfc21d0 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -306,4 +306,5 @@ struct prctl_mm_map { # define PR_RISCV_V_VSTATE_CTRL_NEXT_MASK 0xc # define PR_RISCV_V_VSTATE_CTRL_MASK 0x1f +#define PR_UPDATE_PBHA_BIT0 0x82312f0 #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/sys.c b/kernel/sys.c index 7465bb7c8919..9ef86e9faa35 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2767,6 +2767,16 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, case PR_RISCV_V_GET_CONTROL: error = RISCV_V_GET_CONTROL(); break; +#ifdef CONFIG_ARM64_PBHA + case PR_UPDATE_PBHA_BIT0: + if (arg3 || arg4 || arg5) + return -EINVAL; + if (arg2 != 0 && arg2 != 1) + return -EINVAL; + + error = pbha_bit0_update_vma(me->mm, arg2); + break; +#endif default: error = -EINVAL; break;

From: Ma Wupeng <mawupeng1@huawei.com> hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/9826 -------------------------------- Enable feature PBHA for arm64 by default. Signed-off-by: Ma Wupeng <mawupeng1@huawei.com> Signed-off-by: Fanhua Li <lifanhua5@huawei.com> Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com> --- arch/arm64/configs/openeuler_defconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index 007158aed18e..7a93e398be06 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -560,6 +560,7 @@ CONFIG_ARM64_USE_LSE_ATOMICS=y CONFIG_ARM64_PMEM=y CONFIG_ARM64_RAS_EXTN=y CONFIG_ARM64_CNP=y +CONFIG_ARM64_PBHA=y # end of ARMv8.2 architectural features #