This series adds KVM virtualization for TLBID so that a guest can use the feature. KVM maintains a vDomain-to-pDomain mapping and writes it into VTLBID(n) on vCPU load; the hardware then translates the guest's vDomain id into the corresponding pDomain id for TLBI broadcast. The series covers: - TLBID-related sysreg definitions (TLBIDIDR_EL1, VTLBIDn_EL2, HCRX_EL2). - KVM internal helpers for TLBID virtualization init/teardown and vDomain configuration. - VTLBID(n) map update on vCPU load/migration, with ioctls for QEMU to query max vDomains and set per-vCPU vDomain bitmaps. - Guest exposure of TLBIDIDR_EL1 and ID_AA64MMFR4_EL1.TLBID, with trap handling. - Optimizations: pCPU-to-vCPU reverse index to avoid vCPU traversal, skipping TLB flush/IPI when the domain map is unchanged, and a tracepoint to replace trace_printk. Jinqian Yang (10): KVM: arm64: Add internal helpers to support TLBID virtualization KVM: arm64: Update VTLBID(n) when vCPU load KVM: arm64: Add ioctls to support TLBID virtualzation KVM: arm64: Ensure DVMBM is disabled when TLBID is enabled KVM: arm64: Expose tlbididr_el1 to guest KVM: arm64: Expose ID_AA64MMFR4_EL1_TLBID to guest KVM: arm64: Enable read TLBIDIDR_EL1 trap in guest KVM: arm64: Use pCPU-to-vCPU reverse index to avoid vCPU traversal KVM: arm64: Skip TLB invalidation and vCPU IPI when domain map is unchanged KVM: arm64: Replace trace_printk with tracepoint in kvm_tlbidomain_vcpu_load Tian Zheng (1): KVM: arm64: Skip VM-level ID_AA64MMFR4 write on vCPU hotplug Zhou Wang (1): arm64/sysreg: Add TLBID sysreg Documentation/virt/kvm/api.rst | 62 ++++ arch/arm64/include/asm/kvm_host.h | 45 +++ arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kvm/arm.c | 453 ++++++++++++++++++++++++++- arch/arm64/kvm/hisilicon/hisi_virt.c | 9 + arch/arm64/kvm/sys_regs.c | 67 +++- arch/arm64/kvm/sys_regs.h | 2 + arch/arm64/kvm/trace_arm.h | 47 +++ arch/arm64/tools/sysreg | 24 +- include/uapi/linux/kvm.h | 14 + 10 files changed, 721 insertions(+), 4 deletions(-) -- 2.33.0
From: Zhou Wang <wangzhou1@hisilicon.com> Add the information about TLBID-related registers. Signed-off-by: Zhou Wang <wangzhou1@hisilicon.com> Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/tools/sysreg | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 9b4dcf1e9919..99e615204a78 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -3177,7 +3177,9 @@ Fields ZCR_ELx EndSysreg Sysreg HCRX_EL2 3 4 1 2 2 -Res0 63:23 +Res0 63:26 +Field 25 VTLBIDEn +Res0 24:23 Field 22 GCSEn Field 21 EnIDCP128 Field 20 EnSDERR @@ -3583,3 +3585,19 @@ Field 12:8 NVIS Res0 7:5 Field 4:0 NIS EndSysreg + +Sysreg VTLBID0_EL2 3 4 2 8 0 +Field 63:0 TD +EndSysreg + +Sysreg VTLBID1_EL2 3 4 2 8 1 +Field 63:0 TD +EndSysreg + +Sysreg VTLBID2_EL2 3 4 2 8 2 +Field 63:0 TD +EndSysreg + +Sysreg VTLBID3_EL2 3 4 2 8 3 +Field 63:0 TD +EndSysreg -- 2.33.0
Add support functions for TLBID virtualization initialization and ioctls. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Zhou Wang <wangzhou1@hisilicon.com> --- Documentation/virt/kvm/api.rst | 62 +++++++++++++ arch/arm64/include/asm/kvm_host.h | 27 ++++++ arch/arm64/kvm/arm.c | 149 ++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 14 +++ 4 files changed, 252 insertions(+) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 77c65f16e2a7..911ebf02ea16 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6424,6 +6424,52 @@ the capability to be present. `flags` must currently be zero. +4.144 KVM_ARM_GET_VDOMAIN_NUM +------------------------------ + +:Capability: KVM_CAP_ARM_TLBIDOMAIN +:Architectures: arm64 +:Type: vm ioctl +:Parameters: struct kvm_arm_get_vdomain (out) +:Returns: 0 on success, < 0 on error + +Parameters are specified via the following structure:: + +:: + + struct kvm_arm_get_vdomain { + __u8 max_vdomains; + }; + +The ``max_vdomains`` field is the max number of vDomains supported by the +hardware. This value is determined by TLBIDIDR_EL1[NVOS/NVIS]. NVOS/NVIS +is the bit width of the vdomain. + +4.145 KVM_ARM_VCPU_SET_VDOMAIN +------------------------------ + +:Capability: KVM_CAP_ARM_TLBIDOMAIN +:Architectures: arm64 +:Type: vcpu ioctl +:Parameters: struct kvm_arm_set_vdomain (in) +:Returns: 0 on success, < 0 on error + +This ioctl allows KVM to obtain TLBI Domain of Guest. + +Parameters are specified via the following structure:: + +:: + + struct kvm_arm_set_vdomain { + __u32 vdomain_bitmap; + __u8 num_vdomains; + }; + +The ``vdomain_bitmap`` field is a bitmap, where '1' indicates that vDomain +includes this vCPU. For example, ``vCPU0: vdomain_bitmap=0b1011`` indicates +that vCPU0 included in vDomain(0, 1, 3). + +The ``num_vdomains`` field is the number of vDomains. 5. The kvm_run structure ======================== @@ -8218,6 +8264,22 @@ Used to configure and set up the memory for a Realm. The available actions are: enter the realm until it has been activated. ================================= ============================================= +7.39 KVM_CAP_ARM_TLBIDOMAIN +------------------------------------- + +:Architectures: arm64 +:Target: VM +:Parameters: None +:Returns: 0 on success, negative value on error + +This capability enables TLBID virtualization, which Optimizes tlbi broadcast +range to avoid unnecessary broadcasts. + +When this capability is enabled, KVM can obtain the vDomain bitmaps of VM. When +a vCPU is loaded, KVM writes the mapping between the vDomain and the pDomain +into the VTLBID(n). If guest enabled TLBID, hardware will convert vDomain id to +pDomain id for TLBI broadcast. + 8. Other capabilities. ====================== diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index bdd2168b6d6a..25c6a5b222b1 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -208,6 +208,32 @@ struct kvm_protected_vm { struct kvm_hyp_memcache teardown_mc; }; +struct vcpu_tlbid_data { + /* Store vDomain bitmap, indexed by vCPU ID */ + u32 vdomain_bitmap; + /* Record the last pCPU where vCPU was running, indexed by vCPU ID */ + int last_pcpu; +}; + +struct tlbidomain { + /* Using xarray to dynamically allocate vCPU-related TLBID info */ + struct xarray vcpu_data_array; + /* The number of vDomains */ + int num_domains; + /* Mapping of pDomain to vDomain, indexed by vDomain ID */ + int *domain_map; + /* Record the bitmap of pCPUs mapped to vDomain, indexed by vDomain ID */ + cpumask_var_t *vdomain_cpumasks; + /* Whether virt-TLBID is enabled by KVM */ + bool kvm_tlbid_enabled; + /* Whether virt-TLBID is enabled by Guest */ + bool guest_tlbid_enabled; + /* TLBIDIDR.NVIS */ + u8 nvis; + /* TLBIDIDR.NIS */ + u8 nis; +}; + struct kvm_arch { struct kvm_s2_mmu mmu; @@ -332,6 +358,7 @@ struct kvm_arch { KABI_EXTEND(u64 midr_el1) KABI_EXTEND(u64 revidr_el1) KABI_EXTEND(u64 aidr_el1) + KABI_EXTEND(struct tlbidomain vdomain); }; struct kvm_vcpu_fault_info { diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 324553fea612..1abcb2b2e7e2 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -319,6 +319,82 @@ static int kvm_arm_default_max_vcpus(void) return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS; } +static int kvm_arm_init_tlbidomain(struct kvm *kvm) +{ + struct tlbidomain *vdomain; + int i, max_vdomains; + u64 tlbididr; + + if (!system_supports_tlbid()) + return 0; + + tlbididr = read_sysreg_s(SYS_TLBIDIDR_EL1); + vdomain = &kvm->arch.vdomain; + vdomain->nis = FIELD_GET(TLBIDIDR_EL1_NIS_MASK, tlbididr); + vdomain->nvis = FIELD_GET(TLBIDIDR_EL1_NVIS_MASK, tlbididr); + vdomain->num_domains = -1; + vdomain->kvm_tlbid_enabled = true; + max_vdomains = 1 << vdomain->nvis; + + kvm_info("TLBIDIDR: 0x%llx, NIS: %d, NVIS: %d\n", tlbididr, vdomain->nis, vdomain->nvis); + + xa_init(&vdomain->vcpu_data_array); + + vdomain->domain_map = kzalloc(sizeof(*vdomain->domain_map) * max_vdomains, + GFP_KERNEL); + if (!vdomain->domain_map) + goto destroy_vcpu_data_array; + + vdomain->vdomain_cpumasks = kcalloc(max_vdomains, sizeof(cpumask_var_t), + GFP_KERNEL); + if (!vdomain->vdomain_cpumasks) + goto free_domain_map; + + for (i = 0; i < max_vdomains; i++) { + if (!alloc_cpumask_var(&vdomain->vdomain_cpumasks[i], GFP_KERNEL)) + goto free_vdomain_cpumasks; + cpumask_clear(vdomain->vdomain_cpumasks[i]); + } + + for (i = 0; i < max_vdomains; i++) + kvm->arch.vdomain.domain_map[i] = -1; + + return 0; + +free_vdomain_cpumasks: + while (--i >= 0) + free_cpumask_var(vdomain->vdomain_cpumasks[i]); + kfree(vdomain->vdomain_cpumasks); +free_domain_map: + kfree(vdomain->domain_map); +destroy_vcpu_data_array: + xa_destroy(&vdomain->vcpu_data_array); + + return -ENOMEM; +} + +static void free_tlbid_vdomain(struct kvm *kvm) +{ + struct vcpu_tlbid_data *data; + int i, max_vdomains; + unsigned long index; + + max_vdomains = 1 << kvm->arch.vdomain.nvis; + + if (system_supports_tlbid()) { + kfree(kvm->arch.vdomain.domain_map); + + for (i = 0; i < max_vdomains; i++) + free_cpumask_var(kvm->arch.vdomain.vdomain_cpumasks[i]); + kfree(kvm->arch.vdomain.vdomain_cpumasks); + + xa_for_each(&kvm->arch.vdomain.vcpu_data_array, index, data) { + kfree(data); + } + xa_destroy(&kvm->arch.vdomain.vcpu_data_array); + } +} + /** * kvm_arch_init_vm - initializes a VM data structure * @kvm: pointer to the KVM struct @@ -1765,6 +1841,40 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, return kvm_reset_vcpu(vcpu); } +static int kvm_arm_tlbidomain_vcpu_init(struct kvm_vcpu *vcpu) +{ + struct vcpu_tlbid_data *data; + int ret; + + if (!system_supports_tlbid()) + return 0; + + data = kzalloc(sizeof(*data), GFP_KERNEL); + if (!data) + return -ENOMEM; + + /* + * When the host has enabled TLBID but the guest has not, + * KVM can map vDomain0 to pDomainN to optimize the TLBI + * instructions for the guest. Therefore, the value is set + * to 0x1, indicating that the vCPU is in vDomain0. If QEMU + * calls KVM_ARM_VCPU_SET_VDOMAIN to configure vdomains, this + * value will be overwritten to support the guest enabling + * TLBID. + */ + data->vdomain_bitmap = 0x1; + data->last_pcpu = -1; + + ret = xa_insert(&vcpu->kvm->arch.vdomain.vcpu_data_array, + vcpu->vcpu_idx, data, GFP_KERNEL); + if (ret) { + kfree(data); + return ret == -EBUSY ? 0 : ret; + } + + return 0; +} + static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_vcpu_init *init) { @@ -1912,6 +2022,37 @@ static int kvm_arm_vcpu_rmm_psci_complete(struct kvm_vcpu *vcpu, return realm_psci_complete(vcpu, target, arg->psci_status); } +static int kvm_arm_vcpu_set_vdomain(struct kvm_vcpu *vcpu, + struct kvm_arm_set_vdomain *vdomain) +{ + u32 vdomain_bitmap = vdomain->vdomain_bitmap; + u8 num_vdomains = vdomain->num_vdomains; + u8 nvis = vcpu->kvm->arch.vdomain.nvis; + u8 max_vdomains = 1 << nvis; + struct vcpu_tlbid_data *data; + + if (num_vdomains > max_vdomains) + return -EINVAL; + + if (vcpu->kvm->arch.vdomain.num_domains != -1 && + vcpu->kvm->arch.vdomain.num_domains != num_vdomains) + return -EINVAL; + + data = xa_load(&vcpu->kvm->arch.vdomain.vcpu_data_array, vcpu->vcpu_idx); + if (!data) { + kvm_err("vCPU%d: Failed to load vcpu_tlbid_data\n", vcpu->vcpu_idx); + return -EINVAL; + } + + data->vdomain_bitmap = vdomain_bitmap; + vcpu->kvm->arch.vdomain.num_domains = num_vdomains; + vcpu->kvm->arch.vdomain.guest_tlbid_enabled = true; + + kvm_info("vCPU%d: vDomain bitmap: 0x%x\n", vcpu->vcpu_idx, vdomain_bitmap); + + return 0; +} + long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -2110,6 +2251,14 @@ static int kvm_vm_set_attr(struct kvm *kvm, struct kvm_device_attr *attr) } } +static int kvm_arm_get_vdomain_num(struct kvm *kvm, + struct kvm_arm_get_vdomain *vdomain) +{ + vdomain->max_vdomains = 1 << kvm->arch.vdomain.nvis; + + return 0; +} + int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { struct kvm *kvm = filp->private_data; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 0887690718a3..97160783a899 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1278,6 +1278,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_HYGON_COCO_EXT_CSV3_SP_MGR (1 << 4) #define KVM_CAP_ARM_HW_DIRTY_STATE_TRACK 502 +#define KVM_CAP_ARM_TLBIDOMAIN 503 #define KVM_CAP_ARM_HISI_IPIV 798 #define KVM_CAP_ARM_VIRT_MSI_BYPASS 799 @@ -2321,4 +2322,17 @@ struct kvm_pre_fault_memory { __u64 padding[5]; }; +struct kvm_arm_get_vdomain { + __u8 max_vdomains; +}; + +#define KVM_ARM_GET_VDOMAIN_NUM _IOW(KVMIO, 0xd7, struct kvm_arm_get_vdomain) + +struct kvm_arm_set_vdomain { + __u32 vdomain_bitmap; + __u8 num_vdomains; +}; + +#define KVM_ARM_VCPU_SET_VDOMAIN _IOW(KVMIO, 0xd8, struct kvm_arm_set_vdomain) + #endif /* __LINUX_KVM_H */ -- 2.33.0
If the vCPU is first online or is migrated, the domain map needs to be checked whether need to update, and new map needs to be written into the VTLBID(n). Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Zhou Wang <wangzhou1@hisilicon.com> --- arch/arm64/include/asm/kvm_host.h | 11 ++ arch/arm64/kvm/arm.c | 209 ++++++++++++++++++++++++++++++ 2 files changed, 220 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 25c6a5b222b1..57e83079c907 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -45,6 +45,14 @@ #define KVM_VCPU_MAX_FEATURES 9 #define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1) +#define NIS_RANGE_MIN_1_8 1 +#define NIS_RANGE_MAX_1_8 8 +#define NIS_RANGE_MIN_9_16 9 +#define NIS_RANGE_MAX_9_16 16 +#define VTLBID_TD_WIDTH_NIS_1_8 8 +#define VTLBID_TD_WIDTH_NIS_9_16 16 +#define VTLBID_EL2_WIDTH 64 + #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) #define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) @@ -57,6 +65,7 @@ #define KVM_REQ_RELOAD_TLBI_DVMBM KVM_ARCH_REQ(8) #define KVM_REQ_RELOAD_WFI_TRAPS KVM_ARCH_REQ(9) #define KVM_REQ_RELOAD_TIMER_EARLY_INJECT KVM_ARCH_REQ(10) +#define KVM_REQ_RELOAD_VTLBID KVM_ARCH_REQ(11) #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) @@ -232,6 +241,8 @@ struct tlbidomain { u8 nvis; /* TLBIDIDR.NIS */ u8 nis; + /* Used to prevent concurrent modifications to the domain mapping. */ + spinlock_t tlbid_lock; }; struct kvm_arch { diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 1abcb2b2e7e2..dac326dbeeaa 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -43,6 +43,7 @@ #include <asm/kvm_rme.h> #include <asm/sections.h> #include <asm/kvm_tmi.h> +#include <asm/tlbidomain.h> #include <kvm/arm_hypercalls.h> #include <kvm/arm_pmu.h> #include <kvm/arm_psci.h> @@ -359,6 +360,8 @@ static int kvm_arm_init_tlbidomain(struct kvm *kvm) for (i = 0; i < max_vdomains; i++) kvm->arch.vdomain.domain_map[i] = -1; + spin_lock_init(&kvm->arch.vdomain.tlbid_lock); + return 0; free_vdomain_cpumasks: @@ -822,6 +825,205 @@ static void update_steal_time(struct kvm_vcpu *vcpu) } #endif +static bool is_vcpu_in_vdomain(u32 vdomain_bitmap, int vdomain_idx) +{ + return vdomain_bitmap & BIT(vdomain_idx); +} + +static int update_domain_map(struct kvm_vcpu *vcpu, int vdomain_idx, + int last_pcpu, bool clear_flag) +{ + struct tlbidomain *vdomain = &vcpu->kvm->arch.vdomain; + int pdomain = vdomain->domain_map[vdomain_idx]; + cpumask_var_t new_cpus; + + cpumask_copy(new_cpus, vdomain->vdomain_cpumasks[vdomain_idx]); + + if (unlikely(pdomain == -1)) { + cpumask_set_cpu(vcpu->cpu, new_cpus); + goto update_mask; + } + + if (last_pcpu != -1 && clear_flag) { + /* remove old_cpu, not need to do so when first load */ + if (cpumask_test_cpu(last_pcpu, new_cpus)) { + cpumask_clear_cpu(last_pcpu, new_cpus); + } + } + + cpumask_set_cpu(vcpu->cpu, new_cpus); + +update_mask: + cpumask_copy(vdomain->vdomain_cpumasks[vdomain_idx], new_cpus); + return pick_best_domain(vdomain->vdomain_cpumasks[vdomain_idx]); +} + +static void vtlbidn_clear_set_s(int idx, u64 clear, u64 set) +{ + switch (idx) { + case 0: + sysreg_clear_set_s(SYS_VTLBID0_EL2, clear, set); + break; + case 1: + sysreg_clear_set_s(SYS_VTLBID1_EL2, clear, set); + break; + case 2: + sysreg_clear_set_s(SYS_VTLBID2_EL2, clear, set); + break; + case 3: + sysreg_clear_set_s(SYS_VTLBID3_EL2, clear, set); + break; + default: + BUG_ON(1); + } +} + +static void kvm_arm_update_tlbid_map(struct tlbidomain *vdomain) +{ + int i, pdomain, vtlbid_idx, offset; + int pdomain_bits, nis = vdomain->nis; + int max_vdomains = 1 << vdomain->nvis; + u64 clear, set; + + if (nis >= NIS_RANGE_MIN_1_8 && nis <= NIS_RANGE_MAX_1_8) + pdomain_bits = VTLBID_TD_WIDTH_NIS_1_8; + else if (nis >= NIS_RANGE_MIN_9_16 && nis <= NIS_RANGE_MAX_9_16) + pdomain_bits = VTLBID_TD_WIDTH_NIS_9_16; + + for (i = 0; i < max_vdomains; i++) { + pdomain = vdomain->domain_map[i]; + if (pdomain == -1) { + if (vdomain->domain_map[0] != -1) + pdomain = vdomain->domain_map[0]; + else + pdomain = 0; + } + + vtlbid_idx = i * pdomain_bits / VTLBID_EL2_WIDTH; + offset = i * pdomain_bits % VTLBID_EL2_WIDTH; + + clear = GENMASK(offset + pdomain_bits - 1, offset); + set = pdomain << offset; + + vtlbidn_clear_set_s(vtlbid_idx, clear, set); + } + + sysreg_clear_set_s(SYS_HCRX_EL2, 0, HCRX_EL2_VTLBIDEn); +} + +static void kvm_vcpu_reload_tlbid(struct kvm *kvm) +{ + if (WARN_ON_ONCE(!kvm->arch.vdomain.kvm_tlbid_enabled)) + return; + + preempt_disable(); + kvm_arm_update_tlbid_map(&kvm->arch.vdomain); + preempt_enable(); +} + +static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + struct tlbidomain *vdomain = &kvm->arch.vdomain; + int max_vdomains = 1 << vdomain->nvis; + DECLARE_BITMAP(clear_bits, max_vdomains); + struct vcpu_tlbid_data *current_data, *other_data; + int i; + + /* + * If we support tlbid, but user does not enable it, we can do + * vdomain0 -> pdomainN as well by default. + */ + if (!vdomain->kvm_tlbid_enabled) + return; + + current_data = xa_load(&vdomain->vcpu_data_array, vcpu->vcpu_idx); + if (!current_data) { + BUG(); + return; + } + + /* check if vCPU thread will move to another pCPU */ + if (likely(vcpu->cpu == current_data->last_pcpu)) + return; + + spin_lock(&vcpu->kvm->arch.vdomain.tlbid_lock); + + /* + * check if another vcpu running on the same pCPU as the one + * the vCPU last ran. If so, do not clear last pCPU from the + * vdomain_cpumask. + */ + bitmap_fill(clear_bits, max_vdomains); + for (i = 0; i < kvm->created_vcpus; i++) { + other_data = xa_load(&vdomain->vcpu_data_array, i); + if (WARN_ON_ONCE(!other_data)) + continue; + + if (i == vcpu->vcpu_idx || other_data->last_pcpu == -1) + continue; + + if (other_data->last_pcpu != current_data->last_pcpu) + continue; + + bitmap_andnot(clear_bits, clear_bits, + (unsigned long *)&other_data->vdomain_bitmap, max_vdomains); + } + + if (!vdomain->guest_tlbid_enabled) { + /* + * If QEMU is not configured with vdomains, only vDomain0 needs + * to be mapped. + */ + vdomain->domain_map[0] = update_domain_map(vcpu, + 0, + current_data->last_pcpu, + test_bit(0, clear_bits)); + } else { + /* + * If QEMU is configured with vdomains, all vdomains need to be + * mapped. For all vdomains which includes this vCPU, check if + * vdomain map should be changed as well. + */ + for (i = 0; i < vdomain->num_domains; i++) { + if (!is_vcpu_in_vdomain(current_data->vdomain_bitmap, i)) + continue; + + vdomain->domain_map[i] = update_domain_map(vcpu, + i, + current_data->last_pcpu, + test_bit(i, clear_bits)); + } + } + + kvm_flush_remote_tlbs(kvm); + + /* + * Before this vcpu load, kick other vCPUs out, so maps for other vCPUs + * will be updated during vCPU load. + * + * Add KVM_REQUEST_WAIT to make sure vCPU is out. + */ + kvm_make_all_cpus_request(kvm, KVM_REQ_RELOAD_VTLBID | KVM_REQUEST_WAIT); + + /* update tlbid hardware map register */ + kvm_arm_update_tlbid_map(vdomain); + + current_data->last_pcpu = vcpu->cpu; + + spin_unlock(&vcpu->kvm->arch.vdomain.tlbid_lock); +} + +static void kvm_tlbidomain_vcpu_put(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + + if (!kvm->arch.vdomain.kvm_tlbid_enabled) + return; + + sysreg_clear_set_s(SYS_HCRX_EL2, HCRX_EL2_VTLBIDEn, 0); +} + void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) { struct kvm_s2_mmu *mmu; @@ -877,6 +1079,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) kvm_tlbi_dvmbm_vcpu_load(vcpu); + kvm_tlbidomain_vcpu_load(vcpu); + /* * When pv_preempted is changed from enabled to disabled, preempted * state will not be updated in kvm_arch_vcpu_put/load. So we must @@ -915,6 +1119,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_tlbi_dvmbm_vcpu_put(vcpu); + kvm_tlbidomain_vcpu_put(vcpu); + if (kvm_arm_is_pvsched_valid(&vcpu->arch) && pv_preempted_enable) kvm_update_pvsched_preempted(vcpu, 1); } @@ -1280,6 +1486,9 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_RELOAD_TLBI_DVMBM, vcpu)) kvm_hisi_reload_lsudvmbm(vcpu->kvm); + if (kvm_check_request(KVM_REQ_RELOAD_VTLBID, vcpu)) + kvm_vcpu_reload_tlbid(vcpu->kvm); + if (kvm_check_request(KVM_REQ_RELOAD_WFI_TRAPS, vcpu)) { if (single_task_running()) vcpu_clear_wfx_traps(vcpu); -- 2.33.0
Expose TLBID virtualization interface to userspace. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/arm.c | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index dac326dbeeaa..f15f7128d67b 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -467,6 +467,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) bitmap_zero(kvm->arch.vcpu_features, KVM_VCPU_MAX_FEATURES); + kvm_arm_init_tlbidomain(kvm); + /* Initialise the realm bits after the generic bits are enabled */ if (kvm_is_realm(kvm)) { ret = kvm_init_realm_vm(kvm); @@ -511,6 +513,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm) kvm_arm_teardown_hypercalls(kvm); kvm_destroy_realm(kvm); + free_tlbid_vdomain(kvm); } #ifdef CONFIG_ARM64_HISI_IPIV @@ -663,6 +666,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = 0; break; #endif + case KVM_CAP_ARM_TLBIDOMAIN: + if (system_supports_tlbid()) + r = 1; + else + r = 0; + break; case KVM_CAP_ARM_RME: r = static_key_enabled(&kvm_rme_is_available); break; @@ -2144,7 +2153,9 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, kvm_arm_pvtimer_status_set_active(vcpu, false); #endif - return 0; + ret = kvm_arm_tlbidomain_vcpu_init(vcpu); + + return ret; } static int kvm_arm_vcpu_set_attr(struct kvm_vcpu *vcpu, @@ -2395,6 +2406,17 @@ long kvm_arch_vcpu_ioctl(struct file *filp, return -EFAULT; return kvm_arm_vcpu_rmm_psci_complete(vcpu, &req); } + case KVM_ARM_VCPU_SET_VDOMAIN: { + struct kvm_arm_set_vdomain vdomain; + + if (!kvm_vcpu_initialized(vcpu)) + return -ENOEXEC; + + if (copy_from_user(&vdomain, argp, sizeof(vdomain))) + return -EFAULT; + + return kvm_arm_vcpu_set_vdomain(vcpu, &vdomain); + } default: r = -EINVAL; } @@ -2575,6 +2597,15 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) return -EFAULT; return kvm_vm_ioctl_get_reg_writable_masks(kvm, &range); } + case KVM_ARM_GET_VDOMAIN_NUM: { + struct kvm_arm_get_vdomain vdomain; + + kvm_arm_get_vdomain_num(kvm, &vdomain); + if (copy_to_user(argp, &vdomain, sizeof(vdomain))) + return -EFAULT; + + return 0; + } default: return -EINVAL; } -- 2.33.0
DVMBM and TLBID are two mutually exclusive features. When BIOS enables DVMBM, the TLBID field in MMFR4 is masked, thus the OS will not enable TLBID. However, when BIOS enables TLBID, the DVMBM field in AIDR remains set to true, causing DVMBM to be enabled in the OS. This issue can be intercepted and handled at the software level. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/hisilicon/hisi_virt.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/arm64/kvm/hisilicon/hisi_virt.c b/arch/arm64/kvm/hisilicon/hisi_virt.c index 9a04e4052abd..c5dda4cd2f83 100644 --- a/arch/arm64/kvm/hisilicon/hisi_virt.c +++ b/arch/arm64/kvm/hisilicon/hisi_virt.c @@ -243,6 +243,15 @@ bool hisi_dvmbm_supported(void) return false; } + /* + * When TLBID is enabled, DVMBM cannot be enabled. + * After BIOS enables TLBID, the DVMBM field of AIDR + * remains set, therefore interception needs to be + * performed in the code. + */ + if (system_supports_tlbid()) + return false; + /* Determine whether DVMBM is supported by the hardware */ if (!(read_sysreg(aidr_el1) & AIDR_EL1_DVMBM_MASK)) return false; -- 2.33.0
Set the NVIS/NVOS fields to populate NIS/NOS and expose TLBIDIDR_EL1 to the guest. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/asm/kvm_host.h | 3 +++ arch/arm64/kvm/sys_regs.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 57e83079c907..ff0463feb290 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -503,6 +503,9 @@ enum vcpu_sysreg { CNTHV_CTL_EL2, CNTHV_CVAL_EL2, + /* TLBI Domains Identification Register (EL1) */ + KABI_EXTEND_ENUM(TLBIDIDR_EL1) + NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index c94b24311525..fce0f1481252 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -276,6 +276,17 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, return true; } +static bool access_tlbididr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return ignore_write(vcpu, p); + + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + return true; +} + static bool access_actlr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -715,6 +726,22 @@ static u64 reset_amair_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) return amair; } +static u64 reset_tlbididr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + if (!system_supports_tlbid()) + return 0; + + u64 tlbididr = read_sysreg_s(SYS_TLBIDIDR_EL1); + u64 nvis = FIELD_GET(TLBIDIDR_EL1_NVIS_MASK, tlbididr); + u64 nvos = FIELD_GET(TLBIDIDR_EL1_NVOS_MASK, tlbididr); + + tlbididr = FIELD_PREP(TLBIDIDR_EL1_NIS_MASK, nvis); + tlbididr |= FIELD_PREP(TLBIDIDR_EL1_NOS_MASK, nvos); + + vcpu_write_sys_reg(vcpu, tlbididr, TLBIDIDR_EL1); + return tlbididr; +} + static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { u64 actlr = read_sysreg(actlr_el1); @@ -2616,6 +2643,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORN_EL1), trap_loregion }, { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_MPAMIDR_EL1), workaround_bad_mpam_abi }, + { SYS_DESC(SYS_TLBIDIDR_EL1), access_tlbididr, reset_tlbididr_el1, TLBIDIDR_EL1 }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, { SYS_DESC(SYS_MPAM1_EL1), workaround_bad_mpam_abi }, -- 2.33.0
Add the ID_AA64MMFR4_EL1 register and set it to be writable in userspace. When configuring vTLBID in QEMU, expose the TLBID field to userspace. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/uapi/asm/kvm.h | 2 ++ arch/arm64/kvm/arm.c | 4 ++++ arch/arm64/kvm/sys_regs.c | 28 +++++++++++++++++++++++++++- arch/arm64/kvm/sys_regs.h | 2 ++ 4 files changed, 35 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 93de3f019e5b..99c0e43c43dd 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -253,6 +253,8 @@ struct kvm_arm_counter_offset { #define ARM64_SYS_REG(...) (__ARM64_SYS_REG(__VA_ARGS__) | KVM_REG_SIZE_U64) +#define KVM_REG_ARM_ID_AA64MMFR4_EL1 ARM64_SYS_REG(3, 0, 0, 7, 4) + /* Physical Timer EL0 Registers */ #define KVM_REG_ARM_PTIMER_CTL ARM64_SYS_REG(3, 3, 14, 2, 1) #define KVM_REG_ARM_PTIMER_CVAL ARM64_SYS_REG(3, 3, 14, 2, 2) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index f15f7128d67b..b2e5f27d89fd 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -51,6 +51,7 @@ static enum kvm_mode kvm_mode = KVM_MODE_DEFAULT; #include "hisilicon/hisi_virt.h" +#include "sys_regs.h" DEFINE_STATIC_KEY_FALSE(kvm_rme_is_available); @@ -2250,6 +2251,7 @@ static int kvm_arm_vcpu_set_vdomain(struct kvm_vcpu *vcpu, u8 nvis = vcpu->kvm->arch.vdomain.nvis; u8 max_vdomains = 1 << nvis; struct vcpu_tlbid_data *data; + int ret; if (num_vdomains > max_vdomains) return -EINVAL; @@ -2270,6 +2272,8 @@ static int kvm_arm_vcpu_set_vdomain(struct kvm_vcpu *vcpu, kvm_info("vCPU%d: vDomain bitmap: 0x%x\n", vcpu->vcpu_idx, vdomain_bitmap); + kvm_update_aa64mmfr4_tlbid(vcpu->kvm); + return 0; } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index fce0f1481252..86e431d87bd7 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1469,6 +1469,10 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu, val &= ID_AA64MMFR3_EL1_TCRX | ID_AA64MMFR3_EL1_S1POE | ID_AA64MMFR3_EL1_S1PIE; break; + case SYS_ID_AA64MMFR4_EL1: + if (!vcpu->kvm->arch.vdomain.guest_tlbid_enabled) + val &= ~ID_AA64MMFR4_EL1_TLBID; + break; case SYS_ID_MMFR4_EL1: val &= ~ARM64_FEATURE_MASK(ID_MMFR4_EL1_CCIDX); break; @@ -2562,7 +2566,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { ID_WRITABLE(ID_AA64MMFR3_EL1, (ID_AA64MMFR3_EL1_TCRX | ID_AA64MMFR3_EL1_S1PIE | ID_AA64MMFR3_EL1_S1POE)), - ID_UNALLOCATED(7,4), + ID_WRITABLE(ID_AA64MMFR4_EL1, ID_AA64MMFR4_EL1_TLBID_MASK), ID_UNALLOCATED(7,5), ID_UNALLOCATED(7,6), ID_UNALLOCATED(7,7), @@ -3777,6 +3781,28 @@ const struct sys_reg_desc *get_reg_by_id(u64 id, return find_reg(¶ms, table, num); } +/* + * Update the VM's stored value for an ID register. + */ +void kvm_update_aa64mmfr4_tlbid(struct kvm *kvm) +{ + u64 val, tlbid_val; + + mutex_lock(&kvm->arch.config_lock); + + val = kvm_read_vm_id_reg(kvm, SYS_ID_AA64MMFR4_EL1); + tlbid_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR4_EL1); + tlbid_val &= ID_AA64MMFR4_EL1_TLBID_MASK; + + val &= ~ID_AA64MMFR4_EL1_TLBID_MASK; + if (kvm->arch.vdomain.guest_tlbid_enabled) + val |= tlbid_val; + + kvm_set_vm_id_reg(kvm, SYS_ID_AA64MMFR4_EL1, val); + + mutex_unlock(&kvm->arch.config_lock); +} + /* Decode an index value, and find the sys_reg_desc entry. */ static const struct sys_reg_desc * id_to_sys_reg_desc(struct kvm_vcpu *vcpu, u64 id, diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index 3080693719d2..4c5ae75bc9eb 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -238,6 +238,8 @@ int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, const struct sys_reg_desc table[], unsigned int num); +void kvm_update_aa64mmfr4_tlbid(struct kvm *kvm); + #define AA32(_x) .aarch32_map = AA32_##_x #define Op0(_x) .Op0 = _x #define Op1(_x) .Op1 = _x -- 2.33.0
Enable the TLBIDIDR_EL1 trap when the vCPU loading. Disable it when the vCPU putting. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/kvm/arm.c | 21 +++++++++++++++++++++ arch/arm64/tools/sysreg | 4 +++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index b2e5f27d89fd..c517cf67d28e 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -931,6 +931,15 @@ static void kvm_vcpu_reload_tlbid(struct kvm *kvm) preempt_enable(); } +static inline void set_tlbididr_trap(void) +{ + u64 val; + val = read_sysreg_s(SYS_HFGRTR2_EL2); + val &= ~HFGRTR2_EL2_nTLBIDIDR_EL1_MASK; + write_sysreg_s(val, SYS_HFGRTR2_EL2); + isb(); +} + static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->kvm; @@ -947,6 +956,8 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) if (!vdomain->kvm_tlbid_enabled) return; + set_tlbididr_trap(); + current_data = xa_load(&vdomain->vcpu_data_array, vcpu->vcpu_idx); if (!current_data) { BUG(); @@ -1024,6 +1035,15 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) spin_unlock(&vcpu->kvm->arch.vdomain.tlbid_lock); } +static inline void clear_tlbididr_trap(void) +{ + u64 val; + val = read_sysreg_s(SYS_HFGRTR2_EL2); + val |= HFGRTR2_EL2_nTLBIDIDR_EL1_MASK; + write_sysreg_s(val, SYS_HFGRTR2_EL2); + isb(); +} + static void kvm_tlbidomain_vcpu_put(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->kvm; @@ -1031,6 +1051,7 @@ static void kvm_tlbidomain_vcpu_put(struct kvm_vcpu *vcpu) if (!kvm->arch.vdomain.kvm_tlbid_enabled) return; + clear_tlbididr_trap(); sysreg_clear_set_s(SYS_HCRX_EL2, HCRX_EL2_VTLBIDEn, 0); } diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 99e615204a78..b416052e0cee 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -3000,7 +3000,9 @@ Field 0 nPMECR_EL1 EndSysreg Sysreg HFGRTR2_EL2 3 4 3 1 2 -Res0 63:15 +Res0 63:31 +Field 30 nTLBIDIDR_EL1 +Res0 29:15 Field 14 nACTLRALIAS_EL1 Field 13 nACTLRMASK_EL1 Field 12 nTCR2ALIAS_EL1 -- 2.33.0
In kvm_tlbidomain_vcpu_load(), when a vCPU migrates to a different pCPU, the code iterates over all created vCPUs to find those whose last_pcpu matches the current vCPU last_pcpu. This O(logN) loop with xa_load() becomes a significant bottleneck when the number of vCPUs is large. Introduce a per-pCPU linked list (pcpu_vcpu_list) in struct tlbidomain to maintain reverse mapping from pCPU to vCPUs. Each vcpu_tlbid_data gets a pcpu_node to link into the list of its last_pcpu. This allows the clear_bits calculation to only iterate over vCPUs that actually share the same last_pcpu, reducing the complexity from O(logN) to O(1) in the typical case. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/include/asm/kvm_host.h | 4 +++ arch/arm64/kvm/arm.c | 46 ++++++++++++++++++++----------- 2 files changed, 34 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index ff0463feb290..181d4c441801 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -218,6 +218,8 @@ struct kvm_protected_vm { }; struct vcpu_tlbid_data { + /* Node in per-pCPU vCPU list of tlbidomain */ + struct list_head pcpu_node; /* Store vDomain bitmap, indexed by vCPU ID */ u32 vdomain_bitmap; /* Record the last pCPU where vCPU was running, indexed by vCPU ID */ @@ -243,6 +245,8 @@ struct tlbidomain { u8 nis; /* Used to prevent concurrent modifications to the domain mapping. */ spinlock_t tlbid_lock; + /* Per-pCPU list of vCPUs whose last_pcpu equals this pCPU */ + struct list_head *pcpu_vcpu_list; }; struct kvm_arch { diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index c517cf67d28e..3979eb9dcae0 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -363,6 +363,14 @@ static int kvm_arm_init_tlbidomain(struct kvm *kvm) spin_lock_init(&kvm->arch.vdomain.tlbid_lock); + vdomain->pcpu_vcpu_list = kcalloc(nr_cpu_ids, + sizeof(*vdomain->pcpu_vcpu_list), GFP_KERNEL); + if (!vdomain->pcpu_vcpu_list) + goto free_vdomain_cpumasks; + + for (i = 0; i < nr_cpu_ids; i++) + INIT_LIST_HEAD(&vdomain->pcpu_vcpu_list[i]); + return 0; free_vdomain_cpumasks: @@ -392,6 +400,8 @@ static void free_tlbid_vdomain(struct kvm *kvm) free_cpumask_var(kvm->arch.vdomain.vdomain_cpumasks[i]); kfree(kvm->arch.vdomain.vdomain_cpumasks); + kfree(kvm->arch.vdomain.pcpu_vcpu_list); + xa_for_each(&kvm->arch.vdomain.vcpu_data_array, index, data) { kfree(data); } @@ -971,24 +981,25 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) spin_lock(&vcpu->kvm->arch.vdomain.tlbid_lock); /* - * check if another vcpu running on the same pCPU as the one - * the vCPU last ran. If so, do not clear last pCPU from the - * vdomain_cpumask. + * Remove current vCPU from its last_pcpu list and compute clear_bits + * only when there was a previous pCPU. On first load (last_pcpu == -1), + * no previous pCPU needs to be cleared from vdomain_cpumask. */ - bitmap_fill(clear_bits, max_vdomains); - for (i = 0; i < kvm->created_vcpus; i++) { - other_data = xa_load(&vdomain->vcpu_data_array, i); - if (WARN_ON_ONCE(!other_data)) - continue; - - if (i == vcpu->vcpu_idx || other_data->last_pcpu == -1) - continue; + if (current_data->last_pcpu != -1) { + list_del_init(¤t_data->pcpu_node); - if (other_data->last_pcpu != current_data->last_pcpu) - continue; - - bitmap_andnot(clear_bits, clear_bits, - (unsigned long *)&other_data->vdomain_bitmap, max_vdomains); + /* + * Iterate only vCPUs that share the same last_pcpu to compute + * clear_bits. This is O(1) in the typical case since most pCPUs + * host only one vCPU at a time. + */ + bitmap_fill(clear_bits, max_vdomains); + list_for_each_entry(other_data, + &vdomain->pcpu_vcpu_list[current_data->last_pcpu], + pcpu_node) { + bitmap_andnot(clear_bits, clear_bits, + (unsigned long *)&other_data->vdomain_bitmap, max_vdomains); + } } if (!vdomain->guest_tlbid_enabled) { @@ -1031,6 +1042,8 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) kvm_arm_update_tlbid_map(vdomain); current_data->last_pcpu = vcpu->cpu; + list_add_tail(¤t_data->pcpu_node, + &vdomain->pcpu_vcpu_list[vcpu->cpu]); spin_unlock(&vcpu->kvm->arch.vdomain.tlbid_lock); } @@ -2104,6 +2117,7 @@ static int kvm_arm_tlbidomain_vcpu_init(struct kvm_vcpu *vcpu) */ data->vdomain_bitmap = 0x1; data->last_pcpu = -1; + INIT_LIST_HEAD(&data->pcpu_node); ret = xa_insert(&vcpu->kvm->arch.vdomain.vcpu_data_array, vcpu->vcpu_idx, data, GFP_KERNEL); -- 2.33.0
In kvm_tlbidomain_vcpu_load(), the domain map is recomputed on every vCPU load, followed by an expensive kvm_flush_remote_tlbs() and kvm_make_all_cpus_request() to invalidate TLBs and IPI all vCPUs. However, when a vCPU is scheduled back onto the same pCPU or the pDomain assignment does not change, the domain map remains the same and the invalidation + IPI is unnecessary. Record the old domain map value before recomputing, and compare it with the new one. If no vDomain's mapping has changed, skip the TLB flush and vCPU IPI, jumping directly to update the tlbid hardware map register. This avoids redundant cross-CPU IPIs and TLB shootdowns in the common case where a vCPU migrates back to its previous pCPU, significantly reducing the overhead of the vCPU load path. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/arm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 3979eb9dcae0..fed48b4ce23c 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -957,7 +957,8 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) int max_vdomains = 1 << vdomain->nvis; DECLARE_BITMAP(clear_bits, max_vdomains); struct vcpu_tlbid_data *current_data, *other_data; - int i; + bool domain_map_changed = false; + int i, old_map; /* * If we support tlbid, but user does not enable it, we can do @@ -1003,6 +1004,8 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) } if (!vdomain->guest_tlbid_enabled) { + old_map = vdomain->domain_map[0]; + /* * If QEMU is not configured with vdomains, only vDomain0 needs * to be mapped. @@ -1011,6 +1014,8 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) 0, current_data->last_pcpu, test_bit(0, clear_bits)); + if (vdomain->domain_map[0] == old_map) + goto unlock; } else { /* * If QEMU is configured with vdomains, all vdomains need to be @@ -1021,11 +1026,18 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) if (!is_vcpu_in_vdomain(current_data->vdomain_bitmap, i)) continue; + old_map = vdomain->domain_map[i]; vdomain->domain_map[i] = update_domain_map(vcpu, i, current_data->last_pcpu, test_bit(i, clear_bits)); + + if (vdomain->domain_map[i] != old_map) + domain_map_changed = true; } + + if (!domain_map_changed) + goto unlock; } kvm_flush_remote_tlbs(kvm); @@ -1038,6 +1050,7 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) */ kvm_make_all_cpus_request(kvm, KVM_REQ_RELOAD_VTLBID | KVM_REQUEST_WAIT); +unlock: /* update tlbid hardware map register */ kvm_arm_update_tlbid_map(vdomain); -- 2.33.0
In kvm_tlbidomain_vcpu_load(), three trace_printk() calls are used to log vCPU load info and vDomain-to-pDomain mapping. trace_printk() is a debug-only facility that is not meant for production use: it always prints to the ftrace ring buffer when enabled, cannot be dynamically controlled per-event, and may impact performance. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/arm.c | 10 ++++++++ arch/arm64/kvm/trace_arm.h | 47 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index fed48b4ce23c..cbad12aa059e 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1002,6 +1002,9 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) (unsigned long *)&other_data->vdomain_bitmap, max_vdomains); } } + trace_kvm_tlbid_vcpu_info(vcpu->vcpu_idx, vcpu->cpu, + current_data->last_pcpu, + current_data->vdomain_bitmap); if (!vdomain->guest_tlbid_enabled) { old_map = vdomain->domain_map[0]; @@ -1014,6 +1017,10 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) 0, current_data->last_pcpu, test_bit(0, clear_bits)); + trace_kvm_tlbid_domain_mapping(0, + vdomain->vdomain_cpumasks[0], + vdomain->domain_map[0]); + if (vdomain->domain_map[0] == old_map) goto unlock; } else { @@ -1031,6 +1038,9 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) i, current_data->last_pcpu, test_bit(i, clear_bits)); + trace_kvm_tlbid_domain_mapping(i, + vdomain->vdomain_cpumasks[i], + vdomain->domain_map[i]); if (vdomain->domain_map[i] != old_map) domain_map_changed = true; diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index 5b91456560b5..5f7fd7a487fc 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -408,6 +408,53 @@ TRACE_EVENT(kvm_pvspin_kick_vcpu, __entry->vcpu_id, __entry->target_vcpu_id) ); +TRACE_EVENT(kvm_tlbid_vcpu_info, + TP_PROTO(int vcpu_idx, int current_pcpu, int last_pcpu, + u32 vdomain_bitmap), + TP_ARGS(vcpu_idx, current_pcpu, last_pcpu, vdomain_bitmap), + + TP_STRUCT__entry( + __field(int, vcpu_idx ) + __field(int, current_pcpu ) + __field(int, last_pcpu ) + __field(u32, vdomain_bitmap ) + ), + + TP_fast_assign( + __entry->vcpu_idx = vcpu_idx; + __entry->current_pcpu = current_pcpu; + __entry->last_pcpu = last_pcpu; + __entry->vdomain_bitmap = vdomain_bitmap; + ), + + TP_printk("vcpu=%d current_pcpu=%d last_pcpu=%d vdomain_bitmap=0x%x", + __entry->vcpu_idx, __entry->current_pcpu, + __entry->last_pcpu, __entry->vdomain_bitmap) +); + +TRACE_EVENT(kvm_tlbid_domain_mapping, + TP_PROTO(int vdomain_id, const struct cpumask *cpumask, + int pdomain_id), + TP_ARGS(vdomain_id, cpumask, pdomain_id), + + TP_STRUCT__entry( + __field(int, vdomain_id ) + __cpumask(cpumask) + __field(int, pdomain_id ) + ), + + TP_fast_assign( + __entry->vdomain_id = vdomain_id; + __assign_cpumask(cpumask, cpumask_bits(cpumask)); + __entry->pdomain_id = pdomain_id; + ), + + TP_printk("Domain mapping: vDomain=%d, cpumask=%s, pDomain=%d", + __entry->vdomain_id, + __get_cpumask(cpumask), + __entry->pdomain_id) +); + #endif /* _TRACE_ARM_ARM64_KVM_H */ #undef TRACE_INCLUDE_PATH -- 2.33.0
From: Tian Zheng <zhengtian10@huawei.com> kvm_update_aa64mmfr4_tlbid() calls kvm_set_vm_id_reg() which triggers KVM_BUG_ON when kvm_vm_has_ran_once() is true, causing a kernel panic on vCPU hotplug. The VM-level ID_AA64MMFR4.TLBID is already set during the first vCPU init and doesn't need to be rewritten. Skip kvm_set_vm_id_reg() when the VM has already started. Fixes: 83eec466269a ("KVM: arm64: Expose ID_AA64MMFR4_EL1_TLBID to guest") Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/kvm/sys_regs.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 86e431d87bd7..4766ece99cba 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3790,6 +3790,17 @@ void kvm_update_aa64mmfr4_tlbid(struct kvm *kvm) mutex_lock(&kvm->arch.config_lock); + if (kvm_vm_has_ran_once(kvm)) { + /* + * ID_AA64MMFR4 is a VM-level register already set during + * the first vCPU init. Skip the write to avoid triggering + * KVM_BUG_ON in kvm_set_vm_id_reg(). The per-vCPU + * vdomain_bitmap is still updated by the caller. + */ + mutex_unlock(&kvm->arch.config_lock); + return; + } + val = kvm_read_vm_id_reg(kvm, SYS_ID_AA64MMFR4_EL1); tlbid_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR4_EL1); tlbid_val &= ID_AA64MMFR4_EL1_TLBID_MASK; -- 2.33.0
反馈: 您发送到kernel@openeuler.org的补丁/补丁集，转换为PR失败！ 邮件列表地址：https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/ZL5... 失败原因：应用补丁/补丁集失败，Patch failed at 0001 arm64/sysreg: Add TLBID sysreg 建议解决方法：请查看失败原因， 确认补丁是否可以应用在当前期望分支的最新代码上 FeedBack: The patch(es) which you have sent to kernel@openeuler.org has been converted to PR failed! Mailing list address: https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/ZL5... Failed Reason: apply patch(es) failed, Patch failed at 0001 arm64/sysreg: Add TLBID sysreg Suggest Solution: please checkout if the failed patch(es) can work on the newest codes in expected branch
participants (2)
-
Jinqian Yang -
patchwork bot