[PATCH OLK-6.6 0/8] Add support for vTLBID
This series adds KVM/arm64 support for the ARM TLB invalidation domain (TLBID) extension. With TLBID a guest can scope its TLB invalidation broadcasts to a virtual domain (vDomain). User space discovers the feature through the new KVM_CAP_ARM_TLBIDOMAIN capability. The number of supported vDomains is queried with the new VM ioctl KVM_ARM_GET_VDOMAIN_NUM, and the per-vCPU ioctl KVM_ARM_VCPU_SET_VDOMAIN tells KVM which vDomains each vCPU belongs to, via a vDomain bitmap. KVM maintains a per-VM mapping from vDomains to the physical CPUs that run their member vCPUs, and reprogrammes the VTLBID(n)_EL2 registers on every vCPU load, so that a guest TLBI tagged with a vDomain ID only reaches the physical CPUs currently running that vDomain's vCPUs. Because DVMBM and TLBID are mutually exclusive, DVMBM is force-disabled once TLBID is enabled for a VM. On the guest side, TLBIDIDR_EL1 is emulated with virtual domain counts clamped to the hardware limits, the ID_AA64MMFR4_EL1.TLBID field is exposed only when the feature is active for the VM, and guest accesses to TLBIDIDR_EL1 are trapped and handled by KVM. The feature is gated behind CONFIG_KVM_ARM_VTLBID \and additionally requires hardware TLBID support at run time. Tracepoints are added for the vCPU load path so that the vDomain-to-pCPU mapping can be observed. Jinqian Yang (8): KVM: arm64: Introduce TLBID virtualization infrastructure KVM: arm64: Add VTLBID sysregs and update VTLBID(n) on vCPU load KVM: arm64: Wire up TLBID vDomain configuration ioctls KVM: arm64: Ensure DVMBM is disabled when TLBID is enabled KVM: arm64: Emulate TLBIDIDR_EL1 with virtual domain counts KVM: arm64: Conditionally expose ID_AA64MMFR4_EL1.TLBID to guest KVM: arm64: Enable TLBIDIDR_EL1 access trap in guest KVM: arm64: Add tracepoints for TLBID vDomain load events Documentation/virt/kvm/api.rst | 58 +++ arch/arm64/include/asm/kvm_host.h | 53 +++ arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kvm/Kconfig | 12 + arch/arm64/kvm/arm.c | 530 ++++++++++++++++++++++++++- arch/arm64/kvm/hisilicon/hisi_virt.c | 9 + arch/arm64/kvm/sys_regs.c | 76 +++- arch/arm64/kvm/sys_regs.h | 6 + arch/arm64/kvm/trace_arm.h | 47 +++ arch/arm64/tools/sysreg | 23 +- include/uapi/linux/kvm.h | 14 + 11 files changed, 826 insertions(+), 4 deletions(-) -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- Add the core TLBID virtualization plumbing: - Define struct tlbidomain embedded in struct kvm_arch, holding per-VM virtual-TLBID state: domain map, per-vDomain cpumasks, per-vCPU data, guest-visible TLBIDIDR value, and guard flags. - Introduce CONFIG_KVM_ARM_VTLBID with #ifdef protection so the feature can be compiled out. - Implement kvm_arm_init_tlbidomain() / free_tlbid_vdomain() to bootstrap and tear down the per-VM domain state on VM creation and destruction. - Implement kvm_arm_tlbidomain_vcpu_init() to allocate the per-vCPU TLBID data. Default the vCPU to vDomain0 so KVM can map vDomain0 to a pDomain for guests that have not enabled TLBID. - Provide the vDomain configuration helpers kvm_arm_vcpu_set_vdomain() / kvm_arm_get_vdomain_num() for the later ioctl wiring. - Define the UAPI constants (capability, ioctl numbers, vDomain structure) and document the new KVM_CAP_ARM_TLBIDOMAIN in Documentation/virt/kvm/api.rst. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Zhou Wang <wangzhou1@hisilicon.com> --- Documentation/virt/kvm/api.rst | 58 +++++++++ arch/arm64/include/asm/kvm_host.h | 53 ++++++++ arch/arm64/kvm/Kconfig | 12 ++ arch/arm64/kvm/arm.c | 210 ++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 14 ++ 5 files changed, 347 insertions(+) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 77c65f16e2a7..940e0a2f9417 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6424,6 +6424,48 @@ the capability to be present. `flags` must currently be zero. +4.145 KVM_ARM_GET_VDOMAIN_NUM +------------------------------ + +:Capability: KVM_CAP_ARM_TLBIDOMAIN +:Architectures: arm64 +:Type: vm ioctl +:Parameters: struct kvm_arm_get_vdomain (out) +:Returns: 0 on success, < 0 on error + +Parameters are specified via the following structure:: + + struct kvm_arm_get_vdomain { + __u8 max_vdomains; + }; + +The ``max_vdomains`` field is the max number of vDomains supported by the +hardware. This value is determined by TLBIDIDR_EL1[NVOS/NVIS]. NVOS/NVIS +is the bit width of the vdomain. + +4.146 KVM_ARM_VCPU_SET_VDOMAIN +------------------------------ + +:Capability: KVM_CAP_ARM_TLBIDOMAIN +:Architectures: arm64 +:Type: vcpu ioctl +:Parameters: struct kvm_arm_set_vdomain (in) +:Returns: 0 on success, < 0 on error + +This ioctl allows KVM to obtain TLBI Domain of Guest. + +Parameters are specified via the following structure:: + + struct kvm_arm_set_vdomain { + __u32 vdomain_bitmap; + __u8 num_vdomains; + }; + +The ``vdomain_bitmap`` field is a bitmap, where '1' indicates that vDomain +includes this vCPU. For example, ``vCPU0: vdomain_bitmap=0b1011`` indicates +that vCPU0 included in vDomain(0, 1, 3). + +The ``num_vdomains`` field is the number of vDomains. 5. The kvm_run structure ======================== @@ -8218,6 +8260,22 @@ Used to configure and set up the memory for a Realm. The available actions are: enter the realm until it has been activated. ================================= ============================================= +7.39 KVM_CAP_ARM_TLBIDOMAIN +------------------------------------- + +:Architectures: arm64 +:Target: VM +:Parameters: None +:Returns: 0 on success, negative value on error + +This capability enables TLBID virtualization, which optimizes tlbi broadcast +range to avoid unnecessary broadcasts. + +When this capability is enabled, KVM can obtain the vDomain bitmaps of VM. When +a vCPU is loaded, KVM writes the mapping between the vDomain and the pDomain +into the VTLBID(n). If guest enabled TLBID, hardware will convert vDomain id to +pDomain id for TLBI broadcast. + 8. Other capabilities. ====================== diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 07e8144d55f7..42a9927f7a4b 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -45,6 +45,20 @@ #define KVM_VCPU_MAX_FEATURES 9 #define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1) +#define NIS_RANGE_MIN_1_8 1 +#define NIS_RANGE_MAX_1_8 8 +#define NIS_RANGE_MIN_9_16 9 +#define NIS_RANGE_MAX_9_16 16 +#define VTLBID_TD_WIDTH_NIS_1_8 8 +#define VTLBID_TD_WIDTH_NIS_9_16 16 +#define VTLBID_EL2_WIDTH 64 +/* + * Cap the supported NVIS so per-vDomain bitmaps have a compile-time + * upper bound; the hardware field itself is 5 bits wide. + */ +#define KVM_VTLBID_MAX_NVIS 5 +#define KVM_VTLBID_MAX_VDOMAINS (1U << KVM_VTLBID_MAX_NVIS) + #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) #define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) @@ -57,6 +71,7 @@ #define KVM_REQ_RELOAD_TLBI_DVMBM KVM_ARCH_REQ(8) #define KVM_REQ_RELOAD_WFI_TRAPS KVM_ARCH_REQ(9) #define KVM_REQ_RELOAD_TIMER_EARLY_INJECT KVM_ARCH_REQ(10) +#define KVM_REQ_RELOAD_VTLBID KVM_ARCH_REQ(11) #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) @@ -208,6 +223,40 @@ struct kvm_protected_vm { struct kvm_hyp_memcache teardown_mc; }; +struct vcpu_tlbid_data { + /* Node in per-pCPU vCPU list of tlbidomain */ + struct list_head pcpu_node; + /* Store vDomain bitmap, indexed by vCPU ID */ + u32 vdomain_bitmap; + /* Record the last pCPU where vCPU was running, indexed by vCPU ID */ + int last_pcpu; +}; + +struct tlbidomain { + /* Using xarray to dynamically allocate vCPU-related TLBID info */ + struct xarray vcpu_data_array; + /* The number of vDomains */ + int num_domains; + /* Mapping of pDomain to vDomain, indexed by vDomain ID */ + int *domain_map; + /* Record the bitmap of pCPUs mapped to vDomain, indexed by vDomain ID */ + cpumask_var_t *vdomain_cpumasks; + /* Whether virt-TLBID is enabled by KVM */ + bool kvm_tlbid_enabled; + /* Whether virt-TLBID is enabled by Guest */ + bool guest_tlbid_enabled; + /* TLBIDIDR.NVIS */ + u8 nvis; + /* TLBIDIDR.NIS */ + u8 nis; + /* Guest-visible TLBIDIDR value (NIS/NOS carry virtual domain counts) */ + u64 tlbididr_val; + /* Used to prevent concurrent modifications to the domain mapping. */ + spinlock_t tlbid_lock; + /* Per-pCPU list of vCPUs whose last_pcpu equals this pCPU */ + struct list_head *pcpu_vcpu_list; +}; + struct kvm_arch { struct kvm_s2_mmu mmu; @@ -332,6 +381,10 @@ struct kvm_arch { KABI_EXTEND(u64 midr_el1) KABI_EXTEND(u64 revidr_el1) KABI_EXTEND(u64 aidr_el1) + +#ifdef CONFIG_KVM_ARM_VTLBID + struct tlbidomain tlbidomain; +#endif }; struct kvm_vcpu_fault_info { diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 689d32b140b7..2a71109ce082 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -145,4 +145,16 @@ config VIRT_TIMER_EARLY_INJECT The hypervisor can control the early injection latency via the module parameter: timer_early_inject_ns +config KVM_ARM_VTLBID + bool "Virtual TLBID support for KVM guests" + depends on KVM && ARM64_TLBID + default n + help + Enable virtualization of FEAT_TLBID so that each KVM guest can + use TLBI Domains independently of the host. The host TLBID + hardware is partitioned per-VM through VTLBID_EL2 registers + managed on the vCPU load/put path. + + If unsure, say N. + endif # VIRTUALIZATION diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 324553fea612..565b26248d4e 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -43,6 +43,7 @@ #include <asm/kvm_rme.h> #include <asm/sections.h> #include <asm/kvm_tmi.h> +#include <asm/tlbid.h> #include <kvm/arm_hypercalls.h> #include <kvm/arm_pmu.h> #include <kvm/arm_psci.h> @@ -319,6 +320,104 @@ static int kvm_arm_default_max_vcpus(void) return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS; } +#ifdef CONFIG_KVM_ARM_VTLBID +static int kvm_arm_init_tlbidomain(struct kvm *kvm) +{ + struct tlbidomain *vdomain = &kvm->arch.tlbidomain; + int i, max_vdomains; + u64 tlbididr; + + if (!system_supports_tlbid()) + return 0; + + tlbididr = read_sysreg_s(SYS_TLBIDIDR_EL1); + + vdomain->nis = FIELD_GET(TLBIDIDR_EL1_NIS_MASK, tlbididr); + vdomain->nvis = min_t(u8, FIELD_GET(TLBIDIDR_EL1_NVIS_MASK, tlbididr), + KVM_VTLBID_MAX_NVIS); + vdomain->num_domains = -1; + max_vdomains = 1 << vdomain->nvis; + + kvm_info("TLBIDIDR: 0x%llx, NIS: %d, NVIS: %d\n", tlbididr, vdomain->nis, vdomain->nvis); + + xa_init(&vdomain->vcpu_data_array); + + vdomain->domain_map = kcalloc(max_vdomains, sizeof(*vdomain->domain_map), + GFP_KERNEL); + if (!vdomain->domain_map) + goto destroy_vcpu_data_array; + + vdomain->vdomain_cpumasks = kcalloc(max_vdomains, sizeof(cpumask_var_t), + GFP_KERNEL); + if (!vdomain->vdomain_cpumasks) + goto free_domain_map; + + for (i = 0; i < max_vdomains; i++) { + if (!alloc_cpumask_var(&vdomain->vdomain_cpumasks[i], GFP_KERNEL)) + goto free_vdomain_cpumasks; + cpumask_clear(vdomain->vdomain_cpumasks[i]); + } + + for (i = 0; i < max_vdomains; i++) + vdomain->domain_map[i] = -1; + + spin_lock_init(&vdomain->tlbid_lock); + + vdomain->pcpu_vcpu_list = kcalloc(nr_cpu_ids, + sizeof(*vdomain->pcpu_vcpu_list), GFP_KERNEL); + if (!vdomain->pcpu_vcpu_list) + goto free_vdomain_cpumasks; + + for (i = 0; i < nr_cpu_ids; i++) + INIT_LIST_HEAD(&vdomain->pcpu_vcpu_list[i]); + + /* Enable vTLBID only after all resources are allocated. */ + vdomain->kvm_tlbid_enabled = true; + + return 0; + +free_vdomain_cpumasks: + while (--i >= 0) + free_cpumask_var(vdomain->vdomain_cpumasks[i]); + kfree(vdomain->vdomain_cpumasks); +free_domain_map: + kfree(vdomain->domain_map); +destroy_vcpu_data_array: + xa_destroy(&vdomain->vcpu_data_array); + + kvm_err("vTLBID init failed with -ENOMEM, feature disabled\n"); + return -ENOMEM; +} + +static void free_tlbid_vdomain(struct kvm *kvm) +{ + struct tlbidomain *vdomain = &kvm->arch.tlbidomain; + struct vcpu_tlbid_data *data; + int i, max_vdomains; + unsigned long index; + + if (!vdomain->kvm_tlbid_enabled) + return; + + max_vdomains = 1 << vdomain->nvis; + + kfree(vdomain->domain_map); + + for (i = 0; i < max_vdomains; i++) + free_cpumask_var(vdomain->vdomain_cpumasks[i]); + kfree(vdomain->vdomain_cpumasks); + + kfree(vdomain->pcpu_vcpu_list); + + xa_for_each(&vdomain->vcpu_data_array, index, data) + kfree(data); + xa_destroy(&vdomain->vcpu_data_array); +} +#else +static inline int kvm_arm_init_tlbidomain(struct kvm *kvm) { return 0; } +static inline void free_tlbid_vdomain(struct kvm *kvm) { } +#endif + /** * kvm_arch_init_vm - initializes a VM data structure * @kvm: pointer to the KVM struct @@ -1765,6 +1864,52 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, return kvm_reset_vcpu(vcpu); } +#ifdef CONFIG_KVM_ARM_VTLBID +static int kvm_arm_tlbidomain_vcpu_init(struct kvm_vcpu *vcpu) +{ + struct tlbidomain *vdomain; + struct vcpu_tlbid_data *data; + int ret; + + if (!system_supports_tlbid()) + return 0; + + data = kzalloc(sizeof(*data), GFP_KERNEL); + if (!data) + return -ENOMEM; + + /* + * When the host has enabled TLBID but the guest has not, + * KVM can map vDomain0 to pDomainN to optimize the TLBI + * instructions for the guest. Therefore, the value is set + * to 0x1, indicating that the vCPU is in vDomain0. If QEMU + * calls KVM_ARM_VCPU_SET_VDOMAIN to configure vdomains, this + * value will be overwritten to support the guest enabling + * TLBID. + */ + data->vdomain_bitmap = 0x1; + data->last_pcpu = -1; + INIT_LIST_HEAD(&data->pcpu_node); + + vdomain = &vcpu->kvm->arch.tlbidomain; + if (!vdomain->kvm_tlbid_enabled) { + kfree(data); + return 0; + } + + ret = xa_insert(&vdomain->vcpu_data_array, + vcpu->vcpu_idx, data, GFP_KERNEL); + if (ret) { + kfree(data); + return ret == -EBUSY ? 0 : ret; + } + + return 0; +} +#else +static inline int kvm_arm_tlbidomain_vcpu_init(struct kvm_vcpu *vcpu) { return 0; } +#endif + static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_vcpu_init *init) { @@ -1912,6 +2057,50 @@ static int kvm_arm_vcpu_rmm_psci_complete(struct kvm_vcpu *vcpu, return realm_psci_complete(vcpu, target, arg->psci_status); } +#ifdef CONFIG_KVM_ARM_VTLBID +static int kvm_arm_vcpu_set_vdomain(struct kvm_vcpu *vcpu, + struct kvm_arm_set_vdomain *vdomain) +{ + struct tlbidomain *td = &vcpu->kvm->arch.tlbidomain; + u32 vdomain_bitmap = vdomain->vdomain_bitmap; + u8 num_vdomains = vdomain->num_vdomains; + u8 nvis, max_vdomains; + struct vcpu_tlbid_data *data; + + if (!td->kvm_tlbid_enabled) + return -EINVAL; + + nvis = td->nvis; + max_vdomains = 1 << nvis; + + if (num_vdomains > max_vdomains) + return -EINVAL; + + if (td->num_domains != -1 && td->num_domains != num_vdomains) + return -EINVAL; + + data = xa_load(&td->vcpu_data_array, vcpu->vcpu_idx); + if (!data) { + kvm_err("vCPU%d: Failed to load vcpu_tlbid_data\n", vcpu->vcpu_idx); + return -EINVAL; + } + + data->vdomain_bitmap = vdomain_bitmap; + td->num_domains = num_vdomains; + td->guest_tlbid_enabled = true; + + kvm_info("vCPU%d: vDomain bitmap: 0x%x\n", vcpu->vcpu_idx, vdomain_bitmap); + + return 0; +} +#else +static inline int kvm_arm_vcpu_set_vdomain(struct kvm_vcpu *vcpu, + struct kvm_arm_set_vdomain *vdomain) +{ + return -EINVAL; +} +#endif + long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -2110,6 +2299,27 @@ static int kvm_vm_set_attr(struct kvm *kvm, struct kvm_device_attr *attr) } } +#ifdef CONFIG_KVM_ARM_VTLBID +static int kvm_arm_get_vdomain_num(struct kvm *kvm, + struct kvm_arm_get_vdomain *vdomain) +{ + struct tlbidomain *td = &kvm->arch.tlbidomain; + + if (!td->kvm_tlbid_enabled) + return -EINVAL; + + vdomain->max_vdomains = 1 << td->nvis; + + return 0; +} +#else +static inline int kvm_arm_get_vdomain_num(struct kvm *kvm, + struct kvm_arm_get_vdomain *vdomain) +{ + return -EINVAL; +} +#endif + int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { struct kvm *kvm = filp->private_data; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 18ed07a68b2d..7d1a03393523 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1315,6 +1315,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_HYGON_COCO_EXT_CSV3_SP_MGR (1 << 4) #define KVM_CAP_ARM_HW_DIRTY_STATE_TRACK 502 +#define KVM_CAP_ARM_TLBIDOMAIN 503 #define KVM_CAP_ARM_HISI_IPIV 798 #define KVM_CAP_ARM_VIRT_MSI_BYPASS 799 @@ -2358,4 +2359,17 @@ struct kvm_pre_fault_memory { __u64 padding[5]; }; +struct kvm_arm_get_vdomain { + __u8 max_vdomains; +}; + +#define KVM_ARM_GET_VDOMAIN_NUM _IOR(KVMIO, 0xd7, struct kvm_arm_get_vdomain) + +struct kvm_arm_set_vdomain { + __u32 vdomain_bitmap; + __u8 num_vdomains; +}; + +#define KVM_ARM_VCPU_SET_VDOMAIN _IOW(KVMIO, 0xd8, struct kvm_arm_set_vdomain) + #endif /* __LINUX_KVM_H */ -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- Add the VTLBID0_EL2 ~ VTLBID3_EL2 sysreg definitions and the HCRX_EL2.VTLBIDEn control field to arch/arm64/tools/sysreg. Implement kvm_tlbidomain_vcpu_load() / kvm_tlbidomain_vcpu_put() to update the VTLBIDn registers on vCPU schedule-in and tear down the hardware context on schedule-out. On first load or migration, the per-vCPU domain map is recomputed and written to VTLBIDn. Handle the KVM_REQ_RELOAD_VTLBID request so that other vCPUs reprogram the VTLBIDn registers after a domain map change. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Zhou Wang <wangzhou1@hisilicon.com> --- arch/arm64/kvm/arm.c | 240 ++++++++++++++++++++++++++++++++++++++++ arch/arm64/tools/sysreg | 19 +++- 2 files changed, 258 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 565b26248d4e..493769b4c436 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -845,6 +845,239 @@ static void update_steal_time(struct kvm_vcpu *vcpu) } #endif +#ifdef CONFIG_KVM_ARM_VTLBID +static bool is_vcpu_in_vdomain(u32 vdomain_bitmap, int vdomain_idx) +{ + return vdomain_bitmap & BIT(vdomain_idx); +} + +static int update_domain_map(struct kvm_vcpu *vcpu, int vdomain_idx, + int last_pcpu, bool clear_flag) +{ + struct tlbidomain *vdomain = &vcpu->kvm->arch.tlbidomain; + int pdomain = vdomain->domain_map[vdomain_idx]; + cpumask_var_t new_cpus; + + cpumask_copy(new_cpus, vdomain->vdomain_cpumasks[vdomain_idx]); + + if (unlikely(pdomain == -1)) { + cpumask_set_cpu(vcpu->cpu, new_cpus); + goto update_mask; + } + + if (last_pcpu != -1 && clear_flag) { + /* remove old_cpu, not need to do so when first load */ + if (cpumask_test_cpu(last_pcpu, new_cpus)) + cpumask_clear_cpu(last_pcpu, new_cpus); + } + + cpumask_set_cpu(vcpu->cpu, new_cpus); + +update_mask: + cpumask_copy(vdomain->vdomain_cpumasks[vdomain_idx], new_cpus); + return pick_best_domain(vdomain->vdomain_cpumasks[vdomain_idx]); +} + +static void vtlbidn_clear_set_s(int idx, u64 clear, u64 set) +{ + switch (idx) { + case 0: + sysreg_clear_set_s(SYS_VTLBID0_EL2, clear, set); + break; + case 1: + sysreg_clear_set_s(SYS_VTLBID1_EL2, clear, set); + break; + case 2: + sysreg_clear_set_s(SYS_VTLBID2_EL2, clear, set); + break; + case 3: + sysreg_clear_set_s(SYS_VTLBID3_EL2, clear, set); + break; + default: + WARN_ON_ONCE(1); + } +} + +static void kvm_arm_update_tlbid_map(struct tlbidomain *vdomain) +{ + int i, pdomain, vtlbid_idx, offset; + int pdomain_bits, nis = vdomain->nis; + int max_vdomains = 1 << vdomain->nvis; + u64 clear, set; + + if (nis >= NIS_RANGE_MIN_1_8 && nis <= NIS_RANGE_MAX_1_8) + pdomain_bits = VTLBID_TD_WIDTH_NIS_1_8; + else if (nis >= NIS_RANGE_MIN_9_16 && nis <= NIS_RANGE_MAX_9_16) + pdomain_bits = VTLBID_TD_WIDTH_NIS_9_16; + + for (i = 0; i < max_vdomains; i++) { + pdomain = vdomain->domain_map[i]; + if (pdomain == -1) { + if (vdomain->domain_map[0] != -1) + pdomain = vdomain->domain_map[0]; + else + pdomain = 0; + } + + vtlbid_idx = i * pdomain_bits / VTLBID_EL2_WIDTH; + offset = i * pdomain_bits % VTLBID_EL2_WIDTH; + + clear = GENMASK(offset + pdomain_bits - 1, offset); + set = pdomain << offset; + + vtlbidn_clear_set_s(vtlbid_idx, clear, set); + } + + sysreg_clear_set_s(SYS_HCRX_EL2, 0, HCRX_EL2_VTLBIDEn); +} + +static void kvm_vcpu_reload_tlbid(struct kvm *kvm) +{ + struct tlbidomain *vdomain = &kvm->arch.tlbidomain; + + if (WARN_ON_ONCE(!vdomain->kvm_tlbid_enabled)) + return; + + preempt_disable(); + kvm_arm_update_tlbid_map(vdomain); + preempt_enable(); +} + +static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + struct tlbidomain *vdomain = &kvm->arch.tlbidomain; + int max_vdomains = 1 << vdomain->nvis; + DECLARE_BITMAP(clear_bits, KVM_VTLBID_MAX_VDOMAINS); + struct vcpu_tlbid_data *current_data, *other_data; + bool domain_map_changed = false; + int i, old_map; + + /* + * If we support tlbid, but user does not enable it, we can do + * vdomain0 -> pdomainN as well by default. + */ + if (!vdomain->kvm_tlbid_enabled) + return; + + current_data = xa_load(&vdomain->vcpu_data_array, vcpu->vcpu_idx); + if (WARN_ON_ONCE(!current_data)) + return; + + /* + * Check if vCPU thread will move to another pCPU. Even when + * staying on the same pCPU, kvm_tlbidomain_vcpu_put() has + * cleared HCRX_EL2.VTLBIDEn, and another vCPU may have + * reprogrammed the per-pCPU VTLBIDn registers in between, so + * reprogram them before re-entering the guest. + */ + if (likely(vcpu->cpu == current_data->last_pcpu)) { + spin_lock(&vdomain->tlbid_lock); + kvm_arm_update_tlbid_map(vdomain); + spin_unlock(&vdomain->tlbid_lock); + return; + } + + spin_lock(&vdomain->tlbid_lock); + + bitmap_zero(clear_bits, max_vdomains); + + /* + * Remove current vCPU from its last_pcpu list and compute clear_bits + * only when there was a previous pCPU. On first load (last_pcpu == -1), + * no previous pCPU needs to be cleared from vdomain_cpumask. + */ + if (current_data->last_pcpu != -1) { + list_del_init(¤t_data->pcpu_node); + + /* + * Iterate only vCPUs that share the same last_pcpu to compute + * clear_bits. This is O(1) in the typical case since most pCPUs + * host only one vCPU at a time. + */ + bitmap_fill(clear_bits, max_vdomains); + list_for_each_entry(other_data, + &vdomain->pcpu_vcpu_list[current_data->last_pcpu], + pcpu_node) { + bitmap_andnot(clear_bits, clear_bits, + (unsigned long *)&other_data->vdomain_bitmap, max_vdomains); + } + } + + if (!vdomain->guest_tlbid_enabled) { + old_map = vdomain->domain_map[0]; + + /* + * If QEMU is not configured with vdomains, only vDomain0 needs + * to be mapped. + */ + vdomain->domain_map[0] = update_domain_map(vcpu, + 0, + current_data->last_pcpu, + test_bit(0, clear_bits)); + if (vdomain->domain_map[0] == old_map) + goto unlock; + } else { + /* + * If QEMU is configured with vdomains, all vdomains need to be + * mapped. For all vdomains which includes this vCPU, check if + * vdomain map should be changed as well. + */ + for (i = 0; i < vdomain->num_domains; i++) { + if (!is_vcpu_in_vdomain(current_data->vdomain_bitmap, i)) + continue; + + old_map = vdomain->domain_map[i]; + vdomain->domain_map[i] = update_domain_map(vcpu, + i, + current_data->last_pcpu, + test_bit(i, clear_bits)); + + if (vdomain->domain_map[i] != old_map) + domain_map_changed = true; + } + + if (!domain_map_changed) + goto unlock; + } + + kvm_flush_remote_tlbs(kvm); + + /* + * Before this vcpu load, kick other vCPUs out, so maps for other vCPUs + * will be updated during vCPU load. + * + * Add KVM_REQUEST_WAIT to make sure vCPU is out. + */ + kvm_make_all_cpus_request(kvm, KVM_REQ_RELOAD_VTLBID | KVM_REQUEST_WAIT); + +unlock: + /* update tlbid hardware map register */ + kvm_arm_update_tlbid_map(vdomain); + + current_data->last_pcpu = vcpu->cpu; + list_add_tail(¤t_data->pcpu_node, + &vdomain->pcpu_vcpu_list[vcpu->cpu]); + + spin_unlock(&vdomain->tlbid_lock); +} + +static void kvm_tlbidomain_vcpu_put(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + struct tlbidomain *vdomain = &kvm->arch.tlbidomain; + + if (!vdomain->kvm_tlbid_enabled) + return; + + sysreg_clear_set_s(SYS_HCRX_EL2, HCRX_EL2_VTLBIDEn, 0); +} +#else +static inline void kvm_vcpu_reload_tlbid(struct kvm *kvm) { } +static inline void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) { } +static inline void kvm_tlbidomain_vcpu_put(struct kvm_vcpu *vcpu) { } +#endif + void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) { struct kvm_s2_mmu *mmu; @@ -900,6 +1133,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) kvm_tlbi_dvmbm_vcpu_load(vcpu); + kvm_tlbidomain_vcpu_load(vcpu); + /* * When pv_preempted is changed from enabled to disabled, preempted * state will not be updated in kvm_arch_vcpu_put/load. So we must @@ -938,6 +1173,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_tlbi_dvmbm_vcpu_put(vcpu); + kvm_tlbidomain_vcpu_put(vcpu); + if (kvm_arm_is_pvsched_valid(&vcpu->arch) && pv_preempted_enable) kvm_update_pvsched_preempted(vcpu, 1); } @@ -1303,6 +1540,9 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_RELOAD_TLBI_DVMBM, vcpu)) kvm_hisi_reload_lsudvmbm(vcpu->kvm); + if (kvm_check_request(KVM_REQ_RELOAD_VTLBID, vcpu)) + kvm_vcpu_reload_tlbid(vcpu->kvm); + if (kvm_check_request(KVM_REQ_RELOAD_WFI_TRAPS, vcpu)) { if (single_task_running()) vcpu_clear_wfx_traps(vcpu); diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 2155315581b2..c10f6985f5db 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -3178,7 +3178,8 @@ Fields ZCR_ELx EndSysreg Sysreg HCRX_EL2 3 4 1 2 2 -Res0 63:25 +Res0 63:26 +Field 25 VTLBIDEn Field 24 PACMEn Field 23 EnFPM Field 22 GCSEn @@ -3586,3 +3587,19 @@ Field 12:8 NVIS Res0 7:5 Field 4:0 NIS EndSysreg + +Sysreg VTLBID0_EL2 3 4 2 8 0 +Field 63:0 TD +EndSysreg + +Sysreg VTLBID1_EL2 3 4 2 8 1 +Field 63:0 TD +EndSysreg + +Sysreg VTLBID2_EL2 3 4 2 8 2 +Field 63:0 TD +EndSysreg + +Sysreg VTLBID3_EL2 3 4 2 8 3 +Field 63:0 TD +EndSysreg -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- Plumb the TLBID virtualization interface into the KVM ioctl layer: - Report KVM_CAP_ARM_TLBIDOMAIN from kvm_vm_ioctl_check_extension() so that userspace can probe for the capability. - Bridge KVM_ARM_VCPU_SET_VDOMAIN and KVM_ARM_GET_VDOMAIN_NUM from kvm_arch_vcpu_ioctl() / kvm_arch_vm_ioctl() to the internal helpers, giving userspace control over per-vCPU vDomain affinity and the maximum number of vDomains. - Invoke kvm_arm_init_tlbidomain() from kvm_arch_init_vm() and free_tlbid_vdomain() from kvm_arch_destroy_vm(), so vDomain state is created when a VM is opened and released when it is destroyed. - Call kvm_arm_tlbidomain_vcpu_init() from kvm_arch_vcpu_ioctl_vcpu_init() so each vCPU gets a default vDomain0 mapping before its first run. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/arm.c | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 493769b4c436..cfd0f90ec915 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -487,6 +487,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) bitmap_zero(kvm->arch.vcpu_features, KVM_VCPU_MAX_FEATURES); + kvm_arm_init_tlbidomain(kvm); + /* Initialise the realm bits after the generic bits are enabled */ if (kvm_is_realm(kvm)) { ret = kvm_init_realm_vm(kvm); @@ -531,6 +533,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm) kvm_arm_teardown_hypercalls(kvm); kvm_destroy_realm(kvm); + free_tlbid_vdomain(kvm); } #ifdef CONFIG_ARM64_HISI_IPIV @@ -683,6 +686,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = 0; break; #endif + case KVM_CAP_ARM_TLBIDOMAIN: + if (IS_ENABLED(CONFIG_KVM_ARM_VTLBID) && system_supports_tlbid()) + r = 1; + else + r = 0; + break; case KVM_CAP_ARM_RME: r = static_key_enabled(&kvm_rme_is_available); break; @@ -2210,7 +2219,9 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, kvm_arm_pvtimer_status_set_active(vcpu, false); #endif - return 0; + ret = kvm_arm_tlbidomain_vcpu_init(vcpu); + + return ret; } static int kvm_arm_vcpu_set_attr(struct kvm_vcpu *vcpu, @@ -2474,6 +2485,17 @@ long kvm_arch_vcpu_ioctl(struct file *filp, return -EFAULT; return kvm_arm_vcpu_rmm_psci_complete(vcpu, &req); } + case KVM_ARM_VCPU_SET_VDOMAIN: { + struct kvm_arm_set_vdomain vdomain; + + if (!kvm_vcpu_initialized(vcpu)) + return -ENOEXEC; + + if (copy_from_user(&vdomain, argp, sizeof(vdomain))) + return -EFAULT; + + return kvm_arm_vcpu_set_vdomain(vcpu, &vdomain); + } default: r = -EINVAL; } @@ -2667,6 +2689,19 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) return -EFAULT; return kvm_vm_ioctl_get_reg_writable_masks(kvm, &range); } + case KVM_ARM_GET_VDOMAIN_NUM: { + struct kvm_arm_get_vdomain vdomain; + int ret; + + ret = kvm_arm_get_vdomain_num(kvm, &vdomain); + if (ret) + return ret; + + if (copy_to_user(argp, &vdomain, sizeof(vdomain))) + return -EFAULT; + + return 0; + } default: return -EINVAL; } -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- DVMBM and TLBID are mutually exclusive features. When the firmware enables DVMBM it masks the TLBID field in ID_AA64MMFR4_EL1, so the OS naturally disables TLBID. However, when the firmware enables TLBID, the DVMBM field in AIDR remains set, and the OS would incorrectly enable DVMBM on top of TLBID. Intercept this case at the software level: gate the DVMBM vCPU load hook on a new kvm_tlbid_enabled flag so that DVMBM is forced off when TLBID is active. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/hisilicon/hisi_virt.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/arm64/kvm/hisilicon/hisi_virt.c b/arch/arm64/kvm/hisilicon/hisi_virt.c index 9a04e4052abd..0be4ebbc540d 100644 --- a/arch/arm64/kvm/hisilicon/hisi_virt.c +++ b/arch/arm64/kvm/hisilicon/hisi_virt.c @@ -243,6 +243,15 @@ bool hisi_dvmbm_supported(void) return false; } + /* + * When TLBID is enabled, DVMBM cannot be enabled. + * After BIOS enables TLBID, the DVMBM field of AIDR + * remains set, therefore interception needs to be + * performed in the code. + */ + if (system_supports_tlbid()) + return false; + /* Determine whether DVMBM is supported by the hardware */ if (!(read_sysreg(aidr_el1) & AIDR_EL1_DVMBM_MASK)) return false; -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- Guest-visible TLBIDIDR_EL1 must reflect virtual domain geometry rather than the physical one, because the guest only sees vDomains. Compute a synthesized tlbididr_val at init time that remaps the NVIS and NVOS field values into the NIS and NOS bit positions. Add an access handler (access_tlbididr) that returns this synthesized value on read and silently drops writes, making the register appear invariant to the guest. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/kvm/arm.c | 7 +++++++ arch/arm64/kvm/sys_regs.c | 26 ++++++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index cfd0f90ec915..472687b4f536 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -336,6 +336,13 @@ static int kvm_arm_init_tlbidomain(struct kvm *kvm) vdomain->nvis = min_t(u8, FIELD_GET(TLBIDIDR_EL1_NVIS_MASK, tlbididr), KVM_VTLBID_MAX_NVIS); vdomain->num_domains = -1; + /* + * Guest-visible TLBIDIDR: the NIS/NOS fields carry the virtual + * domain counts (NVIS/NVOS), since the guest uses vDomains. + */ + vdomain->tlbididr_val = FIELD_PREP(TLBIDIDR_EL1_NIS_MASK, vdomain->nvis) | + FIELD_PREP(TLBIDIDR_EL1_NOS_MASK, + FIELD_GET(TLBIDIDR_EL1_NVOS_MASK, tlbididr)); max_vdomains = 1 << vdomain->nvis; kvm_info("TLBIDIDR: 0x%llx, NIS: %d, NVIS: %d\n", tlbididr, vdomain->nis, vdomain->nvis); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index fc2a1bc3c314..160c1b57f2c4 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -276,6 +276,31 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, return true; } +#ifdef CONFIG_KVM_ARM_VTLBID +static bool access_tlbididr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct tlbidomain *vdomain = &vcpu->kvm->arch.tlbidomain; + + if (p->is_write) + return ignore_write(vcpu, p); + + if (!vdomain->kvm_tlbid_enabled) + return false; + + p->regval = vdomain->tlbididr_val; + return true; +} +#else +static bool access_tlbididr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return false; +} +#endif + static bool access_actlr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -2633,6 +2658,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORN_EL1), trap_loregion }, { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_MPAMIDR_EL1), workaround_bad_mpam_abi }, + { SYS_DESC(SYS_TLBIDIDR_EL1), access_tlbididr }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, { SYS_DESC(SYS_MPAM1_EL1), workaround_bad_mpam_abi }, -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- The TLBID feature bit in ID_AA64MMFR4_EL1 is only advertised to the guest after userspace configures vDomains via KVM_ARM_VCPU_SET_VDOMAIN, which sets the guest_tlbid_enabled flag. Until then the bit is masked to prevent the guest from enabling TLBID without a valid vDomain affinity. Wire up the logic across three places: - In __kvm_read_sanitised_id_reg() (sys_regs.c): mask TLBID when !vdomain->guest_tlbid_enabled. - In kvm_update_aa64mmfr4_tlbid() (sys_regs.c): write the TLBID field into the VM-wide ID register mirror on the first VCPU_SET_VDOMAIN call, before any vCPU has run. - In kvm_arm_vcpu_set_vdomain() (arm.c): call kvm_update_aa64mmfr4_tlbid() after accepting a valid vDomain configuration. Also declare KVM_REG_ARM_ID_AA64MMFR4_EL1 in the arm64 UAPI header so userspace can read the sanitised register value via KVM_GET_ONE_REG. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/uapi/asm/kvm.h | 2 ++ arch/arm64/kvm/arm.c | 3 ++ arch/arm64/kvm/sys_regs.c | 50 ++++++++++++++++++++++++++++++- arch/arm64/kvm/sys_regs.h | 6 ++++ 4 files changed, 60 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 93de3f019e5b..99c0e43c43dd 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -253,6 +253,8 @@ struct kvm_arm_counter_offset { #define ARM64_SYS_REG(...) (__ARM64_SYS_REG(__VA_ARGS__) | KVM_REG_SIZE_U64) +#define KVM_REG_ARM_ID_AA64MMFR4_EL1 ARM64_SYS_REG(3, 0, 0, 7, 4) + /* Physical Timer EL0 Registers */ #define KVM_REG_ARM_PTIMER_CTL ARM64_SYS_REG(3, 3, 14, 2, 1) #define KVM_REG_ARM_PTIMER_CVAL ARM64_SYS_REG(3, 3, 14, 2, 2) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 472687b4f536..c87a912c5d03 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -51,6 +51,7 @@ static enum kvm_mode kvm_mode = KVM_MODE_DEFAULT; #include "hisilicon/hisi_virt.h" +#include "sys_regs.h" DEFINE_STATIC_KEY_FALSE(kvm_rme_is_available); @@ -2349,6 +2350,8 @@ static int kvm_arm_vcpu_set_vdomain(struct kvm_vcpu *vcpu, kvm_info("vCPU%d: vDomain bitmap: 0x%x\n", vcpu->vcpu_idx, vdomain_bitmap); + kvm_update_aa64mmfr4_tlbid(vcpu->kvm); + return 0; } #else diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 160c1b57f2c4..7a4e42d2a754 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1470,6 +1470,18 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu, val &= ID_AA64MMFR3_EL1_TCRX | ID_AA64MMFR3_EL1_S1POE | ID_AA64MMFR3_EL1_S1PIE; break; + case SYS_ID_AA64MMFR4_EL1: + { +#ifdef CONFIG_KVM_ARM_VTLBID + struct tlbidomain *vdomain = &vcpu->kvm->arch.tlbidomain; + + if (!vdomain->guest_tlbid_enabled) + val &= ~ID_AA64MMFR4_EL1_TLBID; +#else + val &= ~ID_AA64MMFR4_EL1_TLBID; +#endif + break; + } case SYS_ID_MMFR4_EL1: val &= ~ARM64_FEATURE_MASK(ID_MMFR4_EL1_CCIDX); break; @@ -2577,7 +2589,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { ID_WRITABLE(ID_AA64MMFR3_EL1, (ID_AA64MMFR3_EL1_TCRX | ID_AA64MMFR3_EL1_S1PIE | ID_AA64MMFR3_EL1_S1POE)), - ID_UNALLOCATED(7,4), + ID_WRITABLE(ID_AA64MMFR4_EL1, ID_AA64MMFR4_EL1_TLBID_MASK), ID_UNALLOCATED(7,5), ID_UNALLOCATED(7,6), ID_UNALLOCATED(7,7), @@ -3797,6 +3809,42 @@ const struct sys_reg_desc *get_reg_by_id(u64 id, return find_reg(¶ms, table, num); } +/* + * Update the VM's stored value for an ID register. + */ +#ifdef CONFIG_KVM_ARM_VTLBID +void kvm_update_aa64mmfr4_tlbid(struct kvm *kvm) +{ + struct tlbidomain *vdomain = &kvm->arch.tlbidomain; + u64 val, tlbid_val; + + mutex_lock(&kvm->arch.config_lock); + + if (kvm_vm_has_ran_once(kvm)) { + /* + * ID_AA64MMFR4 is a VM-level register already set during + * the first vCPU init. Skip the write to avoid triggering + * KVM_BUG_ON in kvm_set_vm_id_reg(). The per-vCPU + * vdomain_bitmap is still updated by the caller. + */ + mutex_unlock(&kvm->arch.config_lock); + return; + } + + val = kvm_read_vm_id_reg(kvm, SYS_ID_AA64MMFR4_EL1); + tlbid_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR4_EL1); + tlbid_val &= ID_AA64MMFR4_EL1_TLBID_MASK; + + val &= ~ID_AA64MMFR4_EL1_TLBID_MASK; + if (vdomain->guest_tlbid_enabled) + val |= tlbid_val; + + kvm_set_vm_id_reg(kvm, SYS_ID_AA64MMFR4_EL1, val); + + mutex_unlock(&kvm->arch.config_lock); +} +#endif + /* Decode an index value, and find the sys_reg_desc entry. */ static const struct sys_reg_desc * id_to_sys_reg_desc(struct kvm_vcpu *vcpu, u64 id, diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index 3080693719d2..8a2415921ba0 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -238,6 +238,12 @@ int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, const struct sys_reg_desc table[], unsigned int num); +#ifdef CONFIG_KVM_ARM_VTLBID +void kvm_update_aa64mmfr4_tlbid(struct kvm *kvm); +#else +static inline void kvm_update_aa64mmfr4_tlbid(struct kvm *kvm) { } +#endif + #define AA32(_x) .aarch32_map = AA32_##_x #define Op0(_x) .Op0 = _x #define Op1(_x) .Op1 = _x -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- When vCPU load enables virtual TLBID, configure HFGRTR2_EL2 to trap reads of TLBIDIDR_EL1 so they can be emulated with the synthesized, guest-appropriate value. On vCPU put, clear the trap and also clear HCRX_EL2.VTLBIDEn to restore the host view. Add the HFGRTR2_EL2 register and its nTLBIDIDR_EL1 field to arch/arm64/tools/sysreg so the new bit-field references compile. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/kvm/arm.c | 23 +++++++++++++++++++++++ arch/arm64/tools/sysreg | 4 +++- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index c87a912c5d03..d5137e8655a6 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -960,6 +960,16 @@ static void kvm_vcpu_reload_tlbid(struct kvm *kvm) preempt_enable(); } +static inline void set_tlbididr_trap(void) +{ + u64 val; + + val = read_sysreg_s(SYS_HFGRTR2_EL2); + val &= ~HFGRTR2_EL2_nTLBIDIDR_EL1_MASK; + write_sysreg_s(val, SYS_HFGRTR2_EL2); + isb(); +} + static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->kvm; @@ -977,6 +987,8 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) if (!vdomain->kvm_tlbid_enabled) return; + set_tlbididr_trap(); + current_data = xa_load(&vdomain->vcpu_data_array, vcpu->vcpu_idx); if (WARN_ON_ONCE(!current_data)) return; @@ -1079,6 +1091,16 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) spin_unlock(&vdomain->tlbid_lock); } +static inline void clear_tlbididr_trap(void) +{ + u64 val; + + val = read_sysreg_s(SYS_HFGRTR2_EL2); + val |= HFGRTR2_EL2_nTLBIDIDR_EL1_MASK; + write_sysreg_s(val, SYS_HFGRTR2_EL2); + isb(); +} + static void kvm_tlbidomain_vcpu_put(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->kvm; @@ -1087,6 +1109,7 @@ static void kvm_tlbidomain_vcpu_put(struct kvm_vcpu *vcpu) if (!vdomain->kvm_tlbid_enabled) return; + clear_tlbididr_trap(); sysreg_clear_set_s(SYS_HCRX_EL2, HCRX_EL2_VTLBIDEn, 0); } #else diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index c10f6985f5db..8896f2310d23 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -3001,7 +3001,9 @@ Field 0 nPMECR_EL1 EndSysreg Sysreg HFGRTR2_EL2 3 4 3 1 2 -Res0 63:15 +Res0 63:31 +Field 30 nTLBIDIDR_EL1 +Res0 29:15 Field 14 nACTLRALIAS_EL1 Field 13 nACTLRMASK_EL1 Field 12 nTCR2ALIAS_EL1 -- 2.33.0
virt inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10066 ---------------------------------------- Add two tracepoints to observe the vTLBID load path at runtime: kvm_tlbid_vcpu_info — logs vCPU idx, current pCPU, last pCPU, and vDomain bitmap when a vCPU is loaded. kvm_tlbid_domain_mapping — logs the vDomain ID, its vCPU cpumask, and the mapped pDomain ID for each vDomain mapping programmed during kvm_tlbidomain_vcpu_load(), covering both the default vDomain0 mapping and the per-vDomain updates. Trace events are compiled in by default and can be enabled or disabled at runtime through tracefs, keeping the overhead of an inactive tracepoint at a single branch. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/arm.c | 10 ++++++++ arch/arm64/kvm/trace_arm.h | 47 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index d5137e8655a6..1a65323425ae 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1032,6 +1032,9 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) (unsigned long *)&other_data->vdomain_bitmap, max_vdomains); } } + trace_kvm_tlbid_vcpu_info(vcpu->vcpu_idx, vcpu->cpu, + current_data->last_pcpu, + current_data->vdomain_bitmap); if (!vdomain->guest_tlbid_enabled) { old_map = vdomain->domain_map[0]; @@ -1044,6 +1047,10 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) 0, current_data->last_pcpu, test_bit(0, clear_bits)); + trace_kvm_tlbid_domain_mapping(0, + vdomain->vdomain_cpumasks[0], + vdomain->domain_map[0]); + if (vdomain->domain_map[0] == old_map) goto unlock; } else { @@ -1061,6 +1068,9 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) i, current_data->last_pcpu, test_bit(i, clear_bits)); + trace_kvm_tlbid_domain_mapping(i, + vdomain->vdomain_cpumasks[i], + vdomain->domain_map[i]); if (vdomain->domain_map[i] != old_map) domain_map_changed = true; diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index 5b91456560b5..9e739cbac942 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -408,6 +408,53 @@ TRACE_EVENT(kvm_pvspin_kick_vcpu, __entry->vcpu_id, __entry->target_vcpu_id) ); +TRACE_EVENT(kvm_tlbid_vcpu_info, + TP_PROTO(int vcpu_idx, int current_pcpu, int last_pcpu, + u32 vdomain_bitmap), + TP_ARGS(vcpu_idx, current_pcpu, last_pcpu, vdomain_bitmap), + + TP_STRUCT__entry( + __field(int, vcpu_idx) + __field(int, current_pcpu) + __field(int, last_pcpu) + __field(u32, vdomain_bitmap) + ), + + TP_fast_assign( + __entry->vcpu_idx = vcpu_idx; + __entry->current_pcpu = current_pcpu; + __entry->last_pcpu = last_pcpu; + __entry->vdomain_bitmap = vdomain_bitmap; + ), + + TP_printk("vcpu=%d current_pcpu=%d last_pcpu=%d vdomain_bitmap=0x%x", + __entry->vcpu_idx, __entry->current_pcpu, + __entry->last_pcpu, __entry->vdomain_bitmap) +); + +TRACE_EVENT(kvm_tlbid_domain_mapping, + TP_PROTO(int vdomain_id, const struct cpumask *cpumask, + int pdomain_id), + TP_ARGS(vdomain_id, cpumask, pdomain_id), + + TP_STRUCT__entry( + __field(int, vdomain_id) + __cpumask(cpumask) + __field(int, pdomain_id) + ), + + TP_fast_assign( + __entry->vdomain_id = vdomain_id; + __assign_cpumask(cpumask, cpumask_bits(cpumask)); + __entry->pdomain_id = pdomain_id; + ), + + TP_printk("Domain mapping: vDomain=%d, cpumask=%s, pDomain=%d", + __entry->vdomain_id, + __get_cpumask(cpumask), + __entry->pdomain_id) +); + #endif /* _TRACE_ARM_ARM64_KVM_H */ #undef TRACE_INCLUDE_PATH -- 2.33.0
反馈： 您发送到kernel@openeuler.org的补丁/补丁集，已成功转换为PR！ PR链接地址： https://atomgit.com/openeuler/kernel/merge_requests/29716 邮件列表地址：https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/CRQ... FeedBack: The patch(es) which you have sent to kernel@openeuler.org mailing list has been converted to a pull request successfully! Pull request link: https://atomgit.com/openeuler/kernel/merge_requests/29716 Mailing list address: https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/CRQ...
participants (2)
-
Jinqian Yang -
patchwork bot