[PATCH OLK-6.6 0/5] KVM: arm64: Directed stage-2 TLB invalidation
This series replaces KVM's broadcast stage-2 TLB invalidation with a directed one: instead of a TLBI over the whole inner-shareable domain , KVM kicks only the vCPUs of the VM via KVM_REQ_TLB_FLUSH and lets each of them locally invalidate its own stage-2 context before re-entering the guest. Jinqian Yang (5): KVM: arm64: Add non-shareable VMID TLB invalidation primitive KVM: arm64: Add infrastructure for directed stage-2 TLB invalidation KVM: arm64: Route remote TLB flushes through the request machinery KVM: arm64: Use directed flushes for stage-2 table break and unmap KVM: arm64: Exclude VMs with pass-through devices from directed flushes arch/arm64/include/asm/kvm_host.h | 23 +++++ arch/arm64/include/asm/kvm_tlbi.h | 126 +++++++++++++++++++++++++++ arch/arm64/kvm/Kconfig | 20 +++++ arch/arm64/kvm/arm.c | 67 +++++++++++++- arch/arm64/kvm/hisilicon/hisi_virt.c | 6 +- arch/arm64/kvm/hyp/pgtable.c | 52 +++++++++-- arch/arm64/kvm/hyp/vhe/tlb.c | 53 +++++++++++ arch/arm64/kvm/mmu.c | 53 +++++++++++ 8 files changed, 392 insertions(+), 8 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_tlbi.h -- 2.33.0
hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10293 ---------------------------------------- Add __kvm_tlb_flush_vmid_nsh() and its range-based companion, which locally invalidate this VM's stage-2 context on the calling pCPU instead of broadcasting over the inner-shareable domain. Without FEAT_TLBIRANGE, single pages still get an exact local invalidation and larger ranges fall back to the whole VMID. VHE-only. No functional change. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/hyp/vhe/tlb.c | 53 ++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 23325e9f3cc3..2e81d94ebd04 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -187,6 +187,59 @@ void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_nsh(struct kvm_s2_mmu *mmu) +{ + struct tlb_inv_context cxt; + + dsb(nshst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __tlbi(vmalls12e1); + dsb(nsh); + isb(); + + __tlb_switch_to_host(&cxt); +} + +void __kvm_tlb_flush_vmid_range_nsh(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages) +{ + struct tlb_inv_context cxt; + unsigned long stride; + + /* + * Without FEAT_TLBIRANGE a single page still gets an exact + * invalidation, larger ranges fall back to the whole VMID. + */ + if (pages == 1 && !system_supports_tlb_range()) { + __kvm_tlb_flush_vmid_ipa_nsh(mmu, start, 3); + return; + } + if (!system_supports_tlb_range() || pages > MAX_TLBI_RANGE_PAGES) { + __kvm_tlb_flush_vmid_nsh(mmu); + return; + } + + stride = PAGE_SIZE; + start = round_down(start, stride); + + dsb(nshst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __flush_s2_tlb_range_op(ipas2e1, start, pages, stride, 0); + + dsb(nsh); + __tlbi(vmalle1); + dsb(nsh); + isb(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; -- 2.33.0
hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10293 ---------------------------------------- Introduce CONFIG_KVM_ARM_DIRECT_TLBI. KVM_REQ_TLB_FLUSH is now consumed on arm64, with each vCPU locally invalidating its stage-2 context on guest entry; invalidation ranges are recorded per vCPU and merged, escalating to a whole-VMID flush when unmergeable. last_ran_cpu tracks the last pCPU each vCPU was loaded on so that a migration landing flushes the stage-2 context left behind by a request consumed elsewhere. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/include/asm/kvm_host.h | 11 +++ arch/arm64/include/asm/kvm_tlbi.h | 126 +++++++++++++++++++++++++++ arch/arm64/kvm/Kconfig | 20 +++++ arch/arm64/kvm/arm.c | 67 +++++++++++++- arch/arm64/kvm/hisilicon/hisi_virt.c | 6 +- 5 files changed, 228 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_tlbi.h diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 240d3db685e2..17ee1135109e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -278,6 +278,8 @@ struct kvm_arch { /* Protects VM-scoped configuration data */ struct mutex config_lock; + bool tlbi_directed; + /* * If we encounter a data abort without valid instruction syndrome * information, report this to user space. User space can (and @@ -770,6 +772,15 @@ struct kvm_vcpu_arch { u64 prod_el2; } hdbss) #endif + +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI + int last_ran_cpu; + + spinlock_t tlbi_pending_lock; + u64 tlbi_pending_start; + u64 tlbi_pending_size; + bool tlbi_pending; +#endif }; /* diff --git a/arch/arm64/include/asm/kvm_tlbi.h b/arch/arm64/include/asm/kvm_tlbi.h new file mode 100644 index 000000000000..7d5a046b6217 --- /dev/null +++ b/arch/arm64/include/asm/kvm_tlbi.h @@ -0,0 +1,126 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_TLBI_H__ +#define __ARM64_KVM_TLBI_H__ + +#include <linux/kvm_host.h> + +extern bool kvm_arm_direct_tlbi_enabled; + +#if defined(__KVM_NVHE_HYPERVISOR__) +static __always_inline void kvm_tlbi_request_flush_range(struct kvm *kvm, + u64 start, u64 size) +{ +} +static __always_inline void kvm_tlbi_request_flush(struct kvm *kvm) +{ + kvm_tlbi_request_flush_range(kvm, 0, 0); +} +#else + +#include <asm/virt.h> + +static __always_inline bool kvm_tlbi_direct_enabled(void) +{ + return IS_ENABLED(CONFIG_KVM_ARM_DIRECT_TLBI) && + READ_ONCE(kvm_arm_direct_tlbi_enabled) && + is_kernel_in_hyp_mode(); +} + +/* + * Merge a pending invalidation into every vCPU's per-vCPU pending + * range, consumed on guest entry. Ranges that can't be merged + * escalate to a whole-VMID flush (size 0). + */ +static __always_inline void kvm_tlbi_mark_pending(struct kvm *kvm, + u64 start, u64 size) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + + kvm_for_each_vcpu(i, vcpu, kvm) { + unsigned long flags; + u64 pend_end, new_end; + + spin_lock_irqsave(&vcpu->arch.tlbi_pending_lock, flags); + if (!vcpu->arch.tlbi_pending) { + vcpu->arch.tlbi_pending = true; + vcpu->arch.tlbi_pending_start = start; + vcpu->arch.tlbi_pending_size = size; + } else if (!size) { + vcpu->arch.tlbi_pending_size = 0; + } else if (vcpu->arch.tlbi_pending_size) { + pend_end = vcpu->arch.tlbi_pending_start + + vcpu->arch.tlbi_pending_size; + new_end = start + size; + + if (start <= pend_end && + new_end >= vcpu->arch.tlbi_pending_start) { + vcpu->arch.tlbi_pending_start = + min(vcpu->arch.tlbi_pending_start, start); + vcpu->arch.tlbi_pending_size = + max(pend_end, new_end) - + vcpu->arch.tlbi_pending_start; + } else { + vcpu->arch.tlbi_pending_size = 0; + } + } + spin_unlock_irqrestore(&vcpu->arch.tlbi_pending_lock, flags); + } +} + +/* Extract and clear the pending invalidation of this vCPU. */ +static __always_inline void kvm_tlbi_take_pending(struct kvm_vcpu *vcpu, + u64 *start, u64 *size) +{ + unsigned long flags; + + spin_lock_irqsave(&vcpu->arch.tlbi_pending_lock, flags); + if (vcpu->arch.tlbi_pending) { + *start = vcpu->arch.tlbi_pending_start; + *size = vcpu->arch.tlbi_pending_size; + vcpu->arch.tlbi_pending = false; + vcpu->arch.tlbi_pending_size = 0; + } else { + *start = 0; + *size = 0; + } + spin_unlock_irqrestore(&vcpu->arch.tlbi_pending_lock, flags); +} + +static __always_inline void kvm_tlbi_request_flush_range(struct kvm *kvm, + u64 start, u64 size) +{ + if (!kvm_tlbi_direct_enabled()) + return; + + smp_store_release(&kvm->arch.tlbi_directed, true); + kvm_tlbi_mark_pending(kvm, start, size); + kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH); +} + +static __always_inline void kvm_tlbi_request_flush(struct kvm *kvm) +{ + kvm_tlbi_request_flush_range(kvm, 0, 0); +} + +void __kvm_tlb_flush_vmid_nsh(struct kvm_s2_mmu *mmu); +void __kvm_tlb_flush_vmid_range_nsh(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages); + +static __always_inline void kvm_tlbi_flush_vmid_nsh(struct kvm_s2_mmu *mmu) +{ + __kvm_tlb_flush_vmid_nsh(mmu); + isb(); +} + +static __always_inline void kvm_tlbi_flush_vmid_range_nsh(struct kvm_s2_mmu *mmu, + phys_addr_t start, + unsigned long pages) +{ + __kvm_tlb_flush_vmid_range_nsh(mmu, start, pages); + isb(); +} + +#endif /* !__KVM_NVHE_HYPERVISOR__ */ + +#endif /* __ARM64_KVM_TLBI_H__ */ diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 2a71109ce082..8534dadcb257 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -157,4 +157,24 @@ config KVM_ARM_VTLBID If unsure, say N. +config KVM_ARM_DIRECT_TLBI + bool "Direct stage-2 TLB invalidation for KVM (EXPERIMENTAL)" + depends on KVM + default n + help + Replace KVM's broadcast stage-2 TLB invalidation with a + directed one: instead of broadcasting a TLBI over the whole + inner-shareable domain, KVM_REQ_TLB_FLUSH requests are sent + to the vCPUs of the VM, each of which locally invalidates its + stage-2 context before re-entering the guest. vCPUs are also + tracked per-pCPU so that migration leaves no stale entry + behind. This is profitable for small VMs running on big + machines, e.g. a 2-vCPU VM on a 128-CPU system. + + The behaviour is controlled at runtime via the + /sys/module/kvm/parameters/direct_tlbi knob and can be + toggled without rebooting: each flush is self-contained. + + If unsure, say N. + endif # VIRTUALIZATION diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index d6156bf0db08..feeba354c377 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -40,6 +40,7 @@ #include <asm/kvm_nested.h> #include <asm/kvm_pkvm.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_tlbi.h> #include <asm/kvm_rme.h> #include <asm/sections.h> #include <asm/kvm_tmi.h> @@ -73,6 +74,30 @@ bool kvm_ncsnp_support; /* Capability of DVMBM */ bool kvm_dvmbm_support; +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI +bool kvm_arm_direct_tlbi_enabled; + +static int direct_tlbi_set(const char *val, const struct kernel_param *kp) +{ + int ret = param_set_bool(val, kp); + + if (!ret && kvm_arm_direct_tlbi_enabled && !is_kernel_in_hyp_mode()) { + kvm_arm_direct_tlbi_enabled = false; + kvm_err("direct_tlbi requires VHE\n"); + return -EINVAL; + } + + return ret; +} + +static const struct kernel_param_ops direct_tlbi_ops = { + .set = direct_tlbi_set, + .get = param_get_bool, +}; + +module_param_cb(direct_tlbi, &direct_tlbi_ops, &kvm_arm_direct_tlbi_enabled, 0644); +#endif + static DEFINE_PER_CPU(unsigned char, kvm_hyp_initialized); bool is_kvm_arm_initialised(void) @@ -804,6 +829,10 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI + vcpu->arch.last_ran_cpu = -1; +#endif + err = kvm_vgic_vcpu_init(vcpu); if (err) return err; @@ -1092,7 +1121,16 @@ static void kvm_tlbidomain_vcpu_load(struct kvm_vcpu *vcpu) goto unlock; } - kvm_flush_remote_tlbs(kvm); + /* + * Don't go through the directed TLBI request path here: this + * runs under tlbid_lock, and synchronously waiting for vCPUs + * (KVM_REQUEST_WAIT) while holding a spinlock risks + * soft-lockups. Stick to the hardware broadcast. + */ + if (kvm_tlbi_direct_enabled()) + kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + else + kvm_flush_remote_tlbs(kvm); /* * Before this vcpu load, kick other vCPUs out, so maps for other vCPUs @@ -1161,6 +1199,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) *last_ran = vcpu->vcpu_id; } +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI + if (smp_load_acquire(&vcpu->kvm->arch.tlbi_directed) && + !vcpu_is_rec(vcpu) && vcpu->arch.last_ran_cpu != cpu) + kvm_tlbi_flush_vmid_nsh(mmu); + vcpu->arch.last_ran_cpu = cpu; +#endif + vcpu->cpu = cpu; kvm_vgic_load(vcpu); @@ -1217,6 +1262,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_timer_vcpu_put(vcpu); kvm_vgic_put(vcpu); +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI + vcpu->arch.last_ran_cpu = vcpu->cpu; +#endif + vcpu->cpu = -1; if (vcpu_is_rec(vcpu)) @@ -1563,6 +1612,22 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu)) return -EIO; +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI + if (kvm_check_request(KVM_REQ_TLB_FLUSH, vcpu)) { + u64 start, size; + + kvm_tlbi_take_pending(vcpu, &start, &size); + + preempt_disable(); + if (size) + kvm_tlbi_flush_vmid_range_nsh(vcpu->arch.hw_mmu, + start, size); + else + kvm_tlbi_flush_vmid_nsh(vcpu->arch.hw_mmu); + preempt_enable(); + } +#endif + if (kvm_check_request(KVM_REQ_SLEEP, vcpu)) kvm_vcpu_sleep(vcpu); diff --git a/arch/arm64/kvm/hisilicon/hisi_virt.c b/arch/arm64/kvm/hisilicon/hisi_virt.c index 0be4ebbc540d..663e9649ffa7 100644 --- a/arch/arm64/kvm/hisilicon/hisi_virt.c +++ b/arch/arm64/kvm/hisilicon/hisi_virt.c @@ -7,6 +7,7 @@ #include <linux/of.h> #include <linux/init.h> #include <linux/kvm_host.h> +#include <asm/kvm_tlbi.h> #include "hisi_virt.h" #include <linux/bitfield.h> @@ -586,7 +587,10 @@ void kvm_tlbi_dvmbm_vcpu_load(struct kvm_vcpu *vcpu) cpumask_copy(kvm->arch.sched_cpus, &mask); - kvm_flush_remote_tlbs(kvm); + if (kvm_tlbi_direct_enabled()) + kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + else + kvm_flush_remote_tlbs(kvm); /* * Re-calculate LSUDVMBM_EL2 for this VM and kick all vcpus -- 2.33.0
hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10293 ---------------------------------------- With directed TLBI enabled, kvm_arch_flush_remote_tlbs() and its range variant turn the invalidation into KVM_REQ_TLB_FLUSH requests: each vCPU locally invalidates its stage-2 context on guest entry, with running vCPUs kicked out synchronously and sleeping ones left alone. The range variant records the range per vCPU so that the consumer invalidates it precisely. The directed branches arm kvm_arch::tlbi_directed; a full broadcast clears it. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/mmu.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 74b9f2629a2e..eb16b8d04908 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -18,6 +18,7 @@ #include <asm/kvm_ras.h> #include <asm/kvm_asm.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_tlbi.h> #include <asm/virt.h> #ifdef CONFIG_HISI_VIRTCCA_CODA #include <asm/kvm_tmi.h> @@ -170,13 +171,42 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) */ int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI + /* + * Let the generic layer turn this into KVM_REQ_TLB_FLUSH + * requests: each vCPU locally invalidates its stage-2 context + * on guest entry. Realm and pKVM VMs keep the broadcast. + */ + if (kvm_tlbi_direct_enabled() && + !is_protected_kvm_enabled() && !_kvm_is_realm(kvm) && + in_task() && !irqs_disabled()) { + /* The generic layer makes the per-vCPU requests below. */ + smp_store_release(&kvm->arch.tlbi_directed, true); + return -ENOTSUPP; + } +#endif kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + /* A full-VMID broadcast leaves no stale entry on any pCPU. */ + smp_store_release(&kvm->arch.tlbi_directed, false); return 0; } int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages) { +#ifdef CONFIG_KVM_ARM_DIRECT_TLBI + /* + * vCPUs invalidate the range locally on guest entry; ranges + * that can't be merged escalate to a whole-VMID flush. + */ + if (kvm_tlbi_direct_enabled() && + !is_protected_kvm_enabled() && !_kvm_is_realm(kvm) && + in_task() && !irqs_disabled()) { + kvm_tlbi_request_flush_range(kvm, gfn << PAGE_SHIFT, + nr_pages << PAGE_SHIFT); + return 0; + } +#endif kvm_tlb_flush_vmid_range(&kvm->arch.mmu, gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT); return 0; -- 2.33.0
hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10293 ---------------------------------------- Route the stage-2 walker flushes (break-before-make, unmap and the deferred range flush) through the directed machinery: the walkers request the exact IPA or table range and each vCPU invalidates it locally on guest entry. Realm VMs keep the broadcast, as does nVHE. The full-VMID broadcast fallback clears kvm_arch::tlbi_directed, while range and IPA-scoped broadcasts leave it set. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/kvm/hyp/pgtable.c | 50 +++++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 9f4e4046c0b5..9bdbbc5ef58a 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -8,9 +8,16 @@ */ #include <linux/bitfield.h> +#include <asm/kvm_emulate.h> +#include <asm/kvm_mmu.h> #include <asm/kvm_pgtable.h> +#include <asm/kvm_tlbi.h> #include <asm/stage2_pgtable.h> +#if !defined(__KVM_NVHE_HYPERVISOR__) +#include "../trace_arm.h" +#endif + #define KVM_PTE_TYPE BIT(1) #define KVM_PTE_TYPE_BLOCK 0 @@ -672,13 +679,44 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt) return !(pgt->flags & KVM_PGTABLE_S2_NOFWB); } +/* + * Directed TLBI routes the invalidation to KVM_REQ_TLB_FLUSH; nVHE + * and every fallback below keep the hardware broadcast. + */ +static __always_inline void kvm_tlbi_flush_vmid_ipa(struct kvm_s2_mmu *mmu, + phys_addr_t ipa, int level) +{ +#if !defined(__KVM_NVHE_HYPERVISOR__) + if (kvm_tlbi_direct_enabled() && !is_protected_kvm_enabled() && + !_kvm_is_realm(kvm_s2_mmu_to_kvm(mmu)) && + in_task() && !irqs_disabled()) { + kvm_tlbi_request_flush_range(kvm_s2_mmu_to_kvm(mmu), ipa, + kvm_granule_size(level)); + return; + } +#endif + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa, level); +} + void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, size_t size) { unsigned long pages = size >> PAGE_SHIFT; +#if !defined(__KVM_NVHE_HYPERVISOR__) + if (kvm_tlbi_direct_enabled() && !is_protected_kvm_enabled() && + !_kvm_is_realm(kvm_s2_mmu_to_kvm(mmu)) && + in_task() && !irqs_disabled()) { + kvm_tlbi_request_flush_range(kvm_s2_mmu_to_kvm(mmu), addr, size); + return; + } +#endif + if (!system_supports_tlb_range() || pages > MAX_TLBI_RANGE_PAGES) { kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + /* Full VMID broadcast: the directed state is gone. */ + smp_store_release(&kvm_s2_mmu_to_kvm(mmu)->arch.tlbi_directed, + false); return; } kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, addr, pages); @@ -824,8 +862,7 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_tlb_flush_vmid_range(mmu, addr, size); } else if (kvm_pte_valid(ctx->old)) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, - ctx->addr, ctx->level); + kvm_tlbi_flush_vmid_ipa(mmu, ctx->addr, ctx->level); } } @@ -875,11 +912,12 @@ static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_clear_pte(ctx->ptep); if (kvm_pte_table(ctx->old, ctx->level)) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, - 0); + u64 size = kvm_granule_size(ctx->level); + + kvm_tlb_flush_vmid_range(mmu, ALIGN_DOWN(ctx->addr, size), + size); } else if (!stage2_unmap_defer_tlb_flush(pgt)) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, - ctx->level); + kvm_tlbi_flush_vmid_ipa(mmu, ctx->addr, ctx->level); } } -- 2.33.0
hulk inclusion category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10293 ---------------------------------------- On systems sharing stage-2 page tables with the SMMU, the SMMU's TLB is only invalidated by the DVM snooping of the broadcast TLBI, so a local-only flush would let device DMA hit freed pages. Implement kvm_arch_start_assignment() and kvm_arch_has_assigned_device() on arm64: the first VFIO device assignment sets a sticky per-VM flag that keeps such VMs on the broadcast path. Device assignment without KVM's involvement is not visible and relies on the current model where the SMMU driver maintains its own stage-2. Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com> --- arch/arm64/include/asm/kvm_host.h | 12 ++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 2 ++ arch/arm64/kvm/mmu.c | 23 +++++++++++++++++++++++ 3 files changed, 37 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 17ee1135109e..d5d165dd8af8 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -280,6 +280,13 @@ struct kvm_arch { bool tlbi_directed; + /* + * Sticky once a pass-through (VFIO) device is attached: on + * systems sharing stage-2 with the SMMU its TLB relies on DVM + * snooping of the broadcast TLBI, so such VMs stay broadcast. + */ + bool has_vfio_devices; + /* * If we encounter a data abort without valid instruction syndrome * information, report this to user space. User space can (and @@ -1449,6 +1456,11 @@ struct kvm *kvm_arch_alloc_vm(void); #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE +#define __KVM_HAVE_ARCH_ASSIGNED_DEVICE +void kvm_arch_start_assignment(struct kvm *kvm); +void kvm_arch_end_assignment(struct kvm *kvm); +bool kvm_arch_has_assigned_device(struct kvm *kvm); + static inline bool kvm_vm_is_protected(struct kvm *kvm) { return false; diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 9bdbbc5ef58a..920fc8f271fe 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -689,6 +689,7 @@ static __always_inline void kvm_tlbi_flush_vmid_ipa(struct kvm_s2_mmu *mmu, #if !defined(__KVM_NVHE_HYPERVISOR__) if (kvm_tlbi_direct_enabled() && !is_protected_kvm_enabled() && !_kvm_is_realm(kvm_s2_mmu_to_kvm(mmu)) && + !kvm_arch_has_assigned_device(kvm_s2_mmu_to_kvm(mmu)) && in_task() && !irqs_disabled()) { kvm_tlbi_request_flush_range(kvm_s2_mmu_to_kvm(mmu), ipa, kvm_granule_size(level)); @@ -706,6 +707,7 @@ void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, #if !defined(__KVM_NVHE_HYPERVISOR__) if (kvm_tlbi_direct_enabled() && !is_protected_kvm_enabled() && !_kvm_is_realm(kvm_s2_mmu_to_kvm(mmu)) && + !kvm_arch_has_assigned_device(kvm_s2_mmu_to_kvm(mmu)) && in_task() && !irqs_disabled()) { kvm_tlbi_request_flush_range(kvm_s2_mmu_to_kvm(mmu), addr, size); return; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index eb16b8d04908..e09e64e54bb5 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -163,6 +163,27 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) return memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY); } +/* + * The SMMU sharing this VM's stage-2 page tables invalidates its TLB + * via the DVM snooping of the broadcast TLBI: a directed (local-only) + * flush would leave stale entries and let device DMA hit freed pages. + * The flag is sticky: the broadcast path is always safe. + */ +void kvm_arch_start_assignment(struct kvm *kvm) +{ + WRITE_ONCE(kvm->arch.has_vfio_devices, true); +} + +void kvm_arch_end_assignment(struct kvm *kvm) +{ + /* Sticky: nothing to do, see above. */ +} + +bool kvm_arch_has_assigned_device(struct kvm *kvm) +{ + return READ_ONCE(kvm->arch.has_vfio_devices); +} + /** * kvm_arch_flush_remote_tlbs() - flush all VM TLB entries for v7/8 * @kvm: pointer to kvm structure. @@ -179,6 +200,7 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm) */ if (kvm_tlbi_direct_enabled() && !is_protected_kvm_enabled() && !_kvm_is_realm(kvm) && + !kvm_arch_has_assigned_device(kvm) && in_task() && !irqs_disabled()) { /* The generic layer makes the per-vCPU requests below. */ smp_store_release(&kvm->arch.tlbi_directed, true); @@ -201,6 +223,7 @@ int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, */ if (kvm_tlbi_direct_enabled() && !is_protected_kvm_enabled() && !_kvm_is_realm(kvm) && + !kvm_arch_has_assigned_device(kvm) && in_task() && !irqs_disabled()) { kvm_tlbi_request_flush_range(kvm, gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT); -- 2.33.0
反馈： 您发送到kernel@openeuler.org的补丁/补丁集，已成功转换为PR！ PR链接地址： https://atomgit.com/openeuler/kernel/merge_requests/30652 邮件列表地址：https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/GAG... FeedBack: The patch(es) which you have sent to kernel@openeuler.org mailing list has been converted to a pull request successfully! Pull request link: https://atomgit.com/openeuler/kernel/merge_requests/30652 Mailing list address: https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/GAG...
participants (2)
-
Jinqian Yang -
patchwork bot