From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org> mainline inclusion from mainline-v7.4-rc1 commit 34cfa9b3af17d0554ffd6c655fc1097854b49487 category: feature bugzilla: https://atomgit.com/openeuler/kernel/issues/10374 CVE: NA Reference: https://github.com/torvalds/linux/commit/34cfa9b3af17d0554ffd6c655fc1097854b... -------------------------------- commit 34cfa9b3af17d0554ffd6c655fc1097854b49487 upstream Implement KVM stage 2 page table pre-faulting for the arm64 architecture, with the core of the implementation in kvm_arch_vcpu_pre_fault_memory(). Pre-fault by first trying a page table walk under the MMU read lock then, if it fails, injecting a synthetic data abort at the page table level at which the page table walk failed. This is necessarily racey as reclaim might happen at any time. Successfully pre-faulting can therefore only guarantee that each GPA was observed to be mapped at least once. Protected KVM (pKVM) is not supported at all because pKVM creates VMs and vCPUs when first run, meaning any attempt to pre-fault prior to this cannot succeed. Additionally, in pKVM mode, the stage 2 page tables are owned by the hypervisor rather than the host - the host can neither walk them nor populate them directly, so it's not clear that the pre-fault mechanism correctly maps onto pKVM. Reject both pKVM hosts and uninitialised vCPUs, which are not in a safe state to be loaded, from the kvm_arch_pre_fault_allowed() hook so they are refused before the vCPU is loaded. A retry mechanic is implemented when user_mem_abort() or gmem_abort() fail to map memory due to a benign failure where a hardware abort would not result in an error. An invalid memslot (i.e. a memslot with the KVM_MEMSLOT_INVALID flag set) results in the operation returning -EAGAIN without a retry mechanic, because an invalid memslot means the pre-fault operation raced with memslot reclaim, and since the SRCU lock is held, progress cannot be made, and the user must retry. This work is based with gratitude on Jack Thomson's original series, its previous revisions and the feedback they received. [Backport note] Adapted for OLK-6.6: - OLK-6.6 has no gmem_abort() or kvm_s2_fault_map(). The pre-fault path calls the existing user_mem_abort() with a synthetic ESR and the canonical kvm_s2_mmu in kvm_s2_fault_desc, plus a kvm_s2_fault_result, which is the equivalent of the upstream s2fd plumbing. - Only the non-gmem user_mem_abort() path exists in this tree, so the gmem handling is omitted. - kvm_pgtable_get_leaf() takes a u32 level and the shared walk flag was added by the previous patch; kvm_walk_s2() uses it under the MMU read lock. - ESR_ELx_FSC_FAULT_L() is not defined in OLK-6.6 and was added to esr.h for the synthetic abort encoding. - The hook also rejects pKVM, PVM and CCA realms, which do not use the canonical stage-2 page tables targeted by this implementation. This work is based with gratitude on Jack Thomson's original series, its previous revisions and the feedback they received. Link: https://patch.msgid.link/20260612162354.73378-1-jackabt.amazon@gmail.com/ Reviewed-by: Oliver Upton <oupton@kernel.org> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260923-kvm-arm-prefault-v4-10-d4b0b4dfa8c3@kernel... Signed-off-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Yanan Wang <wangyanan55@huawei.com> --- arch/arm64/include/asm/esr.h | 1 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 3 + arch/arm64/kvm/mmu.c | 140 +++++++++++++++++++++++++++++++++++ 4 files changed, 145 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index eb32ec325cac..f4b88197e4ce 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -120,6 +120,7 @@ #define ESR_ELx_FSC_FAULT_L1 (0x05) #define ESR_ELx_FSC_FAULT_L2 (0x06) #define ESR_ELx_FSC_FAULT_L3 (0x07) +#define ESR_ELx_FSC_FAULT_L(level) (ESR_ELx_FSC_FAULT + (level)) #define ESR_ELx_FSC_PERM (0x0C) #define ESR_ELx_FSC_SEA_TTW0 (0x14) #define ESR_ELx_FSC_SEA_TTW1 (0x15) diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 2a71109ce082..5669e68e9af7 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -28,6 +28,7 @@ menuconfig KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT + select KVM_GENERIC_PRE_FAULT_MEMORY select KVM_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_EVENTFD diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 978b0a3f2794..17632496ed30 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -606,6 +606,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_COUNTER_OFFSET: r = !kvm_is_realm(kvm) && !is_pvm_mode(); break; + case KVM_CAP_PRE_FAULT_MEMORY: + r = 1; + break; case KVM_CAP_SET_GUEST_DEBUG2: return is_pvm_mode() ? 0 : KVM_GUESTDBG_VALID_MASK; case KVM_CAP_ARM_SET_DEVICE_ADDR: diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 98691166687d..653639c8debe 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2738,3 +2738,143 @@ int kvm_mmu_mark_touched_log(struct kvm *kvm) return 0; } + +/* + * Try to walk to the specified GPA in canonical mmu - if unmapped returns 0, if + * mapped returns the granule size, otherwise returns an error. + */ +static long kvm_walk_s2(struct kvm_pgtable *pgt, gpa_t gpa, u32 *level) +{ + struct kvm *kvm = kvm_s2_mmu_to_kvm(pgt->mmu); + kvm_pte_t pte; + long ret; + + read_lock(&kvm->mmu_lock); + ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, level, + KVM_PGTABLE_WALK_SHARED); + read_unlock(&kvm->mmu_lock); + if (ret) + return ret; + /* Unpopulated, must fault. */ + if (!kvm_pte_valid(pte)) + return 0; + return kvm_granule_size(*level); +} + +/* Synthesised data abort at specified page table level. */ +#define PRE_FAULT_ESR(level) \ + ((ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | \ + ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(level)) + +/* Retrieve either a read-only or a read/write hva. */ +static hva_t gfn_to_hva_memslot_read(struct kvm_memory_slot *slot, gfn_t gfn) +{ + return gfn_to_hva_memslot_prot(slot, gfn, /*writable=*/NULL); +} + +static long __pre_fault_s2(struct kvm_s2_mmu *mmu, struct kvm_vcpu *vcpu, + gpa_t gpa, struct kvm_memory_slot *memslot, u32 level) +{ + const gfn_t gfn = gpa_to_gfn(gpa); + const hva_t hva = gfn_to_hva_memslot_read(memslot, gfn); + struct kvm_s2_fault_result result = {}; + unsigned long esr = PRE_FAULT_ESR(level); + long ret; + + if (kvm_is_error_hva(hva)) + return -EFAULT; + + const struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .memslot = memslot, + .hva = hva, + .esr = esr, + .mmu = mmu, + }; + + ret = user_mem_abort(&s2fd, NULL, ESR_ELx_FSC_FAULT, 0, 0, &result); + if (IS_ERR_VALUE(ret)) + return ret; + return result.mapping_size; +} + +static long pre_fault_s2(struct kvm_s2_mmu *mmu, struct kvm_vcpu *vcpu, + gpa_t gpa, struct kvm_memory_slot *memslot) +{ + u32 level; + long ret; + + /* Try a walk first. */ + ret = kvm_walk_s2(mmu->pgt, gpa, &level); + if (ret) + return ret; + /* OK, have to fault page in. */ + return __pre_fault_s2(mmu, vcpu, gpa, memslot, level); +} + +static unsigned long +pre_fault_bytes_consumed(gpa_t gpa, unsigned long granule_size, + unsigned long bytes_remaining) +{ + /* Granules are always a power-of-2. */ + const unsigned long granule_bytes_remaining = + granule_size - (gpa % granule_size); + + return min(granule_bytes_remaining, bytes_remaining); +} + +/* If you lose the race this many times, time to give up. */ +#define MAX_PRE_FAULT_RETRIES 3 + +int kvm_arch_pre_fault_allowed(struct kvm_vcpu *vcpu) +{ + if (is_protected_kvm_enabled() || is_pvm_mode() || + _kvm_is_realm(vcpu->kvm)) + return -EOPNOTSUPP; + if (!kvm_vcpu_initialized(vcpu)) + return -ENOEXEC; + + return 0; +} + +/** + * kvm_arch_vcpu_pre_fault_memory - pre-fault stage-2 page tables for the + * specified GPA. + * @vcpu: The VCPU pointer + * @range: {gpa, size, flags} tuple + * + * The mapping performed is always best-effort - faulting in is necessarily + * racey. The ranges faulted in are canonical, nested page tables are ignored. + * + * @range->gpa specifies the GPA to pre-fault, @range->size specifies how many + * bytes remain to be pre-faulted and @range->flags is reserved and must be 0. + * + * Returns: the number of bytes the pre-fault consumed, or an error. + */ +long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, + struct kvm_pre_fault_memory *range) +{ + struct kvm *kvm = vcpu->kvm; + const u64 bytes_remaining = range->size; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; /* Canonical. */ + struct kvm_memory_slot *memslot; + const gpa_t gpa = range->gpa; + int num_retries = 0; + long ret; + + memslot = gfn_to_memslot(kvm, gpa_to_gfn(gpa)); + if (!memslot) + return -ENOENT; + /* SRCU must be released for progress and only userland can do that. */ + if (memslot->flags & KVM_MEMSLOT_INVALID) + return -EAGAIN; + + do { + ret = pre_fault_s2(mmu, vcpu, gpa, memslot); + } while (ret == -EAGAIN && num_retries++ < MAX_PRE_FAULT_RETRIES); + + if (IS_ERR_VALUE(ret)) + return ret; + return pre_fault_bytes_consumed(gpa, ret, bytes_remaining); +} -- 2.43.0