[PATCH v5 0/8] Support the FEAT_HDBSS introduced in Armv9.5
This series of patches add support to the Hardware Dirty state tracking Structure (HDBSS) feature, which is introduced by the ARM architecture in the DDI0601 (ID121123) version. The HDBSS feature is an extension to the architecture that enhances tracking translation table descriptors' dirty state, identified as FEAT_HDBSS. This feature utilizes hardware assistance to achieve dirty page tracking, aiming to significantly reduce the overhead of scanning for dirty pages. The purpose of this feature is to make the execution overhead of live migration lower to both the guest and the host, compared to existing approaches (write-protect or search stage-2 tables). The required sysreg definitions for FEAT_HDBSS have been merged into arm64 /sysregs: [1/5] arm64/sysreg: Add HDBSS related register information https://git.kernel.org/arm64/c/72f7be0c2e30 After these patches, the kernel automatically enables HDBSS when dirty logging is enabled on any memslot, and disables HDBSS when dirty logging is disabled on all memslots. This series supports both dirty-bitmap mode and dirty ring mode. v4: https://lore.kernel.org/linux-arm-kernel/20260709104026.2612599-1-zhengtian1... v4->v5 changes: - Drop Leonardo Bras' "Enable eager hugepage splitting if HDBSS is available" patch and the dependency on it. v5 supports lazy split natively: stage2_attr_walker() strips DBM from block entries (level < KVM_PGTABLE_LAST_LEVEL) during write-protect, forcing blocks into the NW state so a guest write generates a permission fault that triggers the existing lazy split path to break the block down to level-3 pages. Level-3 pages keep DBM, staying in WC state so the first guest write after write-protect marks the page dirty automatically. - Rework the DBM dirty tracking into a three-state model (NW/WC/WD). v4 set DBM conditionally through the KVM_PGTABLE_S2_DBM page-table flag; v5 sets DBM unconditionally on all writable pages and relies on VTCR_EL2.HD to decide whether hardware manages the WC->WD transition. This removes the KVM_PGTABLE_S2_DBM flag definition. Block DBM stripping is added per the point above. - Decouple the HDBSS buffer lifecycle from dirty logging state: buffers are pre-allocated at vCPU creation time (when hardware supports HDBSS) and freed at vCPU destruction, instead of being allocated on enable and freed on disable. This eliminates cross-CPU state mutation during enable/disable and removes use-after-free risks. The per-VM enable_hdbss flag and hdbss_order are replaced by a single hdbss_buffer_size (in bytes); the buffer base_phys field is replaced by hdbss_pg plus buddy_order. A kvm_hdbss_buffer_size() helper falls back to the default (PAGE_SIZE) when unset. - Use a single HDBSS buffer flush point: the buffer is flushed on every VM exit in kvm_arch_vcpu_ioctl_run(), before handling the exit reason. This replaces the KVM_REQ_FLUSH_HDBSS request used in v4 (on vcpu_put, check_vcpu_requests and sync_dirty_log), ensuring entries pushed to the dirty ring are accounted for when check_vcpu_requests() runs dirty_ring_check_request() on the next loop iteration, preventing ring overflow. - Protect the flush with SRCU instead of mmu_lock. Flush can be called from kvm_arch_vcpu_ioctl_run() while holding the scheduler rq->lock, so taking mmu_lock there would risk an ABBA deadlock. - Enable/disable now touches VTCR only: it modifies the VTCR_EL2 bits and kicks all vCPUs out of guest mode, without allocating or freeing buffers. sync_dirty_log() uses kvm_for_each_vcpu() + kvm_vcpu_kick() instead of kvm_make_all_cpus_request(KVM_REQ_FLUSH_HDBSS), relying on the per-VM-exit flush to drain the buffer. - Add nested-virtualization-aware helpers: kvm_hdbss_enabled() tests kvm->arch.mmu.vtcr (L1 only), and vcpu_hdbss_enabled() tests hw_mmu->vtcr, filtering out vCPUs running a nested (L2) guest whose shadow stage-2 has no HDBSS bits. - Add KVM_CAP_ARM_HDBSS_BUFFER_SIZE (251) to let userspace configure the per-VM HDBSS buffer size before vCPUs are created. The size is specified in bytes via KVM_ENABLE_CAP, must be a power of two in the range [PAGE_SIZE, SZ_2M], and can only be set before any vCPU is created. KVM_CHECK_EXTENSION returns the configured size for a VM, or the maximum supported size (SZ_2M) when queried globally. - Add dirty ring mode support. v4 rejected dirty ring mode; v5 allows HDBSS to coexist with the dirty ring. In dirty ring mode the buffer is fixed at PAGE_SIZE to keep the reserved entries low and the soft_limit high, maximizing the drain interval, so userspace cannot override it via KVM_CAP_ARM_HDBSS_BUFFER_SIZE. A new generic kvm_arch_dirty_ring_size_updated() hook (called after dirty_ring_size is set) is overridden by arm64 to auto-configure the buffer, and kvm_cpu_dirty_log_size() reports the buffer entry count so the dirty ring framework computes correct rsvd/soft_limit values. - Add a KVM: arm64: Document HDBSS buffer size ioctl patch documenting the new capability in Documentation/virt/kvm/api.rst. - Add WARN_ON_ONCE guards for !system_supports_hdbss() and !vcpu_hdbss_enabled() in the fault handler to defend against non-VHE and nested contexts, and add a min_t() bounds check in the flush loop. Tian Zheng (6): KVM: arm64: Add support for FEAT_HDBSS KVM: arm64: Add auto DBM support for hardware dirty tracking KVM: arm64: Add auto HDBSS enable/disable on dirty logging change KVM: arm64: Add HDBSS buffer size ioctl for dirty-bitmap mode KVM: arm64: Support HDBSS with dirty ring mode KVM: arm64: Document HDBSS buffer size ioctl eillon (2): KVM: arm64: Add HDBSS per-vCPU buffer management KVM: arm64: Add HDBSS fault handling and buffer flush Documentation/virt/kvm/api.rst | 29 ++++ arch/arm64/include/asm/cpufeature.h | 5 + arch/arm64/include/asm/esr.h | 5 + arch/arm64/include/asm/kvm_dirty_bit.h | 75 +++++++++++ arch/arm64/include/asm/kvm_host.h | 13 ++ arch/arm64/include/asm/kvm_pgtable.h | 2 + arch/arm64/include/asm/sysreg.h | 9 ++ arch/arm64/kernel/cpufeature.c | 12 ++ arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/arm.c | 103 +++++++++++++- arch/arm64/kvm/dirty_bit.c | 179 +++++++++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 36 ++++- arch/arm64/kvm/hyp/vhe/switch.c | 16 +++ arch/arm64/kvm/mmu.c | 15 +++ arch/arm64/kvm/reset.c | 3 + arch/arm64/tools/cpucaps | 1 + include/linux/kvm_dirty_ring.h | 1 + include/uapi/linux/kvm.h | 1 + virt/kvm/dirty_ring.c | 4 + virt/kvm/kvm_main.c | 1 + 20 files changed, 507 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_dirty_bit.h create mode 100644 arch/arm64/kvm/dirty_bit.c base-commit: a4ff2be345d0abc943da8dd8da98151843b750dc -- 2.33.0
Armv9.5 introduces the Hardware Dirty Bit State Structure (HDBSS) feature, indicated by ID_AA64MMFR1_EL1.HAFDBS == 0b0100. Add CPU capability detection for HDBSS. The feature requires VHE mode and is detected via ID_AA64MMFR1_EL1.HAFDBS.HDBSS field. A helper function system_supports_hdbss() is provided to check hardware support. Suggested-by: Zhou Wang <wangzhou1@hisilicon.com> Co-developed-by: Eillon <yezhenyu2@huawei.com> Signed-off-by: Eillon <yezhenyu2@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/asm/cpufeature.h | 5 +++++ arch/arm64/kernel/cpufeature.c | 12 ++++++++++++ arch/arm64/tools/cpucaps | 1 + 3 files changed, 18 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index d90040fb9de6..4c48e0fef53b 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -863,6 +863,11 @@ static inline bool system_supports_haft(void) return cpus_have_final_cap(ARM64_HAFT); } +static inline bool system_supports_hdbss(void) +{ + return cpus_have_final_cap(ARM64_HAS_HDBSS); +} + static __always_inline bool system_supports_mpam(void) { return alternative_has_cap_unlikely(ARM64_MPAM); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 4025d51d2d93..2ce49164d811 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2131,6 +2131,11 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } +static bool has_vhe_hdbss(const struct arm64_cpu_capabilities *entry, int scope) +{ + return is_kernel_in_hyp_mode() && has_cpuid_feature(entry, scope); +} + bool cpu_supports_bbml3(void) { /* CPUs that support BBML3 but dont advertise through ID_AA64MMFR2_EL1 */ @@ -2775,6 +2780,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, HAFT) }, #endif + { + .desc = "Hardware Dirty state tracking structure (HDBSS)", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HAS_HDBSS, + .matches = has_vhe_hdbss, + ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, HDBSS) + }, { .desc = "CRC32 instructions", .capability = ARM64_HAS_CRC32, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 0ee42b0cc8d3..b12268bdf1bd 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -69,6 +69,7 @@ HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT HAS_XNX +HAS_HDBSS HAFT HW_DBM KVM_HVHE -- 2.33.0
The DBM (Dirty Bit Modifier) attribute, introduced in ARMv8.1, enables hardware to automatically promote write-clean pages to write-dirty. This prevents the guest from being trapped in EL2 due to missing write permissions. Implement a three-state dirty tracking model for stage-2 page tables where DBM is unconditionally set on all writable pages: State DBM S2AP[1] Meaning ---------- ---- ------- ------------------------------------ NW 0 0 Non-writable, write -> permission fault WC 1 0 Writable-clean, write -> hw sets WD WD 1 1 Writable-dirty, no fault When VTCR_EL2.HD=1 (HDBSS active), hardware manages the WC->WD transition automatically. When HD=0, DBM has no hardware effect and dirty tracking falls back to software. stage2_set_prot_attr() and kvm_pgtable_stage2_relax_perms() set DBM unconditionally on writable pages. Pages not participating in dirty logging are naturally filtered during buffer processing by mark_page_dirty_in_slot(). For dirty logging write-protect, stage2_attr_walker() strips DBM from block entries (level < KVM_PGTABLE_LAST_LEVEL), forcing them into state NW. This is necessary because lazy splitting depends on permission faults to break down blocks to level-3: with DBM=1 on a block, a guest write transitions it from WC to WD (via hardware WC->WD transition) without faulting, so the block is never split and per-page dirty tracking is lost. Level-3 pages keep DBM, staying in state WC so the first guest write after write-protect marks them dirty automatically (WC->WD). Co-developed-by: Eillon <yezhenyu2@huawei.com> Signed-off-by: Eillon <yezhenyu2@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/asm/kvm_pgtable.h | 2 ++ arch/arm64/kvm/hyp/pgtable.c | 36 ++++++++++++++++++++++++++-- 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 41a8687938eb..6d751f23b23a 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -93,6 +93,8 @@ typedef u64 kvm_pte_t; #define KVM_PTE_LEAF_ATTR_HI_S2_XN GENMASK(54, 53) +#define KVM_PTE_LEAF_ATTR_HI_S2_DBM BIT(51) + #define KVM_PTE_LEAF_ATTR_HI_S1_GP BIT(50) #define KVM_PTE_LEAF_ATTR_S2_PERMS (KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R | \ diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b74dd5ce1efd..d55010430fd5 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -731,9 +731,24 @@ static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot p if (prot & KVM_PGTABLE_PROT_R) attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R; - if (prot & KVM_PGTABLE_PROT_W) + if (prot & KVM_PGTABLE_PROT_W) { attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W; + /* + * Set DBM bit unconditionally for all writable pages as + * part of the three-state dirty tracking model. When + * VTCR_EL2.HD=1 (HDBSS enabled), hardware manages the + * WC->WD transition automatically. When HD=0, DBM has no + * hardware effect and dirty state is managed by software. + * + * Pages that do not participate in dirty logging are + * naturally filtered during buffer processing by + * mark_page_dirty_in_slot() which only marks pages covered + * by an active dirty-logging memslot. + */ + attr |= KVM_PTE_LEAF_ATTR_HI_S2_DBM; + } + if (!kvm_lpa2_is_enabled()) attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh); @@ -1234,6 +1249,21 @@ static int stage2_attr_walker(const struct kvm_pgtable_visit_ctx *ctx, pte &= ~data->attr_clr; pte |= data->attr_set; + /* + * Strip DBM from block entries (level < LAST_LEVEL) during + * write-protect. With DBM=1, a writable block entry is in state + * WC and a guest write directly transitions it to WD (WC->WD) + * without generating a permission fault. Since lazy splitting + * relies on permission faults to break down blocks into level-3 + * pages, leaving DBM on blocks would prevent them from ever + * being split, making per-page dirty tracking impossible. + * Level-3 pages keep DBM to stay in state WC, so the first guest + * write after write-protect marks the page dirty automatically + * (WC->WD). + */ + if (ctx->level < KVM_PGTABLE_LAST_LEVEL) + pte &= ~KVM_PTE_LEAF_ATTR_HI_S2_DBM; + /* * We may race with the CPU trying to set the access flag here, * but worst-case the access flag update gets lost and will be @@ -1367,8 +1397,10 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, if (prot & KVM_PGTABLE_PROT_R) set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R; - if (prot & KVM_PGTABLE_PROT_W) + if (prot & KVM_PGTABLE_PROT_W) { set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W; + set |= KVM_PTE_LEAF_ATTR_HI_S2_DBM; + } if (prot & KVM_PGTABLE_PROT_X) { ret = stage2_set_xn_attr(prot, &xn); -- 2.33.0
From: eillon <yezhenyu2@huawei.com> Add HDBSS (Hardware Dirty Bit State Structure) per-vCPU buffer management including allocation, freeing, and loading of HDBSS registers during vCPU load. The buffer is allocated at vCPU creation time and freed at vCPU destruction. It is always allocated when hardware supports HDBSS, regardless of whether dirty logging is currently enabled, which avoids cross-CPU state mutation during enable/disable operations and eliminates use-after-free risks. The buffer size defaults to one page per vCPU, determined by the per-VM hdbss_buffer_size field. A vcpu_hdbss_enabled() helper checks whether HDBSS is active for the current vCPU by testing hw_mmu->vtcr, correctly handling nested virtualization where the shadow stage-2 MMU does not have HDBSS bits set. On vCPU load, __load_hdbss() writes HDBSSBR_EL2 and HDBSSPROD_EL2 into the hardware registers. On vCPU put, HDBSSPROD_EL2 is saved back to allow cross-CPU buffer access. Signed-off-by: Eillon <yezhenyu2@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/asm/kvm_dirty_bit.h | 51 +++++++++++++++++++++++++ arch/arm64/include/asm/kvm_host.h | 13 +++++++ arch/arm64/include/asm/sysreg.h | 9 +++++ arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/arm.c | 14 ++++++- arch/arm64/kvm/dirty_bit.c | 52 ++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/switch.c | 16 ++++++++ arch/arm64/kvm/reset.c | 3 ++ 8 files changed, 157 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_dirty_bit.h create mode 100644 arch/arm64/kvm/dirty_bit.c diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h new file mode 100644 index 000000000000..2a7de96775ef --- /dev/null +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2026 Huawei Technologies Co., Ltd + * Author: Tian Zheng <zhengtian10@huawei.com> + */ + +#ifndef __ARM64_KVM_DIRTY_BIT_H__ +#define __ARM64_KVM_DIRTY_BIT_H__ + +#include <asm/kvm_pgtable.h> +#include <asm/sysreg.h> +#include <linux/sizes.h> + +/* Default HDBSS buffer size: one full page per vCPU */ +#define KVM_ARM_HDBSS_DEFAULT_SIZE PAGE_SIZE + +/* Maximum HDBSS buffer size: 2MB */ +#define KVM_ARM_HDBSS_MAX_SIZE SZ_2M + +/* + * Returns the effective HDBSS buffer size for the VM. Returns the + * user-configured value if set, or the default PAGE_SIZE otherwise. + * Used by alloc, flush and dirty_log_size to avoid open-coding the + * fallback in multiple call sites. + */ +static inline u32 kvm_hdbss_buffer_size(struct kvm *kvm) +{ + return kvm->arch.hdbss_buffer_size ?: KVM_ARM_HDBSS_DEFAULT_SIZE; +} + +/* + * vCPU-level HDBSS check -- filters both L1 and L2. + * + * Tests hw_mmu->vtcr, the S2 MMU actually in use for this vCPU's + * current execution context. When running L1, hw_mmu points to + * kvm->arch.mmu (HDBSS bits present). When running L2, hw_mmu points + * to a shadow S2 MMU (HDBSS bits absent). + * + * Use in all per-vCPU paths that directly touch HDBSS registers or + * buffer (load, put, flush, fault handler). + */ +static inline bool vcpu_hdbss_enabled(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.hw_mmu && + (vcpu->arch.hw_mmu->vtcr & VTCR_EL2_HDBSS); +} + +int kvm_arm_vcpu_alloc_hdbss(struct kvm_vcpu *vcpu); +void kvm_arm_vcpu_free_hdbss(struct kvm_vcpu *vcpu); + +#endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index bae2c4f92ef5..9cc23a1a9693 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -420,6 +420,9 @@ struct kvm_arch { */ struct kvm_protected_vm pkvm; + /* HDBSS: per-VM buffer size in bytes (0 = not configured, use default) */ + u32 hdbss_buffer_size; + #ifdef CONFIG_PTDUMP_STAGE2_DEBUGFS /* Nested virtualization info */ struct dentry *debugfs_nv_dentry; @@ -838,6 +841,13 @@ struct vcpu_reset_state { bool reset; }; +struct vcpu_hdbss_state { + struct page *hdbss_pg; /* HDBSS buffer page */ + u64 hdbssbr_el2; /* load directly */ + u64 hdbssprod_el2; /* save directly */ + unsigned int buddy_order; /* allocation order for __free_pages */ +}; + struct vncr_tlb; struct kvm_vcpu_arch { @@ -945,6 +955,9 @@ struct kvm_vcpu_arch { /* Hyp-readable copy of kvm_vcpu::pid */ pid_t pid; + + /* HDBSS registers info */ + struct vcpu_hdbss_state hdbss; }; /* diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 7aa08d59d494..7c71560b57e4 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -1039,6 +1039,15 @@ #define GCS_CAP(x) ((((unsigned long)x) & GCS_CAP_ADDR_MASK) | \ GCS_CAP_VALID_TOKEN) + +/* + * Definitions for the HDBSS feature + */ +#define HDBSSBR_EL2(baddr, sz) (((baddr) & HDBSSBR_EL2_BADDR_MASK) | \ + FIELD_PREP(HDBSSBR_EL2_SZ_MASK, sz)) + +#define HDBSSPROD_IDX(prod) FIELD_GET(HDBSSPROD_EL2_INDEX_MASK, prod) + /* * Definitions for GICv5 instructions */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 59612d2f277c..ec2749af64fa 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -18,6 +18,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ guest.o debug.o reset.o sys_regs.o stacktrace.o \ vgic-sys-reg-v3.o fpsimd.o pkvm.o \ arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \ + dirty_bit.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9a6c72a18672..76e2417e72dc 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -38,6 +38,7 @@ #include <asm/kvm_asm.h> #include <asm/kvm_emulate.h> #include <asm/kvm_hyp.h> +#include <asm/kvm_dirty_bit.h> #include <asm/kvm_mmu.h> #include <asm/kvm_nested.h> #include <asm/kvm_pkvm.h> @@ -575,10 +576,19 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) } err = kvm_share_hyp(vcpu, vcpu + 1); - if (err) + if (err) { kvm_vgic_vcpu_destroy(vcpu); + return err; + } - return err; + err = kvm_arm_vcpu_alloc_hdbss(vcpu); + if (err) { + kvm_unshare_hyp(vcpu, vcpu + 1); + kvm_vgic_vcpu_destroy(vcpu); + return err; + } + + return 0; } void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c new file mode 100644 index 000000000000..6a76dc2a6371 --- /dev/null +++ b/arch/arm64/kvm/dirty_bit.c @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2026 Huawei Technologies Co., Ltd + * Author: Tian Zheng <zhengtian10@huawei.com> + */ + +#include <asm/kvm_dirty_bit.h> +#include <asm/kvm_mmu.h> +#include <asm/sysreg.h> +#include <linux/gfp.h> +#include <linux/kconfig.h> +#include <linux/log2.h> +#include <linux/mm.h> + +int kvm_arm_vcpu_alloc_hdbss(struct kvm_vcpu *vcpu) +{ + struct page *hdbss_pg; + u32 size; + unsigned int buddy_order; + u32 sz_encoded; + + if (vcpu->arch.hdbss.hdbss_pg || !system_supports_hdbss()) + return 0; + + size = kvm_hdbss_buffer_size(vcpu->kvm); + buddy_order = get_order(size); + sz_encoded = ilog2(size) - 12; + + hdbss_pg = alloc_pages(GFP_KERNEL_ACCOUNT, buddy_order); + if (!hdbss_pg) + return -ENOMEM; + + vcpu->arch.hdbss = (struct vcpu_hdbss_state) { + .hdbss_pg = hdbss_pg, + .hdbssbr_el2 = HDBSSBR_EL2(page_to_phys(hdbss_pg), sz_encoded), + .hdbssprod_el2 = 0, + .buddy_order = buddy_order, + }; + + return 0; +} + +void kvm_arm_vcpu_free_hdbss(struct kvm_vcpu *vcpu) +{ + if (!vcpu->arch.hdbss.hdbss_pg) + return; + + __free_pages(vcpu->arch.hdbss.hdbss_pg, vcpu->arch.hdbss.buddy_order); + + vcpu->arch.hdbss.hdbss_pg = NULL; + vcpu->arch.hdbss.hdbssbr_el2 = 0; +} diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index bbe9cebd3d9d..c28c1e380127 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -22,6 +22,7 @@ #include <asm/kvm_emulate.h> #include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_dirty_bit.h> #include <asm/fpsimd.h> #include <asm/debug-monitors.h> #include <asm/processor.h> @@ -213,6 +214,17 @@ static void __vcpu_put_deactivate_traps(struct kvm_vcpu *vcpu) local_irq_restore(flags); } +static void __load_hdbss(struct kvm_vcpu *vcpu) +{ + if (!vcpu_hdbss_enabled(vcpu)) + return; + + write_sysreg_s(vcpu->arch.hdbss.hdbssbr_el2, SYS_HDBSSBR_EL2); + write_sysreg_s(vcpu->arch.hdbss.hdbssprod_el2, SYS_HDBSSPROD_EL2); + + isb(); +} + void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu) { host_data_ptr(host_ctxt)->__hyp_running_vcpu = vcpu; @@ -220,10 +232,14 @@ void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu) __vcpu_load_switch_sysregs(vcpu); __vcpu_load_activate_traps(vcpu); __load_stage2(vcpu->arch.hw_mmu); + __load_hdbss(vcpu); } void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu) { + if (vcpu_hdbss_enabled(vcpu)) + vcpu->arch.hdbss.hdbssprod_el2 = read_sysreg_s(SYS_HDBSSPROD_EL2); + __vcpu_put_deactivate_traps(vcpu); __vcpu_put_switch_sysregs(vcpu); diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index b963fd975aac..76088b77cbf8 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -27,6 +27,7 @@ #include <asm/kvm_asm.h> #include <asm/kvm_emulate.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_dirty_bit.h> #include <asm/kvm_nested.h> #include <asm/virt.h> @@ -161,6 +162,8 @@ void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) free_page((unsigned long)vcpu->arch.ctxt.vncr_array); kfree(vcpu->arch.vncr_tlb); kfree(vcpu->arch.ccsidr); + + kvm_arm_vcpu_free_hdbss(vcpu); } static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu) -- 2.33.0
From: eillon <yezhenyu2@huawei.com> Add HDBSS fault handling for buffer full, external abort, and general protection fault (GPF) events. When the HDBSS buffer becomes full, the hardware traps to EL2 with an HDBSSF event; the fault handler returns to the VM exit path which flushes the buffer unconditionally. Add kvm_flush_hdbss_buffer() to consume HDBSS buffer entries and propagate dirty information into the dirty bitmap or dirty ring. Flush is protected by SRCU to avoid the ABBA deadlock that would occur with mmu_lock, since flush may be called under the scheduler's rq->lock. A single flush point is used in kvm_arch_vcpu_ioctl_run(): the buffer is flushed on every VM exit before handling the exit reason. This ensures that entries pushed to the dirty ring are accounted for when check_vcpu_requests() runs dirty_ring_check_request() on the next loop iteration, preventing ring overflow. kvm_arch_sync_dirty_log() kicks vCPUs out of guest mode via KVM_REQ_OUTSIDE_GUEST_MODE; the exit path then flushes automatically. Add esr_iss2_is_hdbssf() helper for HDBSS fault detection in guest abort handling, with guards against HDBSS being unsupported or inactive for the current vCPU including nested virtualization. Signed-off-by: Eillon <yezhenyu2@huawei.com> Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/asm/esr.h | 5 ++ arch/arm64/include/asm/kvm_dirty_bit.h | 20 +++++++ arch/arm64/kvm/arm.c | 24 ++++++++ arch/arm64/kvm/dirty_bit.c | 76 ++++++++++++++++++++++++++ arch/arm64/kvm/mmu.c | 4 ++ 5 files changed, 129 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index f816f5d77f1a..4b3ccd407faa 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -437,6 +437,11 @@ #ifndef __ASSEMBLER__ #include <asm/types.h> +static inline bool esr_iss2_is_hdbssf(unsigned long esr) +{ + return !!(ESR_ELx_ISS2(esr) & ESR_ELx_HDBSSF); +} + static inline unsigned long esr_brk_comment(unsigned long esr) { return esr & ESR_ELx_BRK64_ISS_COMMENT_MASK; diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h index 2a7de96775ef..20b3a5d8e080 100644 --- a/arch/arm64/include/asm/kvm_dirty_bit.h +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -11,6 +11,10 @@ #include <asm/sysreg.h> #include <linux/sizes.h> +/* HDBSS entry field definitions */ +#define HDBSS_ENTRY_VALID BIT(0) +#define HDBSS_ENTRY_IPA GENMASK_ULL(55, 12) + /* Default HDBSS buffer size: one full page per vCPU */ #define KVM_ARM_HDBSS_DEFAULT_SIZE PAGE_SIZE @@ -28,6 +32,20 @@ static inline u32 kvm_hdbss_buffer_size(struct kvm *kvm) return kvm->arch.hdbss_buffer_size ?: KVM_ARM_HDBSS_DEFAULT_SIZE; } +/* + * VM-level HDBSS check -- L1 only, does not filter nested (L2). + * + * Tests kvm->arch.mmu.vtcr, which always reflects the L1 VM's HDBSS + * state regardless of whether a vCPU is currently running L2. + * + * Use in VM-level control paths (enable/disable/destroy/sync_dirty_log) + * that never touch HDBSS registers directly. + */ +static inline bool kvm_hdbss_enabled(struct kvm *kvm) +{ + return kvm->arch.mmu.vtcr & VTCR_EL2_HDBSS; +} + /* * vCPU-level HDBSS check -- filters both L1 and L2. * @@ -47,5 +65,7 @@ static inline bool vcpu_hdbss_enabled(struct kvm_vcpu *vcpu) int kvm_arm_vcpu_alloc_hdbss(struct kvm_vcpu *vcpu); void kvm_arm_vcpu_free_hdbss(struct kvm_vcpu *vcpu); +void kvm_flush_hdbss_buffer(struct kvm_vcpu *vcpu); +int kvm_handle_hdbss_fault(struct kvm_vcpu *vcpu); #endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 76e2417e72dc..0d2e9f79d3ae 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1410,6 +1410,17 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) trace_kvm_exit(ret, kvm_vcpu_trap_get_class(vcpu), *vcpu_pc(vcpu)); + /* + * Flush HDBSS buffer on every VM exit, before handling the + * exit reason. This is the single flush point for the HDBSS + * buffer. Flushing here ensures that entries pushed to the + * dirty ring are accounted for when check_vcpu_requests() + * runs dirty_ring_check_request() on the next loop iteration, + * preventing ring overflow. + */ + if (vcpu_hdbss_enabled(vcpu)) + kvm_flush_hdbss_buffer(vcpu); + /* Exit types that need handling before we can be preempted */ handle_exit_early(vcpu, ret); @@ -1987,7 +1998,20 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl, void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) { + unsigned long i; + struct kvm_vcpu *vcpu; + /* + * Kick all vCPUs out of guest mode. The HDBSS buffer is flushed + * on every VM exit, so ensuring vCPUs have exited ensures all + * buffered dirty entries have been pushed to the dirty bitmap or + * dirty ring. + */ + if (!kvm_hdbss_enabled(kvm)) + return; + + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_vcpu_kick(vcpu); } static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 6a76dc2a6371..46ad47ab3fd5 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -11,6 +11,7 @@ #include <linux/kconfig.h> #include <linux/log2.h> #include <linux/mm.h> +#include <linux/srcu.h> int kvm_arm_vcpu_alloc_hdbss(struct kvm_vcpu *vcpu) { @@ -50,3 +51,78 @@ void kvm_arm_vcpu_free_hdbss(struct kvm_vcpu *vcpu) vcpu->arch.hdbss.hdbss_pg = NULL; vcpu->arch.hdbss.hdbssbr_el2 = 0; } + +void kvm_flush_hdbss_buffer(struct kvm_vcpu *vcpu) +{ + int idx, curr_idx; + u32 entries; + u64 *hdbss_buf; + struct kvm *kvm = vcpu->kvm; + int srcu_idx; + + if (!vcpu_hdbss_enabled(vcpu)) + return; + + curr_idx = HDBSSPROD_IDX(read_sysreg_s(SYS_HDBSSPROD_EL2)); + + /* Do nothing if HDBSS buffer is empty or not allocated */ + if (curr_idx == 0 || !vcpu->arch.hdbss.hdbss_pg) + return; + + hdbss_buf = page_address(vcpu->arch.hdbss.hdbss_pg); + if (!hdbss_buf) + return; + + entries = kvm_hdbss_buffer_size(kvm) / sizeof(u64); + + srcu_idx = srcu_read_lock(&kvm->srcu); + for (idx = 0; idx < min_t(u32, curr_idx, entries); idx++) { + u64 gpa; + + gpa = hdbss_buf[idx]; + if (!(gpa & HDBSS_ENTRY_VALID)) + continue; + + gpa &= HDBSS_ENTRY_IPA; + kvm_vcpu_mark_page_dirty(vcpu, gpa >> PAGE_SHIFT); + } + srcu_read_unlock(&kvm->srcu, srcu_idx); + + /* reset HDBSS index */ + write_sysreg_s(0, SYS_HDBSSPROD_EL2); + vcpu->arch.hdbss.hdbssprod_el2 = 0; + isb(); +} + +int kvm_handle_hdbss_fault(struct kvm_vcpu *vcpu) +{ + u64 prod; + u64 fsc; + + if (WARN_ON_ONCE(!system_supports_hdbss())) + return -EFAULT; + + if (WARN_ON_ONCE(!vcpu_hdbss_enabled(vcpu))) + return -EFAULT; + + prod = read_sysreg_s(SYS_HDBSSPROD_EL2); + fsc = FIELD_GET(HDBSSPROD_EL2_FSC_MASK, prod); + + switch (fsc) { + case HDBSSPROD_EL2_FSC_OK: + /* + * Buffer full. No need to set a flush request, the + * exit path flushes unconditionally before handle_exit. + */ + return 1; + case HDBSSPROD_EL2_FSC_ExternalAbort: + case HDBSSPROD_EL2_FSC_GPF: + return -EFAULT; + default: + /* Unknown fault. */ + WARN_ONCE(1, + "Unexpected HDBSS fault type, FSC: 0x%llx (prod=0x%llx, vcpu=%d)\n", + fsc, prod, vcpu->vcpu_id); + return -EFAULT; + } +} diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 2d95203386ba..a76324fd9be8 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -15,6 +15,7 @@ #include <asm/pgalloc.h> #include <asm/cacheflush.h> #include <asm/kvm_arm.h> +#include <asm/kvm_dirty_bit.h> #include <asm/kvm_mmu.h> #include <asm/kvm_pgtable.h> #include <asm/kvm_pkvm.h> @@ -2271,6 +2272,9 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) is_iabt = kvm_vcpu_trap_is_iabt(vcpu); + if (esr_iss2_is_hdbssf(esr)) + return kvm_handle_hdbss_fault(vcpu); + if (esr_fsc_is_translation_fault(esr)) { /* Beyond sanitised PARange (which is the IPA limit) */ if (fault_ipa >= BIT_ULL(get_kvm_ipa_limit())) { -- 2.33.0
Add VM-level HDBSS enable/disable support following the PML-style buffer lifecycle: buffers are pre-allocated at vCPU create time and freed at vCPU destroy time, decoupled from dirty logging state. When dirty logging is enabled on any memslot, HDBSS is automatically enabled via VTCR_EL2 bits. When dirty logging is disabled on all memslots, HDBSS is automatically disabled. The enable/disable path modifies VTCR only and kicks all vCPUs out of guest mode so the new VTCR value takes effect on re-entry. kvm_arch_vcpu_put() flushes HDBSS before kvm_vcpu_put_hw_mmu() so the hw_mmu VTCR check remains valid for nested virtualization. kvm_arch_destroy_vm() disables HDBSS before kvm_uninit_stage2_mmu() to prevent the hardware accessing buffers that are being destroyed. All VM-level paths use kvm_hdbss_enabled() (kvm->arch.mmu.vtcr) and all per-vCPU paths use vcpu_hdbss_enabled() (hw_mmu->vtcr), correctly handling nested virtualization where the shadow stage-2 has no HDBSS bits. Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/include/asm/kvm_dirty_bit.h | 4 ++ arch/arm64/kvm/arm.c | 9 +++++ arch/arm64/kvm/dirty_bit.c | 54 ++++++++++++++++++++++++++ arch/arm64/kvm/mmu.c | 3 ++ 4 files changed, 70 insertions(+) diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h index 20b3a5d8e080..9dc97c84c43a 100644 --- a/arch/arm64/include/asm/kvm_dirty_bit.h +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -67,5 +67,9 @@ int kvm_arm_vcpu_alloc_hdbss(struct kvm_vcpu *vcpu); void kvm_arm_vcpu_free_hdbss(struct kvm_vcpu *vcpu); void kvm_flush_hdbss_buffer(struct kvm_vcpu *vcpu); int kvm_handle_hdbss_fault(struct kvm_vcpu *vcpu); +void kvm_arm_enable_hdbss_global(struct kvm *kvm); +void kvm_arm_disable_hdbss_global(struct kvm *kvm); +void kvm_arm_hdbss_on_dirty_logging_change(struct kvm *kvm, + int nr_memslots_logging); #endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 0d2e9f79d3ae..aa6dd3c3f06a 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -330,6 +330,15 @@ void kvm_arch_destroy_vm(struct kvm *kvm) if (is_protected_kvm_enabled()) pkvm_destroy_hyp_vm(kvm); + /* + * Userspace may destroy the VM without disabling dirty logging, + * so the auto-disable path is never reached. Force disable HDBSS + * here to kick all vCPUs out of guest mode before kvm_destroy_vcpus + * frees their buffers, preventing hardware UAF. + */ + if (kvm_hdbss_enabled(kvm)) + kvm_arm_disable_hdbss_global(kvm); + kvm_uninit_stage2_mmu(kvm); kvm_destroy_mpidr_data(kvm); diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 46ad47ab3fd5..120e0d52abfd 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -126,3 +126,57 @@ int kvm_handle_hdbss_fault(struct kvm_vcpu *vcpu) return -EFAULT; } } + +void kvm_arm_enable_hdbss_global(struct kvm *kvm) +{ + unsigned long i; + struct kvm_vcpu *vcpu; + + if (!system_supports_hdbss()) + return; + + if (kvm->dirty_ring_size) /* Don't support HDBSS in dirty ring mode */ + return; + + if (kvm_hdbss_enabled(kvm)) /* Already On */ + return; + + /* Turn it on */ + kvm->arch.mmu.vtcr |= VTCR_EL2_HD | VTCR_EL2_HDBSS | VTCR_EL2_HA; + + /* + * Kick all vCPUs out of guest mode so the new VTCR value is + * loaded on re-entry. The per-VM-exit flush path handles + * draining the HDBSS buffer. + */ + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_vcpu_kick(vcpu); +} + +void kvm_arm_disable_hdbss_global(struct kvm *kvm) +{ + unsigned long i; + struct kvm_vcpu *vcpu; + + if (!kvm_hdbss_enabled(kvm)) /* Already Off */ + return; + + /* Turn it off */ + kvm->arch.mmu.vtcr &= ~(VTCR_EL2_HD | VTCR_EL2_HDBSS | VTCR_EL2_HA); + + /* + * Kick all vCPUs out of guest mode so the new VTCR value is + * loaded on re-entry. + */ + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_vcpu_kick(vcpu); +} + +void kvm_arm_hdbss_on_dirty_logging_change(struct kvm *kvm, + int nr_memslots_logging) +{ + if (nr_memslots_logging > 0 && !kvm_hdbss_enabled(kvm)) + kvm_arm_enable_hdbss_global(kvm); + else if (nr_memslots_logging == 0 && kvm_hdbss_enabled(kvm)) + kvm_arm_disable_hdbss_global(kvm); +} diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index a76324fd9be8..b00546bb0fb8 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2580,6 +2580,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, { bool log_dirty_pages = new && new->flags & KVM_MEM_LOG_DIRTY_PAGES; + kvm_arm_hdbss_on_dirty_logging_change(kvm, + atomic_read(&kvm->nr_memslots_dirty_logging)); + /* * At this point memslot has been committed and there is an * allocated dirty_bitmap[], dirty pages will be tracked while the -- 2.33.0
Add KVM_CAP_ARM_HDBSS_BUFFER_SIZE (251) to allow userspace to configure the per-VM HDBSS buffer size before vCPUs are created. The size is specified in bytes via KVM_ENABLE_CAP and must be a power of two in the range [PAGE_SIZE, SZ_2M]. The setting must be applied before any vCPU is created, and attempts to change it after vCPUs exist or after it has already been set return -EINVAL and -EBUSY respectively. KVM_CHECK_EXTENSION returns the configured size for a specific VM, or the maximum supported size (SZ_2M) when queried globally with a NULL kvm argument. Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/kvm/arm.c | 34 ++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 1 + 2 files changed, 35 insertions(+) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index aa6dd3c3f06a..a89d094efd12 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -19,6 +19,7 @@ #include <linux/kvm.h> #include <linux/kvm_irqfd.h> #include <linux/irqbypass.h> +#include <linux/log2.h> #include <linux/sched/stat.h> #include <linux/psci.h> #include <trace/events/kvm.h> @@ -203,6 +204,31 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, r = 0; set_bit(KVM_ARCH_FLAG_EXIT_SEA, &kvm->arch.flags); break; + case KVM_CAP_ARM_HDBSS_BUFFER_SIZE: { + u64 size = cap->args[0]; + + if (!system_supports_hdbss()) + break; + if (kvm->dirty_ring_size) + break; + if (size < KVM_ARM_HDBSS_DEFAULT_SIZE || + size > KVM_ARM_HDBSS_MAX_SIZE) + break; + if (!is_power_of_2(size)) + break; + + mutex_lock(&kvm->lock); + if (kvm->created_vcpus) { + r = -EINVAL; + } else if (kvm->arch.hdbss_buffer_size) { + r = -EBUSY; + } else { + kvm->arch.hdbss_buffer_size = size; + r = 0; + } + mutex_unlock(&kvm->lock); + break; + } default: break; } @@ -496,6 +522,14 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) else r = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT; break; + case KVM_CAP_ARM_HDBSS_BUFFER_SIZE: + if (!system_supports_hdbss()) + r = 0; + else if (kvm) + r = kvm->arch.hdbss_buffer_size ?: KVM_ARM_HDBSS_DEFAULT_SIZE; + else + r = KVM_ARM_HDBSS_MAX_SIZE; + break; case KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES: r = kvm_supported_block_sizes(); break; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 70e36e6a0ad4..ee81ffa0080d 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -998,6 +998,7 @@ struct kvm_enable_cap { #define KVM_CAP_S390_VSIE_ESAMODE 248 #define KVM_CAP_S390_HPAGE_2G 249 #define KVM_CAP_PPC_COMPAT_CAPS 250 +#define KVM_CAP_ARM_HDBSS_BUFFER_SIZE 251 struct kvm_irq_routing_irqchip { __u32 irqchip; -- 2.33.0
Enable HDBSS to coexist with dirty ring mode, following the x86 PML design pattern. HDBSS buffer entries are flushed into the dirty ring via kvm_vcpu_mark_page_dirty(), and the dirty ring reservation mechanism guarantees sufficient ring space. Add a generic kvm_arch_dirty_ring_size_updated() hook, called after kvm->dirty_ring_size is set, which arm64 overrides to auto-configure the HDBSS buffer size. In dirty-ring mode the buffer is fixed at PAGE_SIZE (512 entries on 4KB pages), matching x86 PML. A larger buffer would increase the reserved entries, reduce the dirty ring soft_limit and cause more frequent userspace drains, so the userspace cannot override it via KVM_CAP_ARM_HDBSS_BUFFER_SIZE. kvm_cpu_dirty_log_size() reports the HDBSS buffer entry count so the dirty ring framework computes correct rsvd/soft_limit values, and the dirty-ring rejection in the auto-enable path is removed so HDBSS is enabled whenever dirty logging starts, in either mode. Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- arch/arm64/kvm/arm.c | 22 ++++++++++++++++++++++ arch/arm64/kvm/dirty_bit.c | 3 --- arch/arm64/kvm/mmu.c | 8 ++++++++ include/linux/kvm_dirty_ring.h | 1 + virt/kvm/dirty_ring.c | 4 ++++ virt/kvm/kvm_main.c | 1 + 6 files changed, 36 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index a89d094efd12..75384b735e1d 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -241,6 +241,28 @@ static int kvm_arm_default_max_vcpus(void) return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS; } +/* + * Called after kvm->dirty_ring_size is set, allowing arch to configure + * the HDBSS buffer size. In dirty-ring mode, the buffer size is + * auto-configured by the kernel to PAGE_SIZE (minimum). The user + * cannot override it via KVM_CAP_ARM_HDBSS_BUFFER_SIZE. + * + * A larger buffer increases rsvd, which reduces the dirty ring's + * soft_limit and causes more frequent userspace drains, degrading + * performance. PAGE_SIZE gives the largest drain interval and the + * lowest drain frequency. + */ +void kvm_arch_dirty_ring_size_updated(struct kvm *kvm) +{ + if (!system_supports_hdbss()) + return; + + if (kvm->arch.hdbss_buffer_size) + return; + + kvm->arch.hdbss_buffer_size = KVM_ARM_HDBSS_DEFAULT_SIZE; +} + /** * kvm_arch_init_vm - initializes a VM data structure * @kvm: pointer to the KVM struct diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 120e0d52abfd..d909cc745358 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -135,9 +135,6 @@ void kvm_arm_enable_hdbss_global(struct kvm *kvm) if (!system_supports_hdbss()) return; - if (kvm->dirty_ring_size) /* Don't support HDBSS in dirty ring mode */ - return; - if (kvm_hdbss_enabled(kvm)) /* Already On */ return; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index b00546bb0fb8..de1a919d6f52 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2800,3 +2800,11 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled); } + +int kvm_cpu_dirty_log_size(struct kvm *kvm) +{ + if (!system_supports_hdbss()) + return 0; + + return kvm_hdbss_buffer_size(kvm) / sizeof(u64); +} diff --git a/include/linux/kvm_dirty_ring.h b/include/linux/kvm_dirty_ring.h index eb10d87adf7d..c2b922791530 100644 --- a/include/linux/kvm_dirty_ring.h +++ b/include/linux/kvm_dirty_ring.h @@ -73,6 +73,7 @@ static inline void kvm_dirty_ring_free(struct kvm_dirty_ring *ring) #else /* CONFIG_HAVE_KVM_DIRTY_RING */ int kvm_cpu_dirty_log_size(struct kvm *kvm); +void kvm_arch_dirty_ring_size_updated(struct kvm *kvm); bool kvm_use_dirty_bitmap(struct kvm *kvm); bool kvm_arch_allow_write_without_running_vcpu(struct kvm *kvm); u32 kvm_dirty_ring_get_rsvd_entries(struct kvm *kvm); diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c index 572b854edf74..784f53c95b54 100644 --- a/virt/kvm/dirty_ring.c +++ b/virt/kvm/dirty_ring.c @@ -16,6 +16,10 @@ int __weak kvm_cpu_dirty_log_size(struct kvm *kvm) return 0; } +void __weak kvm_arch_dirty_ring_size_updated(struct kvm *kvm) +{ +} + u32 kvm_dirty_ring_get_rsvd_entries(struct kvm *kvm) { return KVM_DIRTY_RING_RSVD_ENTRIES + kvm_cpu_dirty_log_size(kvm); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 45e784462ec6..e3273be17277 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -4969,6 +4969,7 @@ static int kvm_vm_ioctl_enable_dirty_log_ring(struct kvm *kvm, u32 size) r = -EINVAL; } else { kvm->dirty_ring_size = size; + kvm_arch_dirty_ring_size_updated(kvm); r = 0; } -- 2.33.0
Document the KVM_CAP_ARM_HDBSS_BUFFER_SIZE capability, which provides a mechanism for userspace to configure the per-vCPU Hardware Dirty State Tracking Structure (HDBSS) buffer size used for hardware-assisted dirty page tracking during live migration. The capability applies to dirty-bitmap mode; it is mutually exclusive with the dirty ring (KVM_CAP_DIRTY_LOG_RING), in which the buffer size is auto-configured to the default and cannot be overridden. Signed-off-by: Tian Zheng <zhengtian10@huawei.com> --- Documentation/virt/kvm/api.rst | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 3660d7478fb9..07d561089180 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -9032,6 +9032,35 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key interpretation are disabled. If cmma has already been enabled or the hpage_2g module parameter is not set to 1, -EINVAL is returned. +7.48 KVM_CAP_ARM_HDBSS_BUFFER_SIZE +----------------------------------- + +:Architectures: arm64 +:Target: VM +:Parameters: args[0] is the per-vCPU HDBSS buffer size in bytes +:Returns: 0 on success; -EINVAL if the size is invalid or vCPUs have already + been created; -EBUSY if the buffer size was already configured. + +This capability configures the Hardware Dirty State Tracking Structure +(HDBSS) buffer size used for hardware-assisted dirty page tracking during +live migration. + +When live migration is initiated, userspace can configure the per-vCPU HDBSS +buffer size through this capability; KVM will allocate per-vCPU HDBSS buffers +of the requested size. + +The size must be a power of two between the default (``PAGE_SIZE``) and the +maximum (2MB). This capability applies to dirty-bitmap mode and is mutually +exclusive with the dirty ring (``KVM_CAP_DIRTY_LOG_RING``); in dirty-ring mode +the buffer size is auto-configured to the default and cannot be overridden. + +This capability may only be set before any vCPUs are created. Setting the +size a second time is rejected with -EBUSY. + +KVM_CHECK_EXTENSION for this capability returns the maximum supported buffer +size when queried without a VM, or the currently configured per-VM buffer size +(defaulting to ``PAGE_SIZE``) when queried with a VM. + 8. Other capabilities. ====================== -- 2.33.0
participants (1)
-
Tian Zheng