December 2024 - Linuxarm - mailweb.openeuler.org

Re: [PATCH v2 0/5] bugfix some driver issues
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

Hi Longfang, > -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 0/5] bugfix some driver issues > > As the test scenarios for the live migration function become > more and more extensive. Some previously undiscovered driver > issues were found. > Update and fix through this patchset. > A changes history will be beneficial to understand what changed from v1. Also I have a request.... As part of this series, could you please include the below fix to this driver as well as we do have the same potential problem. https://lore.kernel.org/all/20241021123843.42979-1-giovanni.cabiddu@intel.c… Thanks, Shameer > Longfang Liu (5): > hisi_acc_vfio_pci: fix XQE dma address error > hisi_acc_vfio_pci: add eq and aeq interruption restore > hisi_acc_vfio_pci: bugfix cache write-back issue > hisi_acc_vfio_pci: bugfix the problem of uninstalling driver > hisi_acc_vfio_pci: bugfix live migration function without VF device > driver > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 83 ++++++++++++++++--- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 9 +- > 2 files changed, 78 insertions(+), 14 deletions(-) > > -- > 2.24.0

1 0

Re: [PATCH v2 5/5] hisi_acc_vfio_pci: bugfix live migration function without VF device driver
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 5/5] hisi_acc_vfio_pci: bugfix live migration function > without VF device driver > > If the driver of the VF device is not loaded in the Guest OS, > then perform device data migration. The migrated data address will > be NULL. > The live migration recovery operation on the destination side will > access a null address value, which will cause access errors. > > Therefore, live migration of VMs without added VF device drivers > does not require device data migration. > In addition, when the queue address data obtained by the destination > is empty, device queue recovery processing will not be performed. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> Why this doesn't need a Fixes tag? > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 18 ++++++++++++++++++ > 1 file changed, 18 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 8d9e07ebf4fd..9a5f7e9bc695 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -436,6 +436,7 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > struct acc_vf_data *vf_data) > { > struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > struct device *dev = &pf_qm->pdev->dev; > int vf_id = hisi_acc_vdev->vf_id; > int ret; > @@ -460,6 +461,13 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + /* Get VF driver insmod state */ > + ret = qm_read_regs(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, > 1); > + if (ret) { > + dev_err(dev, "failed to read QM_VF_STATE!\n"); > + return ret; > + } > + > return 0; > } > > @@ -499,6 +507,12 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > qm->qp_base = vf_data->qp_base; > qm->qp_num = vf_data->qp_num; > > + if (!vf_data->eqe_dma || !vf_data->aeqe_dma || > + !vf_data->sqc_dma || !vf_data->cqc_dma) { > + dev_err(dev, "resume dma addr is NULL!\n"); > + return -EINVAL; > + } > + So this is to cover the corner case where the Guest has loaded the driver (QM_READY set) but not configured the DMA address? When this will happen? I thought we are setting QM_READY in guest after all configurations. Thanks, Shameer

1 0

Re: [PATCH v2 4/5] hisi_acc_vfio_pci: bugfix the problem of uninstalling driver
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 4/5] hisi_acc_vfio_pci: bugfix the problem of uninstalling > driver > > In a live migration scenario. If the number of VFs at the destination is > greater than the source, the recovery operation will fail and qemu will not > be able to complete the process and exit after shutting down the device FD. > > This will cause the driver to be unable to be unloaded normally due to > abnormal reference counting of the live migration driver caused by the > abnormal closing operation of fd. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index c057c0e24693..8d9e07ebf4fd 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -1501,6 +1501,7 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + hisi_acc_vf_disable_fds(hisi_acc_vdev); > mutex_lock(&hisi_acc_vdev->open_mutex); > hisi_acc_vdev->dev_opened = false; > iounmap(vf_qm->io_base); Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v2 3/5] hisi_acc_vfio_pci: bugfix cache write-back issue
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 3/5] hisi_acc_vfio_pci: bugfix cache write-back issue > > At present, cache write-back is placed in the device data > copy stage after stopping the device operation. > Writing back to the cache at this stage will cause the data > obtained by the cache to be written back to be empty. > > In order to ensure that the cache data is written back > successfully, the data needs to be written back into the > stop device stage. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 14 +++++++------- > 1 file changed, 7 insertions(+), 7 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 4c8f1ae5b636..c057c0e24693 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -559,7 +559,6 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > { > struct acc_vf_data *vf_data = &migf->vf_data; > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > - struct device *dev = &vf_qm->pdev->dev; > int ret; > > if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > @@ -573,12 +572,6 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > vf_data->vf_qm_state = QM_READY; > hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > > - ret = vf_qm_cache_wb(vf_qm); > - if (ret) { > - dev_err(dev, "failed to writeback QM Cache!\n"); > - return ret; > - } > - > ret = vf_qm_read_data(vf_qm, vf_data); > if (ret) > return -EINVAL; > @@ -1005,6 +998,13 @@ static int hisi_acc_vf_stop_device(struct > hisi_acc_vf_core_device *hisi_acc_vdev > dev_err(dev, "failed to check QM INT state!\n"); > return ret; > } > + > + ret = vf_qm_cache_wb(vf_qm); > + if (ret) { > + dev_err(dev, "failed to writeback QM cache!\n"); > + return ret; > + } > + > return 0; > } Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v2 2/5] hisi_acc_vfio_pci: add eq and aeq interruption restore
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 2/5] hisi_acc_vfio_pci: add eq and aeq interruption > restore > > In order to ensure that the task packets of the accelerator > device are not lost during the migration process, it is necessary > to send an EQ and AEQ command to the device after the live migration > is completed and to update the completion position of the task queue. > > Let the device recheck the completed tasks data and if there are > uncollected packets, device resend a task completion interrupt > to the software. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 8518efea3a52..4c8f1ae5b636 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -463,6 +463,19 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return 0; > } > > +static void vf_qm_xeqc_save(struct hisi_qm *qm, > + struct hisi_acc_vf_migration_file *migf) > +{ > + struct acc_vf_data *vf_data = &migf->vf_data; > + u16 eq_head, aeq_head; > + > + eq_head = vf_data->qm_eqc_dw[0] & 0xFFFF; > + qm_db(qm, 0, QM_DOORBELL_CMD_EQ, eq_head, 0); > + > + aeq_head = vf_data->qm_aeqc_dw[0] & 0xFFFF; > + qm_db(qm, 0, QM_DOORBELL_CMD_AEQ, aeq_head, 0); > +} > + > static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev, > struct hisi_acc_vf_migration_file *migf) > { > @@ -571,6 +584,9 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return -EINVAL; > > migf->total_length = sizeof(struct acc_vf_data); > + /* Save eqc and aeqc interrupt information */ > + vf_qm_xeqc_save(vf_qm, migf); > + > return 0; > } Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v2 1/5] hisi_acc_vfio_pci: fix XQE dma address error
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 1/5] hisi_acc_vfio_pci: fix XQE dma address error > > The dma addresses of EQE and AEQE are wrong after migration and > results in guest kernel-mode encryption services failure. > Comparing the definition of hardware registers, we found that > there was an error when the data read from the register was > combined into an address. Therefore, the address combination > sequence needs to be corrected. > > Even after fixing the above problem, we still have an issue > where the Guest from an old kernel can get migrated to > new kernel and may result in wrong data. > > In order to ensure that the address is correct after migration, > if an old magic number is detected, the dma address needs to be > updated. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 34 +++++++++++++++---- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 9 ++++- > 2 files changed, 36 insertions(+), 7 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 451c639299eb..8518efea3a52 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -350,6 +350,27 @@ static int vf_qm_func_stop(struct hisi_qm *qm) > return hisi_qm_mb(qm, QM_MB_CMD_PAUSE_QM, 0, 0, 0); > } > > +static int vf_qm_magic_check(struct acc_vf_data *vf_data) > +{ > + switch (vf_data->acc_magic) { > + case ACC_DEV_MAGIC_V2: > + break; > + case ACC_DEV_MAGIC_V1: > + /* Correct dma address */ > + vf_data->eqe_dma = vf_data- > >qm_eqc_dw[QM_XQC_ADDR_HIGH]; > + vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > + vf_data->eqe_dma |= vf_data- > >qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > + vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > + break; > + default: > + return -EINVAL; > + } > + > + return 0; > +} > + > static int vf_qm_check_match(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > struct hisi_acc_vf_migration_file *migf) > { > @@ -363,7 +384,8 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > if (migf->total_length < QM_MATCH_SIZE || hisi_acc_vdev- > >match_done) > return 0; > > - if (vf_data->acc_magic != ACC_DEV_MAGIC) { > + ret = vf_qm_magic_check(vf_data); > + if (ret) { > dev_err(dev, "failed to match ACC_DEV_MAGIC\n"); > return -EINVAL; > } > @@ -418,7 +440,7 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > int vf_id = hisi_acc_vdev->vf_id; > int ret; > > - vf_data->acc_magic = ACC_DEV_MAGIC; > + vf_data->acc_magic = ACC_DEV_MAGIC_V2; > /* Save device id */ > vf_data->dev_id = hisi_acc_vdev->vf_dev->device; > > @@ -496,12 +518,12 @@ static int vf_qm_read_data(struct hisi_qm > *vf_qm, struct acc_vf_data *vf_data) > return -EINVAL; > > /* Every reg is 32 bit, the dma address is 64 bit. */ > - vf_data->eqe_dma = vf_data->qm_eqc_dw[1]; > + vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->eqe_dma |= vf_data->qm_eqc_dw[0]; > - vf_data->aeqe_dma = vf_data->qm_aeqc_dw[1]; > + vf_data->eqe_dma |= vf_data->qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[0]; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > > /* Through SQC_BT/CQC_BT to get sqc and cqc address */ > ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 245d7537b2bc..2afce68f5a34 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -39,6 +39,9 @@ > #define QM_REG_ADDR_OFFSET 0x0004 > > #define QM_XQC_ADDR_OFFSET 32U > +#define QM_XQC_ADDR_LOW 0x1 > +#define QM_XQC_ADDR_HIGH 0x2 > + > #define QM_VF_AEQ_INT_MASK 0x0004 > #define QM_VF_EQ_INT_MASK 0x000c > #define QM_IFC_INT_SOURCE_V 0x0020 > @@ -50,10 +53,14 @@ > #define QM_EQC_DW0 0X8000 > #define QM_AEQC_DW0 0X8020 > > +enum acc_magic_num { > + ACC_DEV_MAGIC_V1 = 0XCDCDCDCDFEEDAACC, > + ACC_DEV_MAGIC_V2 = 0xAACCFEEDDECADEDE, I think we have discussed this before that having some kind of version info embed into magic_num will be beneficial going forward. ie, may be use the last 4 bytes for denoting version. ACC_DEV_MAGIC_V2 = 0xAACCFEEDDECA0002 The reason being, otherwise we have to come up with a random magic each time when a fix like this is required in future. Thanks, Shameer

1 0