- Linuxarm - mailweb.openeuler.org

Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of the XQC address
by Shameerali Kolothum Thodi 13 May '24

13 May '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Monday, May 13, 2024 9:16 AM > To: Alex Williamson <alex.williamson(a)redhat.com>; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com> > Cc: jgg(a)nvidia.com; Jonathan Cameron <jonathan.cameron(a)huawei.com>; > kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of > the XQC address > > On 2024/5/9 22:29, Alex Williamson wrote: > > On Thu, 9 May 2024 09:37:51 +0000 > > Shameerali Kolothum Thodi <shameerali.kolothum.thodi(a)huawei.com> > wrote: > > > >>> -----Original Message----- > >>> From: Alex Williamson <alex.williamson(a)redhat.com> > >>> Sent: Wednesday, May 8, 2024 7:00 PM > >>> To: liulongfang <liulongfang(a)huawei.com> > >>> Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > >>> <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > >>> <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > >>> kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > >>> Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register > location of > >>> the XQC address > >> > >> [...] > >> > >>>> HiSilicon accelerator equipment can perform general services after > >>> completing live migration. > >>>> This kind of business is executed through the user mode driver and only > >>> needs to use SQE and CQE. > >>>> > >>>> At the same time, this device can also perform kernel-mode services in > the > >>> VM through the crypto > >>>> subsystem. This kind of service requires the use of EQE. > >>>> > >>>> Finally, if the device is abnormal, the driver needs to perform a device > >>> reset, and AEQE needs to > >>>> be used in this case. > >>>> > >>>> Therefore, a complete device live migration function needs to ensure > that > >>> device functions are > >>>> normal in all these scenarios. > >>>> Therefore, this data still needs to be migrated. > >>> > >>> Ok, I had jumped to an in-kernel host driver in reference to "kernel > >>> mode" rather than a guest kernel. Migrating with bad data only affects > >>> the current configuration of the device, reloading a guest driver to > >>> update these registers or a reset of the device would allow proper > >>> operation of the device, correct? > >> > >> Yes, after talking to Longfang, the device RAS will trigger a reset and > >> would function after reset. > >> > >>> > >>> But I think this still isn't really a complete solution, we know > >>> there's a bug in the migration data stream, so not only would we fix > >>> the data stream, but I think we should also take measures to prevent > >>> loading a known bad data stream. AIUI migration of this device while > >>> running in kernel mode (ie. a kernel driver within a guest VM) is > >>> broken. Therefore, the least we can do in a new kernel, knowing that > >>> there was previously a bug in the migration data stream, is to fail to > >>> load that migration data because it risks this scenario where the > >>> device is broken after migration. Shouldn't we then also increment a > >>> migration version field in the data stream to block migrations that > >>> risk this breakage, or barring that, change the magic data field to > >>> prevent the migration? Thanks, > >> > >> Ok. We could add a new ACC_DEV_MAGIC_V2 and prevent the migration > >> in vf_qm_check_match(). The only concern here is that, it will completely > >> block old kernel to new kernel migration and since we can recover the > >> device after the reset whether it is too restrictive or not. > > > > What's the impact to the running driver, kernel or userspace, if the > > device is reset? Migration is intended to be effectively transparent > > If the device is reset, the user's task needs to be restarted. > If an exception has been detected, the best way is not to migrate. > > > to the driver. If the driver stalls and needs to reset the device, > > what has the migration driver accomplished versus an offline migration? > > > > If there's a way to detect from the migration data if the device is > > running in kernel mode or user mode then you could potentially accept > > and send v1 magic conditional that the device is in user mode and > > require v2 magic for any migration where the device is in kernel mode. > > This all adds complication though and seems like it has corner cases > > where we might allow migration to an old kernel that might trap the > > device there if the use case changes. > > > > The driver does not support checking whether the device is running in > kernel mode or user mode. > Moreover, the device supports user-mode services and kernel-mode services > to run at the same time. > > > Essentially it comes down to what should the migration experience be > > and while restricting old->new and new->old migration is undesirable, > > it seems old->old migration is effectively already broken anyway. As > > you consider a v2 magic, perhaps consider how the migration data > > structure might be improved overall to better handle new features and > > bugs. Thanks, > > > > We discussed a plan: > Update ACC_DEV_MAGIC to ACC_DEV_MAGIC_VERSION and configure its > last byte > as version information: > > /* QM match information, last byte is version number */ > #define ACC_DEV_MAGIC_VERSION 0XACCDEVFEEDCAFE01 Oops..cant have V there. But the idea is replace magic with last byte as version info which can be used in future for handling bugs/features etc. Thanks, Shameer

1 0

Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of the XQC address
by Shameerali Kolothum Thodi 09 May '24

09 May '24

> -----Original Message----- > From: Alex Williamson <alex.williamson(a)redhat.com> > Sent: Wednesday, May 8, 2024 7:00 PM > To: liulongfang <liulongfang(a)huawei.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of > the XQC address [...] > > HiSilicon accelerator equipment can perform general services after > completing live migration. > > This kind of business is executed through the user mode driver and only > needs to use SQE and CQE. > > > > At the same time, this device can also perform kernel-mode services in the > VM through the crypto > > subsystem. This kind of service requires the use of EQE. > > > > Finally, if the device is abnormal, the driver needs to perform a device > reset, and AEQE needs to > > be used in this case. > > > > Therefore, a complete device live migration function needs to ensure that > device functions are > > normal in all these scenarios. > > Therefore, this data still needs to be migrated. > > Ok, I had jumped to an in-kernel host driver in reference to "kernel > mode" rather than a guest kernel. Migrating with bad data only affects > the current configuration of the device, reloading a guest driver to > update these registers or a reset of the device would allow proper > operation of the device, correct? Yes, after talking to Longfang, the device RAS will trigger a reset and would function after reset. > > But I think this still isn't really a complete solution, we know > there's a bug in the migration data stream, so not only would we fix > the data stream, but I think we should also take measures to prevent > loading a known bad data stream. AIUI migration of this device while > running in kernel mode (ie. a kernel driver within a guest VM) is > broken. Therefore, the least we can do in a new kernel, knowing that > there was previously a bug in the migration data stream, is to fail to > load that migration data because it risks this scenario where the > device is broken after migration. Shouldn't we then also increment a > migration version field in the data stream to block migrations that > risk this breakage, or barring that, change the magic data field to > prevent the migration? Thanks, Ok. We could add a new ACC_DEV_MAGIC_V2 and prevent the migration in vf_qm_check_match(). The only concern here is that, it will completely block old kernel to new kernel migration and since we can recover the device after the reset whether it is too restrictive or not. Thanks, Shameer

1 0

Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of the XQC address
by Shameerali Kolothum Thodi 07 May '24

07 May '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, May 7, 2024 9:29 AM > To: Alex Williamson <alex.williamson(a)redhat.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of > the XQC address > > On 2024/5/4 0:11, Alex Williamson wrote: > > On Thu, 25 Apr 2024 21:23:19 +0800 > > Longfang Liu <liulongfang(a)huawei.com> wrote: > > > >> According to the latest hardware register specification. The DMA > >> addresses of EQE and AEQE are not at the front of their respective > >> register groups, but start from the second. > >> So, previously fetching the value starting from the first register > >> would result in an incorrect address. > >> > >> Therefore, the register location from which the address is obtained > >> needs to be modified. > > > > How does this affect migration? Has it ever worked? Does this make > > The general HiSilicon accelerator task will only use SQE and CQE. > EQE is only used when user running kernel mode task and uses interrupt mode. > AEQE is only used when user running task exceptions occur and software reset > is required. > > The DMA addresses of these four queues are written to the device by the device > driver through the mailbox command during driver initialization. > The DMA addresses of EQE and AEQE are migrated through the device register. > > EQE and AEQE are not used in general task, after the live migration is > completed, > this DMA address error will not be found. until we added a new kernel-mode test > case > that we discovered that this address was abnormal. > > > the migration data incompatible? > > > > This address only affects the kernel mode interrupt mode task function and > device > exception recovery function. > They do not affect live migration functionality > > > Fixes: ??? > > OK! Hi, Could you please add the Fixes tag and resend this separately if there are no outstanding comments on this. This is not related to the debugfs support anyway. Thanks, Shameer > Thanks. > Longfang. > >> Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > >> --- > >> drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 8 ++++---- > >> drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 3 +++ > >> 2 files changed, 7 insertions(+), 4 deletions(-) > >> > >> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > >> index 45351be8e270..0c7e31076ff4 100644 > >> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > >> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > >> @@ -516,12 +516,12 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > >> return -EINVAL; > >> > >> /* Every reg is 32 bit, the dma address is 64 bit. */ > >> - vf_data->eqe_dma = vf_data->qm_eqc_dw[1]; > >> + vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; > >> vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > >> - vf_data->eqe_dma |= vf_data->qm_eqc_dw[0]; > >> - vf_data->aeqe_dma = vf_data->qm_aeqc_dw[1]; > >> + vf_data->eqe_dma |= vf_data->qm_eqc_dw[QM_XQC_ADDR_LOW]; > >> + vf_data->aeqe_dma = vf_data->qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > >> vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > >> - vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[0]; > >> + vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[QM_XQC_ADDR_LOW]; > >> > >> /* Through SQC_BT/CQC_BT to get sqc and cqc address */ > >> ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > >> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > >> index 5bab46602fad..f887ab98581c 100644 > >> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > >> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > >> @@ -38,6 +38,9 @@ > >> #define QM_REG_ADDR_OFFSET 0x0004 > >> > >> #define QM_XQC_ADDR_OFFSET 32U > >> +#define QM_XQC_ADDR_LOW 0x1 > >> +#define QM_XQC_ADDR_HIGH 0x2 > >> + > >> #define QM_VF_AEQ_INT_MASK 0x0004 > >> #define QM_VF_EQ_INT_MASK 0x000c > >> #define QM_IFC_INT_SOURCE_V 0x0020 > > > > . > >

1 0

Re: [PATCH v6 4/5] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 07 May '24

07 May '24

> -----Original Message----- > From: Alex Williamson <alex.williamson(a)redhat.com> > Sent: Friday, May 3, 2024 6:21 PM > To: liulongfang <liulongfang(a)huawei.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 4/5] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > On Thu, 25 Apr 2024 21:23:21 +0800 > Longfang Liu <liulongfang(a)huawei.com> wrote: > > > On the debugfs framework of VFIO, if the CONFIG_VFIO_DEBUGFS macro is > > enabled, the debug function is registered for the live migration driver > > of the HiSilicon accelerator device. > > > > After registering the HiSilicon accelerator device on the debugfs > > framework of live migration of vfio, a directory file "hisi_acc" > > of debugfs is created, and then three debug function files are > > created in this directory: > > > > vfio > > | > > +---<dev_name1> > > | +---migration > > | +--state > > | +--hisi_acc > > | +--dev_data > > | +--migf_data > > | +--cmd_state > > | > > +---<dev_name2> > > +---migration > > +--state > > +--hisi_acc > > +--dev_data > > +--migf_data > > +--cmd_state > > > > dev_data file: read device data that needs to be migrated from the > > current device in real time > > migf_data file: read the migration data of the last live migration > > from the current driver. > > cmd_state: used to get the cmd channel state for the device. > > > > +----------------+ +--------------+ +---------------+ > > | migration dev | | src dev | | dst dev | > > +-------+--------+ +------+-------+ +-------+-------+ > > | | | > > | +------v-------+ +-------v-------+ > > | | saving_mif | | resuming_migf | > > read | | file | | file | > > | +------+-------+ +-------+-------+ > > | | copy | > > | +------------+----------+ > > | | > > +-------v---------+ +-------v--------+ > > | data buffer | | debug_migf | > > +-------+---------+ +-------+--------+ > > | | > > cat | cat | > > +-------v--------+ +-------v--------+ > > | dev_data | | migf_data | > > +----------------+ +----------------+ > > > > When accessing debugfs, user can obtain the real-time status data > > of the device through the "dev_data" file. It will directly read > > the device status data and will not affect the live migration > > function. Its data is stored in the allocated memory buffer, > > and the memory is released after data returning to user mode. > > > > To obtain the data of the last complete migration, user need to > > obtain it through the "migf_data" file. Since the live migration > > data only exists during the migration process, it is destroyed > > after the migration is completed. > > In order to save this data, a debug_migf file is created in the > > driver. At the end of the live migration process, copy the data > > to debug_migf. > > > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > > --- > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 225 ++++++++++++++++++ > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + > > 2 files changed, 232 insertions(+) > > > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > index bf358ba94b5d..656b3d975940 100644 > > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > @@ -627,15 +627,33 @@ static void hisi_acc_vf_disable_fd(struct > hisi_acc_vf_migration_file *migf) > > mutex_unlock(&migf->lock); > > } > > > > +static void hisi_acc_debug_migf_copy(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > > + struct hisi_acc_vf_migration_file *src_migf) > > +{ > > + struct hisi_acc_vf_migration_file *dst_migf = hisi_acc_vdev- > >debug_migf; > > + > > + if (!dst_migf) > > + return; > > + > > + mutex_lock(&hisi_acc_vdev->enable_mutex); > > + dst_migf->disabled = src_migf->disabled; > > + dst_migf->total_length = src_migf->total_length; > > + memcpy(&dst_migf->vf_data, &src_migf->vf_data, > > + sizeof(struct acc_vf_data)); > > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > > +} > > + > > static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > > { > > if (hisi_acc_vdev->resuming_migf) { > > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >resuming_migf); > > hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf); > > fput(hisi_acc_vdev->resuming_migf->filp); > > hisi_acc_vdev->resuming_migf = NULL; > > } > > > > if (hisi_acc_vdev->saving_migf) { > > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >saving_migf); > > hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf); > > fput(hisi_acc_vdev->saving_migf->filp); > > hisi_acc_vdev->saving_migf = NULL; > > @@ -1144,6 +1162,7 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > > if (!vf_qm->io_base) > > return -EIO; > > > > + mutex_init(&hisi_acc_vdev->enable_mutex); > > vf_qm->fun_type = QM_HW_VF; > > vf_qm->pdev = vf_dev; > > mutex_init(&vf_qm->mailbox_lock); > > @@ -1294,6 +1313,203 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > > return vfio_pci_core_ioctl(core_vdev, cmd, arg); > > } > > > > +static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device > *vdev) > > +{ > > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + struct device *dev = vdev->dev; > > + int ret; > > + > > + if (!vdev->mig_ops) { > > + dev_err(dev, "device does not support live migration!\n"); > > Sorry, every error path should not spam dmesg with dev_err(). I'm > going to wait until your co-maintainer approves this before looking at > any further iterations of this series. Thanks, Sure. I will sync up with Longfang and also make sure we address all the existing comments on this before posting the next revision. Thanks, Shameer

1 0

Re: [PATCH v3 2/4] hisi_acc_vfio_pci: Create subfunction for data reading
by Shameerali Kolothum Thodi 07 Mar '24

07 Mar '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, March 7, 2024 6:04 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v3 2/4] hisi_acc_vfio_pci: Create subfunction for data > reading > > During the live migration process. It needs to obtain various status > data of drivers and devices. In order to facilitate calling it in the > debugfs function. For all operations that read data from device registers, > the driver creates a subfunction. > Also fixed the location of address data. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 55 ++++++++++--------- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 3 + > 2 files changed, 33 insertions(+), 25 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 45351be8e270..1881f3fa9266 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -486,42 +486,22 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return 0; > } > > -static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, > - struct hisi_acc_vf_migration_file *migf) > +static int vf_qm_read_data(struct hisi_qm *vf_qm, struct acc_vf_data > *vf_data) > { > - struct acc_vf_data *vf_data = &migf->vf_data; > - struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > struct device *dev = &vf_qm->pdev->dev; > int ret; > > - if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > - /* Update state and return with match data */ > - vf_data->vf_qm_state = QM_NOT_READY; > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > - migf->total_length = QM_MATCH_SIZE; > - return 0; > - } > - > - vf_data->vf_qm_state = QM_READY; > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > - > - ret = vf_qm_cache_wb(vf_qm); > - if (ret) { > - dev_err(dev, "failed to writeback QM Cache!\n"); > - return ret; > - } > - > ret = qm_get_regs(vf_qm, vf_data); So this doesn't need the qm_wait_dev_not_ready(vf_qm) check above for the debugfs ? What happens if you read when device is not ready? > if (ret) > return -EINVAL; > > /* Every reg is 32 bit, the dma address is 64 bit. */ > - vf_data->eqe_dma = vf_data->qm_eqc_dw[1]; > + vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; Also since there is no serialization with core migration now, what will be the data returned in debugfs when there is a vf_qm_load_data() is in progress? So I guess the intention or assumption here is that the debugfs data is only valid when the user knows that devices not under migration process. Is that right? Thanks, Shameer > vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->eqe_dma |= vf_data->qm_eqc_dw[0]; > - vf_data->aeqe_dma = vf_data->qm_aeqc_dw[1]; > + vf_data->eqe_dma |= vf_data->qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[0]; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > > /* Through SQC_BT/CQC_BT to get sqc and cqc address */ > ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > @@ -536,6 +516,31 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return -EINVAL; > } > > + return 0; > +} > + > +static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct hisi_acc_vf_migration_file *migf) > +{ > + struct acc_vf_data *vf_data = &migf->vf_data; > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + int ret; > + > + if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > + /* Update state and return with match data */ > + vf_data->vf_qm_state = QM_NOT_READY; > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + migf->total_length = QM_MATCH_SIZE; > + return 0; > + } > + > + vf_data->vf_qm_state = QM_READY; > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + > + ret = vf_qm_read_data(vf_qm, vf_data); > + if (ret) > + return -EINVAL; > + > migf->total_length = sizeof(struct acc_vf_data); > return 0; > } > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 5bab46602fad..7a9dc87627cd 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -38,6 +38,9 @@ > #define QM_REG_ADDR_OFFSET 0x0004 > > #define QM_XQC_ADDR_OFFSET 32U > +#define QM_XQC_ADDR_LOW 0x1 > +#define QM_XQC_ADDR_HIGH 0x2 > + > #define QM_VF_AEQ_INT_MASK 0x0004 > #define QM_VF_EQ_INT_MASK 0x000c > #define QM_IFC_INT_SOURCE_V 0x0020 > -- > 2.24.0

1 0

Re: [PATCH v2 2/3] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 05 Feb '24

05 Feb '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Sunday, February 4, 2024 8:56 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 2/3] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > On the debugfs framework of VFIO, if the CONFIG_VFIO_DEBUGFS macro is > enabled, the debug function is registered for the live migration driver > of the HiSilicon accelerator device. > > After registering the HiSilicon accelerator device on the debugfs > framework of live migration of vfio, a directory file "hisi_acc" > of debugfs is created, and then three debug function files are > created in this directory: > > vfio > | > +---<dev_name1> > | +---migration > | +--state > | +--hisi_acc > | +--attr > | +--data > | +--save > | +--cmd_state > | > +---<dev_name2> > +---migration > +--state > +--hisi_acc > +--attr > +--data > +--save > +--cmd_state > > data file: used to get the migration data from the driver > attr file: used to get device attributes parameters from the driver > save file: used to read the data of the live migration device and save > it to the driver. > cmd_state: used to get the cmd channel state for the device. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 197 ++++++++++++++++++ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 11 + > 2 files changed, 208 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 00a22a990eb8..e51bbb41c2b3 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -15,6 +15,7 @@ > #include <linux/anon_inodes.h> > > #include "hisi_acc_vfio_pci.h" > +#include "../../vfio.h" > > /* Return 0 on VM acc device ready, -ETIMEDOUT hardware timeout */ > static int qm_wait_dev_not_ready(struct hisi_qm *qm) > @@ -606,6 +607,18 @@ hisi_acc_check_int_state(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > } > } > > +static void hisi_acc_vf_migf_save(struct hisi_acc_vf_migration_file > *dst_migf, Can we please rename this to indicate the debug, similar to other debugfs support functions here, something like hisi_acc_vf_debug_migf_save()? > + struct hisi_acc_vf_migration_file *src_migf) > +{ > + if (!dst_migf) > + return; > + > + dst_migf->disabled = false; Is this set to true anywhere for debugfs migf ? > + dst_migf->total_length = src_migf->total_length; > + memcpy(&dst_migf->vf_data, &src_migf->vf_data, > + sizeof(struct acc_vf_data)); > +} > + > static void hisi_acc_vf_disable_fd(struct hisi_acc_vf_migration_file *migf) > { > mutex_lock(&migf->lock); > @@ -618,12 +631,16 @@ static void hisi_acc_vf_disable_fd(struct > hisi_acc_vf_migration_file *migf) > static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > { > if (hisi_acc_vdev->resuming_migf) { > + hisi_acc_vf_migf_save(hisi_acc_vdev->debug_migf, > + hisi_acc_vdev- > >resuming_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf); > fput(hisi_acc_vdev->resuming_migf->filp); > hisi_acc_vdev->resuming_migf = NULL; > } > > if (hisi_acc_vdev->saving_migf) { > + hisi_acc_vf_migf_save(hisi_acc_vdev->debug_migf, > + hisi_acc_vdev->saving_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf); > fput(hisi_acc_vdev->saving_migf->filp); > hisi_acc_vdev->saving_migf = NULL; > @@ -1156,6 +1173,7 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > if (!vf_qm->io_base) > return -EIO; > > + mutex_init(&hisi_acc_vdev->enable_mutex); > vf_qm->fun_type = QM_HW_VF; > vf_qm->pdev = vf_dev; > mutex_init(&vf_qm->mailbox_lock); > @@ -1306,6 +1324,176 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > return vfio_pci_core_ioctl(core_vdev, cmd, arg); > } > > +static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device > *vdev) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev->debug_migf; > + > + if (!vdev->mig_ops || !migf) { > + seq_printf(seq, "%s\n", "device does not support live > migration!"); > + return -EINVAL; > + } > + > + /** > + * When the device is not opened, the io_base is not mapped. > + * The driver cannot perform device read and write operations. > + */ > + if (atomic_read(&hisi_acc_vdev->dev_opened) != DEV_OPEN) { > + seq_printf(seq, "%s\n", "device not opened!"); > + return -EINVAL; > + } > + > + return 0; > +} > + > +static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + u64 value; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->enable_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + return 0; > + } > + > + ret = qm_wait_dev_not_ready(vf_qm); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s\n", "VF device not ready!"); > + return 0; > + } > + > + value = readl(vf_qm->io_base + QM_MB_CMD_SEND_BASE); > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s:0x%llx\n", "mailbox cmd channel state is OK", > value); > + > + return 0; > +} > + > +static int hisi_acc_vf_debug_save(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev->debug_migf; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->enable_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + return 0; > + } > + > + ret = vf_qm_state_save(hisi_acc_vdev, migf); Are you sure this will not interfere with the core migration APIs as we are not holding the state_mutex?. I think Alex also raised a similar concern previously. Applies to other debugfs interfaces here as well. Please check. > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s\n", "failed to save device data!"); > + return 0; > + } > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s\n", "successful to save device data!"); > + > + return 0; > +} > + > +static int hisi_acc_vf_data_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev- > >debug_migf; > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + > + if (debug_migf && debug_migf->total_length) > + seq_hex_dump(seq, "Mig Data:", DUMP_PREFIX_OFFSET, 16, > 1, > + (unsigned char *)&debug_migf->vf_data, > + vf_data_sz, false); > + else > + seq_printf(seq, "%s\n", "device not migrated!"); > + > + return 0; > +} > + > +static int hisi_acc_vf_attr_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev- > >debug_migf; > + > + if (debug_migf && debug_migf->total_length) { > + seq_printf(seq, > + "acc device:\n" > + "device state: %d\n" > + "device ready: %u\n" > + "data valid: %d\n" > + "data size: %lu\n", > + hisi_acc_vdev->mig_state, > + hisi_acc_vdev->vf_qm_state, > + debug_migf->disabled, > + debug_migf->total_length); > + } else { > + seq_printf(seq, "%s\n", "device not migrated!"); > + } > + > + return 0; > +} > + > +static int hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > + struct dentry *vfio_dev_migration = NULL; > + struct dentry *vfio_hisi_acc = NULL; > + struct device *dev = vdev->dev; > + void *migf = NULL; > + > + if (!debugfs_initialized()) > + return 0; > + > + migf = kzalloc(sizeof(struct hisi_acc_vf_migration_file), GFP_KERNEL); > + if (!migf) > + return -ENOMEM; > + hisi_acc_vdev->debug_migf = migf; > + > + vfio_dev_migration = debugfs_lookup("migration", vdev- > >debug_root); > + if (!vfio_dev_migration) { > + kfree(migf); > + dev_err(dev, "failed to lookup migration debugfs file!\n"); > + return -ENODEV; > + } > + > + vfio_hisi_acc = debugfs_create_dir("hisi_acc", vfio_dev_migration); > + debugfs_create_devm_seqfile(dev, "data", vfio_hisi_acc, > + hisi_acc_vf_data_read); > + debugfs_create_devm_seqfile(dev, "attr", vfio_hisi_acc, > + hisi_acc_vf_attr_read); > + debugfs_create_devm_seqfile(dev, "cmd_state", vfio_hisi_acc, > + hisi_acc_vf_debug_cmd); > + debugfs_create_devm_seqfile(dev, "save", vfio_hisi_acc, > + hisi_acc_vf_debug_save); > + > + return 0; > +} > + > +static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + if (!debugfs_initialized()) > + return; > + > + kfree(hisi_acc_vdev->debug_migf); > +} > + > static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev) > { > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > @@ -1323,9 +1511,11 @@ static int hisi_acc_vfio_pci_open_device(struct > vfio_device *core_vdev) > return ret; > } > hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING; > + atomic_set(&hisi_acc_vdev->dev_opened, DEV_OPEN); > } > > vfio_pci_core_finish_enable(vdev); > + > return 0; > } > > @@ -1334,7 +1524,10 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + atomic_set(&hisi_acc_vdev->dev_opened, DEV_CLOSE); > + mutex_lock(&hisi_acc_vdev->enable_mutex); > iounmap(vf_qm->io_base); > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > vfio_pci_core_close_device(core_vdev); > } > > @@ -1425,6 +1618,9 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev > *pdev, const struct pci_device > ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); > if (ret) > goto out_put_vdev; > + > + if (ops == &hisi_acc_vfio_pci_migrn_ops) > + hisi_acc_vfio_debug_init(hisi_acc_vdev); > return 0; > > out_put_vdev: > @@ -1437,6 +1633,7 @@ static void hisi_acc_vfio_pci_remove(struct pci_dev > *pdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_drvdata(pdev); > > vfio_pci_core_unregister_device(&hisi_acc_vdev->core_device); > + hisi_acc_vf_debugfs_exit(hisi_acc_vdev); > vfio_put_device(&hisi_acc_vdev->core_device.vdev); > } > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index dcabfeec6ca1..283bd8acdc42 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -49,6 +49,11 @@ > #define QM_EQC_DW0 0X8000 > #define QM_AEQC_DW0 0X8020 > > +enum acc_dev_state { > + DEV_CLOSE = 0x0, > + DEV_OPEN, > +}; Do we really need this enum as you can directly use 0 or 1? Thanks, Shameer > + > struct acc_vf_data { > #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) > /* QM match information */ > @@ -113,5 +118,11 @@ struct hisi_acc_vf_core_device { > spinlock_t reset_lock; > struct hisi_acc_vf_migration_file *resuming_migf; > struct hisi_acc_vf_migration_file *saving_migf; > + > + /* To make sure the device is enabled */ > + struct mutex enable_mutex; > + atomic_t dev_opened; > + /* For debugfs */ > + struct hisi_acc_vf_migration_file *debug_migf; > }; > #endif /* HISI_ACC_VFIO_PCI_H */ > -- > 2.24.0

1 0

[PATCH 0/3] fix return value type issue and some cleanup
by Xiaofei Tan 11 May '23

11 May '23

Fix return value type issue and some cleanup. Xiaofei Tan (3): rasdaemon: fix return value type issue of read/write function from unistd.h rasdaemon: fix issue of signed and unsigned integer comparison rasdaemon: remove redundant header file and do some cleaup non-standard-hisi_hip08.c | 2 +- non-standard-hisilicon.c | 4 ++-- ras-diskerror-handler.c | 2 +- ras-events.c | 45 ++++++++++++++++++++++++------------ ras-memory-failure-handler.c | 7 +++--- 5 files changed, 37 insertions(+), 23 deletions(-) -- 2.30.0

1 3

[PATCH v7 0/5] add debugfs to migration driver
by Longfang Liu 02 Feb '23

02 Feb '23

Add a debugfs function to the migration driver in VFIO to provide a step-by-step test function for the migration driver. When the execution of live migration fails, the user can view the status and data during the migration process separately from the source and the destination, which is convenient for users to analyze and locate problems. Changes v6 -> v7 Fix some code style issues. Changes v5 -> v6 Control the creation of debugfs through the CONFIG_DEBUG_FS. Changes v4 -> v5 Remove the newly added vfio_migration_ops and use seq_printf to optimize the implementation of debugfs. Changes v3 -> v4 Change the migration_debug_operate interface to debug_root file. Changes v2 -> v3 Extend the debugfs function from hisilicon device to vfio. Changes v1 -> v2 Change the registration method of root_debugfs to register with module initialization. Longfang Liu (5): vfio/migration: Add debugfs to live migration driver hisi_acc_vfio_pci: extract public functions for container_of hisi_acc_vfio_pci: register debugfs for hisilicon migration driver Documentation: add debugfs description for hisi_acc_vfio_pci vfio: update live migration device status .../ABI/testing/debugfs-hisi-migration | 32 ++ drivers/vfio/pci/Makefile | 1 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 306 +++++++++++++++++- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 11 + drivers/vfio/pci/mlx5/main.c | 3 + drivers/vfio/pci/vfio_pci_core.c | 3 + drivers/vfio/pci/vfio_pci_debugfs.c | 80 +++++ drivers/vfio/pci/vfio_pci_debugfs.h | 28 ++ include/linux/vfio.h | 8 + 9 files changed, 462 insertions(+), 10 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.c create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.h -- 2.24.0

3 14

Liquidación de 3.000 ordenadores portátiles Lenovo Yoga retirados de la administración
by PcRenting / LaCasaAzul 01 Feb '23

01 Feb '23

Liquidacin de ordenadores retirados de la administracin Buenos das: Nuestra empresa ha retirado en varios lotes ms de 3.000 PORTTILES LENOVO YOGA DE LA ADMINISTRACIN, y nos han encargado liquidarlos por un importe de slo 110 euros la unidad. Estos equipos se han revisado, formateado, y se les ha instalado Windows 10 pro con su licencia por tiempo indefinido. Se trata de porttiles perfectos para trabajos de ofimtica, administrativos, teletrabajo, teleformacin o navegar por internet. Si quiere puede pedirnos informacin detallada contestando al correo alberto(a)almeriaocasion.es [1] En espera de que esta informacin haya sido de su inters, reciba un cordial saludo. Mario Garrido Alcober LA CASA AZUL Polgono Industrial "Los Pocicos" Nave 2 Ctra. N 332 Km 11 Cuevas del Almanzora, 04610 Almera. Tlf: 950 455 892 alberto(a)almeriaocasion.es [2] 683 583 017 Links: ------ [1] mailto:alberto@almeriaocasion.es [2] mailto:alberto@almeriaocasion.es

1 0

Faktoring
by "Szczepan Kiełbasa" 01 Feb '23

01 Feb '23

Dzień dobry, rozważali Państwo wybór finansowania, które spełni potrzeby firmy, zapewniając natychmiastowy dostęp do gotówki, bez zbędnych przestojów? Przygotowaliśmy rozwiązania faktoringowe dopasowane do Państwa branży i wielkości firmy, dzięki którym, nie muszą Państwo martwić się o niewypłacalność kontrahentów, ponieważ transakcje są zabezpieczone i posiadają gwarancję spłaty. Chcą Państwo przeanalizować dostępne opcje? Pozdrawiam Szczepan Kiełbasa

1 0