- Linuxarm - mailweb.openeuler.org

Re: [PATCH v2 1/4] uacce: fix for cdev memory leak
by linwenkai (C) 26 Sep '25

26 Sep '25

在 2025/9/26 16:47, huangchenghai 写道: > > On Wed, Sep 17, 2025 at 06:18 PM +0800, Greg KH wrote: >> On Wed, Sep 17, 2025 at 05:56:16PM +0800, huangchenghai wrote: >>> On Mon, Sep 16, 2025 at 11:15 PM +0800, Greg KH wrote: >>>> On Tue, Sep 16, 2025 at 10:48:08PM +0800, Chenghai Huang wrote: >>>>> From: Wenkai Lin <linwenkai6(a)hisilicon.com> >>>>> >>>>> If cdev_device_add failed, it is hard to determine >>>>> whether cdev_del has been executed, which lead to a >>>>> memory leak issue, so we use cdev_init to avoid it. >>>> I do not understand, what is wrong with the current code? It checks if >>>> add fails: >>>> >>>>> Fixes: 015d239ac014 ("uacce: add uacce driver") >>>>> Cc: stable(a)vger.kernel.org >>>>> Signed-off-by: Wenkai Lin <linwenkai6(a)hisilicon.com> >>>>> Signed-off-by: Chenghai Huang <huangchenghai2(a)huawei.com> >>>>> --- >>>>> drivers/misc/uacce/uacce.c | 13 ++++--------- >>>>> include/linux/uacce.h | 2 +- >>>>> 2 files changed, 5 insertions(+), 10 deletions(-) >>>>> >>>>> diff --git a/drivers/misc/uacce/uacce.c b/drivers/misc/uacce/uacce.c >>>>> index 42e7d2a2a90c..12370469f646 100644 >>>>> --- a/drivers/misc/uacce/uacce.c >>>>> +++ b/drivers/misc/uacce/uacce.c >>>>> @@ -522,14 +522,10 @@ int uacce_register(struct uacce_device *uacce) >>>>> if (!uacce) >>>>> return -ENODEV; >>>>> - uacce->cdev = cdev_alloc(); >>>>> - if (!uacce->cdev) >>>>> - return -ENOMEM; >>>> This is the check. >>>> >>>> >>>>> - >>>>> - uacce->cdev->ops = &uacce_fops; >>>>> - uacce->cdev->owner = THIS_MODULE; >>>>> + cdev_init(&uacce->cdev, &uacce_fops); >>>>> + uacce->cdev.owner = THIS_MODULE; >>>>> - return cdev_device_add(uacce->cdev, &uacce->dev); >>>>> + return cdev_device_add(&uacce->cdev, &uacce->dev); >>>> And so is this. So what is wrong here? >>>> >>>> >>>>> } >>>>> EXPORT_SYMBOL_GPL(uacce_register); >>>>> @@ -568,8 +564,7 @@ void uacce_remove(struct uacce_device *uacce) >>>>> unmap_mapping_range(q->mapping, 0, 0, 1); >>>>> } >>>>> - if (uacce->cdev) >>>>> - cdev_device_del(uacce->cdev, &uacce->dev); >>>>> + cdev_device_del(&uacce->cdev, &uacce->dev); >>>>> xa_erase(&uacce_xa, uacce->dev_id); >>>>> /* >>>>> * uacce exists as long as there are open fds, but ops will >>>>> be freed >>>>> diff --git a/include/linux/uacce.h b/include/linux/uacce.h >>>>> index e290c0269944..98b896192a44 100644 >>>>> --- a/include/linux/uacce.h >>>>> +++ b/include/linux/uacce.h >>>>> @@ -126,7 +126,7 @@ struct uacce_device { >>>>> bool is_vf; >>>>> u32 flags; >>>>> u32 dev_id; >>>>> - struct cdev *cdev; >>>>> + struct cdev cdev; >>>>> struct device dev; >>>> You can not do this, you now have 2 different reference counts >>>> controlling the lifespan of this one structure. That is just going to >>>> cause so many more bugs... >>>> >>>> How was this tested? What is currently failing that requires this >>>> change? >>>> >>>> thanks, >>>> >>>> greg k-h >>> We analyze it theoretically there may be a memory leak >>> issue here, if the cdev_device_add returns a failure, >>> the uacce_remove will not be executed, which results in the >>> uacce cdev memory not being released. >> Then properly clean up if that happens. >> >>> Therefore, we have decided to align with the design of other >>> drivers by making cdev a static member of uacce_device and >>> releasing the memory through uacce_device. >> But again, this is wrong to do. >> >>> found one example in drivers/watchdog/watchdog_dev.h. >>> struct watchdog_core_data { >>> struct device dev; >>> struct cdev cdev; >> This is also wrong and needs to be fixed. Please send a patch to >> resolve it as well, as it should not be copied as a valid example. >> >> thanks, >> >> greg k-h > Very sorry for the delayed response. > > In v1, our first thought was that if cdev_device_add returns a > failure, we could release the resources allocated by cdev_alloc > using cdev_del. For this, we attempted the following modification: > > @@ -519,6 +519,8 @@ EXPORT_SYMBOL_GPL(uacce_alloc); > */ > int uacce_register(struct uacce_device *uacce) > { > + int ret; > + > if (!uacce) > return -ENODEV; > > @@ -529,7 +531,14 @@ int uacce_register(struct uacce_device *uacce) > uacce->cdev->ops = &uacce_fops; > uacce->cdev->owner = THIS_MODULE; > > - return cdev_device_add(uacce->cdev, &uacce->dev); > + ret = cdev_device_add(uacce->cdev, &uacce->dev); > + if (ret) { > + cdev_del(uacce->cdev); > + uacce->cdev = NULL; > + return ret; > + } > + > + return 0; > } > > However, after further analysis, we found that cdev_device_add does > not increment the reference count when it fails. Therefore, in this > case, cdev_del is not necessary. This means that the resources > allocated by cdev_alloc will not cause a memory leak in the failure > path. > > Thus, I believe this patch modification is unnecessary. In the > upcoming v3 version, I will remove this modification. > > Thank you for your patient guidance! > > Best regards, > Chenghai >> After further analysis, it was found that during the cdev_alloc process, the reference count of the cdev is initialized to 1. If kobject_put is not actively executed, the release function of the cdev cannot be called, and the memory will never be released, leading to a leak, so kobject_put is still necessary here? Here is the new solution: @@ -519,6 +519,8 @@ EXPORT_SYMBOL_GPL(uacce_alloc); */ int uacce_register(struct uacce_device *uacce) { + int ret; + if (!uacce) return -ENODEV; @@ -529,7 +531,14 @@ int uacce_register(struct uacce_device *uacce) uacce->cdev->ops = &uacce_fops; uacce->cdev->owner = THIS_MODULE; - return cdev_device_add(uacce->cdev, &uacce->dev); + ret = cdev_device_add(uacce->cdev, &uacce->dev); + if (ret) { + kobject_put(&uacce->cdev->kobject); + uacce->cdev = NULL; + return ret; + } + + return 0; } Thanks, WenKai

1 0

Re: [PATCH v6 3/3] migration: adapt to new migration configuration
by Shameerali Kolothum Thodi 28 Jul '25

28 Jul '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, July 17, 2025 2:15 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; > herbert(a)gondor.apana.org.au; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: linux-crypto(a)vger.kernel.org; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org; liulongfang > <liulongfang(a)huawei.com> > Subject: [PATCH v6 3/3] migration: adapt to new migration configuration > > On new platforms greater than QM_HW_V3, the migration region has been > relocated from the VF to the PF. The driver must also be modified > accordingly to adapt to the new hardware device. > > Utilize the PF's I/O base directly on the new hardware platform, > and no mmap operation is required. If it is on an old platform, > the driver needs to be compatible with the old solution. I think it's good to clarify here how the new hardware platform will continue to work with the existing driver. From our off-list discussions, my understanding is that the QM_HW_V4 hardware is backward compatible and will function identically to QM_HW_V3 as long as the PF is not configured to enable the migration region, as introduced in patch #2. This backward compatibility is the reason we are not preventing migration from older kernels to the new one. With that in place: Reviewed-by: Shameer Kolothum shameerali.kolothum.thodi(a)huawei.com Thanks, Shameer > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 164 ++++++++++++------ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + > 2 files changed, 118 insertions(+), 53 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 515ff87f9ed9..bf4a7468bca0 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 > *addr) > return 0; > } > > +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) > { > struct device *dev = &qm->pdev->dev; > @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to read QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to read QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to write QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to write QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -522,6 +560,10 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return ret; > + > ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0); > if (ret) { > dev_err(dev, "set sqc failed\n"); > @@ -589,6 +631,10 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > vf_data->vf_qm_state = QM_READY; > hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > > + ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return ret; > + > ret = vf_qm_read_data(vf_qm, vf_data); > if (ret) > return ret; > @@ -1186,34 +1232,45 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > { > struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device; > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; > struct pci_dev *vf_dev = vdev->pdev; > > - /* > - * ACC VF dev BAR2 region consists of both functional register space > - * and migration control register space. For migration to work, we > - * need access to both. Hence, we map the entire BAR2 region here. > - * But unnecessarily exposing the migration BAR region to the Guest > - * has the potential to prevent/corrupt the Guest migration. Hence, > - * we restrict access to the migration control space from > - * Guest(Please see mmap/ioctl/read/write override functions). > - * > - * Please note that it is OK to expose the entire VF BAR if migration > - * is not supported or required as this cannot affect the ACC PF > - * configurations. > - * > - * Also the HiSilicon ACC VF devices supported by this driver on > - * HiSilicon hardware platforms are integrated end point devices > - * and the platform lacks the capability to perform any PCIe P2P > - * between these devices. > - */ > - > - vf_qm->io_base = > - ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > - pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > - if (!vf_qm->io_base) > - return -EIO; > + if (pf_qm->ver == QM_HW_V3) { > + /* > + * ACC VF dev BAR2 region consists of both functional > register space > + * and migration control register space. For migration to > work, we > + * need access to both. Hence, we map the entire BAR2 > region here. > + * But unnecessarily exposing the migration BAR region to > the Guest > + * has the potential to prevent/corrupt the Guest migration. > Hence, > + * we restrict access to the migration control space from > + * Guest(Please see mmap/ioctl/read/write override > functions). > + * > + * Please note that it is OK to expose the entire VF BAR if > migration > + * is not supported or required as this cannot affect the ACC > PF > + * configurations. > + * > + * Also the HiSilicon ACC VF devices supported by this driver > on > + * HiSilicon hardware platforms are integrated end point > devices > + * and the platform lacks the capability to perform any PCIe > P2P > + * between these devices. > + */ > > + vf_qm->io_base = > + ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > + pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > + if (!vf_qm->io_base) > + return -EIO; > + } else { > + /* > + * On hardware platforms greater than QM_HW_V3, the > migration function > + * register is placed in the BAR2 configuration region of the > PF, > + * and each VF device occupies 8KB of configuration space. > + */ > + vf_qm->io_base = pf_qm->io_base + > QM_MIG_REGION_OFFSET + > + hisi_acc_vdev->vf_id * > QM_MIG_REGION_SIZE; > + } > vf_qm->fun_type = QM_HW_VF; > + vf_qm->ver = pf_qm->ver; > vf_qm->pdev = vf_dev; > mutex_init(&vf_qm->mailbox_lock); > > @@ -1539,7 +1596,8 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > hisi_acc_vf_disable_fds(hisi_acc_vdev); > mutex_lock(&hisi_acc_vdev->open_mutex); > hisi_acc_vdev->dev_opened = false; > - iounmap(vf_qm->io_base); > + if (vf_qm->ver == QM_HW_V3) > + iounmap(vf_qm->io_base); > mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_close_device(core_vdev); > } > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 91002ceeebc1..348f8bb5b42c 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -59,6 +59,13 @@ > #define ACC_DEV_MAGIC_V1 0XCDCDCDCDFEEDAACC > #define ACC_DEV_MAGIC_V2 0xAACCFEEDDECADEDE > > +#define QM_MIG_REGION_OFFSET 0x180000 > +#define QM_MIG_REGION_SIZE 0x2000 > + > +#define QM_SUB_VERSION_ID 0x100210 > +#define QM_EQC_PF_DW0 0x1c00 > +#define QM_AEQC_PF_DW0 0x1c20 > + > struct acc_vf_data { > #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) > /* QM match information */ > -- > 2.24.0

1 0

Re: [PATCH v5 3/3] migration: adapt to new migration configuration
by Shameerali Kolothum Thodi 08 Jul '25

08 Jul '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Monday, June 30, 2025 9:54 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v5 3/3] migration: adapt to new migration configuration > > On new platforms greater than QM_HW_V3, the migration region has been > relocated from the VF to the PF. The driver must also be modified > accordingly to adapt to the new hardware device. > > Utilize the PF's I/O base directly on the new hardware platform, > and no mmap operation is required. If it is on an old platform, > the driver needs to be compatible with the old solution. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 166 ++++++++++++------ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + > 2 files changed, 120 insertions(+), 53 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 1ddc9dbadb70..3aec3b92787f 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 > *addr) > return 0; > } > > +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) > { > struct device *dev = &qm->pdev->dev; > @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to read QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to read QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to write QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to write QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -522,6 +560,10 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return ret; > + > ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0); > if (ret) { > dev_err(dev, "set sqc failed\n"); > @@ -589,6 +631,10 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > vf_data->vf_qm_state = QM_READY; > hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > > + ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return ret; > + > ret = vf_qm_read_data(vf_qm, vf_data); > if (ret) > return ret; > @@ -1186,34 +1232,47 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > { > struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device; > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; > struct pci_dev *vf_dev = vdev->pdev; > > - /* > - * ACC VF dev BAR2 region consists of both functional register space > - * and migration control register space. For migration to work, we > - * need access to both. Hence, we map the entire BAR2 region here. > - * But unnecessarily exposing the migration BAR region to the Guest > - * has the potential to prevent/corrupt the Guest migration. Hence, > - * we restrict access to the migration control space from > - * Guest(Please see mmap/ioctl/read/write override functions). > - * > - * Please note that it is OK to expose the entire VF BAR if migration > - * is not supported or required as this cannot affect the ACC PF > - * configurations. > - * > - * Also the HiSilicon ACC VF devices supported by this driver on > - * HiSilicon hardware platforms are integrated end point devices > - * and the platform lacks the capability to perform any PCIe P2P > - * between these devices. > - */ > + if (pf_qm->ver == QM_HW_V3) { > + /* > + * ACC VF dev BAR2 region consists of both functional > register space > + * and migration control register space. For migration to > work, we > + * need access to both. Hence, we map the entire BAR2 > region here. > + * But unnecessarily exposing the migration BAR region to > the Guest > + * has the potential to prevent/corrupt the Guest migration. > Hence, > + * we restrict access to the migration control space from > + * Guest(Please see mmap/ioctl/read/write override > functions). > + * > + * Please note that it is OK to expose the entire VF BAR if > migration > + * is not supported or required as this cannot affect the ACC > PF > + * configurations. > + * > + * Also the HiSilicon ACC VF devices supported by this driver > on > + * HiSilicon hardware platforms are integrated end point > devices > + * and the platform lacks the capability to perform any PCIe > P2P > + * between these devices. > + */ > > - vf_qm->io_base = > - ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > - pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > - if (!vf_qm->io_base) > - return -EIO; > + vf_qm->io_base = > + ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > + pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > + if (!vf_qm->io_base) > + return -EIO; > > - vf_qm->fun_type = QM_HW_VF; > + vf_qm->fun_type = QM_HW_VF; > + vf_qm->ver = pf_qm->ver; > + } else { > + /* > + * On hardware platforms greater than QM_HW_V3, the > migration function > + * register is placed in the BAR2 configuration region of the > PF, > + * and each VF device occupies 8KB of configuration space. > + */ > + vf_qm->io_base = pf_qm->io_base + > QM_MIG_REGION_OFFSET + > + hisi_acc_vdev->vf_id * > QM_MIG_REGION_SIZE; > + vf_qm->fun_type = QM_HW_PF; I don't think you can use the QM fun_type to distinguish this, because it is still a VF dev. Better to have another field to detect this. Also I have another question. In the hisi_acc_vfio_pci_probe() we currently look for pf_qm-> >= QM_HW_V3. This means current driver can get loaded on this new hardware, right? If so, I think we need to prevent that. And also we need to block any migration attempt from existing host kernels to new ones. Thanks, Shameer > + } > vf_qm->pdev = vf_dev; > mutex_init(&vf_qm->mailbox_lock); > > @@ -1539,7 +1598,8 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > hisi_acc_vf_disable_fds(hisi_acc_vdev); > mutex_lock(&hisi_acc_vdev->open_mutex); > hisi_acc_vdev->dev_opened = false; > - iounmap(vf_qm->io_base); > + if (vf_qm->ver == QM_HW_V3) > + iounmap(vf_qm->io_base); > mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_close_device(core_vdev); > } > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 91002ceeebc1..348f8bb5b42c 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -59,6 +59,13 @@ > #define ACC_DEV_MAGIC_V1 0XCDCDCDCDFEEDAACC > #define ACC_DEV_MAGIC_V2 0xAACCFEEDDECADEDE > > +#define QM_MIG_REGION_OFFSET 0x180000 > +#define QM_MIG_REGION_SIZE 0x2000 > + > +#define QM_SUB_VERSION_ID 0x100210 > +#define QM_EQC_PF_DW0 0x1c00 > +#define QM_AEQC_PF_DW0 0x1c20 > + > struct acc_vf_data { > #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) > /* QM match information */ > -- > 2.24.0

1 0

Re: [PATCH v5 2/3] migration: qm updates BAR configuration
by Shameerali Kolothum Thodi 08 Jul '25

08 Jul '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Monday, June 30, 2025 9:54 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v5 2/3] migration: qm updates BAR configuration > > On new platforms greater than QM_HW_V3, the configuration region for > the > live migration function of the accelerator device is no longer > placed in the VF, but is instead placed in the PF. > > Therefore, the configuration region of the live migration function > needs to be opened when the QM driver is loaded. When the QM driver > is uninstalled, the driver needs to clear this configuration. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/crypto/hisilicon/qm.c | 29 +++++++++++++++++++++++++++++ > 1 file changed, 29 insertions(+) > > diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c > index d3f5d108b898..0a8888304e15 100644 > --- a/drivers/crypto/hisilicon/qm.c > +++ b/drivers/crypto/hisilicon/qm.c > @@ -242,6 +242,9 @@ > #define QM_QOS_MAX_CIR_U 6 > #define QM_AUTOSUSPEND_DELAY 3000 > > +#define QM_MIG_REGION_SEL 0x100198 > +#define QM_MIG_REGION_EN 0x1 > + > /* abnormal status value for stopping queue */ > #define QM_STOP_QUEUE_FAIL 1 > #define QM_DUMP_SQC_FAIL 3 > @@ -3004,11 +3007,36 @@ static void qm_put_pci_res(struct hisi_qm *qm) > pci_release_mem_regions(pdev); > } > > +static void hisi_mig_region_clear(struct hisi_qm *qm) > +{ > + u32 val; > + > + /* Clear migration region set of PF */ > + if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) { > + val = readl(qm->io_base + QM_MIG_REGION_SEL); > + val &= ~BIT(0); > + writel(val, qm->io_base + QM_MIG_REGION_SEL); > + } > +} > + > +static void hisi_mig_region_enable(struct hisi_qm *qm) > +{ > + u32 val; > + > + /* Select migration region of PF */ > + if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) { > + val = readl(qm->io_base + QM_MIG_REGION_SEL); > + val |= QM_MIG_REGION_EN; > + writel(val, qm->io_base + QM_MIG_REGION_SEL); > + } > +} May adding a comment for above functions will be helpful. > + > static void hisi_qm_pci_uninit(struct hisi_qm *qm) > { > struct pci_dev *pdev = qm->pdev; > > pci_free_irq_vectors(pdev); > + hisi_mig_region_clear(qm); > qm_put_pci_res(qm); > pci_disable_device(pdev); > } > @@ -5630,6 +5658,7 @@ int hisi_qm_init(struct hisi_qm *qm) > goto err_free_qm_memory; > > qm_cmd_init(qm); > + hisi_mig_region_enable(qm); > > return 0; Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> You need to CC QM driver maintainers for this. Thanks, SHameer

1 0

Re: [PATCH v5 1/3] migration: update BAR space size
by Shameerali Kolothum Thodi 08 Jul '25

08 Jul '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Monday, June 30, 2025 9:54 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v5 1/3] migration: update BAR space size > > On new platforms greater than QM_HW_V3, the live migration > configuration > region is moved from VF to PF. The VF's own configuration space is > restored to the complete 64KB, and there is no need to divide the > size of the BAR configuration space equally. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- See one minor comment below. LGTM: Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 36 ++++++++++++++----- > 1 file changed, 27 insertions(+), 9 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 2149f49aeec7..1ddc9dbadb70 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -1250,6 +1250,28 @@ static struct hisi_qm *hisi_acc_get_pf_qm(struct > pci_dev *pdev) > return !IS_ERR(pf_qm) ? pf_qm : NULL; > } > > +static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev, > + unsigned int index) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > + hisi_acc_drvdata(vdev->pdev); > + > + /* > + * On the old QM_HW_V3 device, the ACC VF device BAR2 > + * region encompasses both functional register space > + * and migration control register space. > + * only the functional region should be report to Guest. > + * > + * On the new HW device, the migration control register > + * has been moved to the PF device BAR2 region. > + * The VF device BAR2 is entirely functional register space. > + */ Nit: May be move the above comment about new HW to the below check is better. > + if (hisi_acc_vdev->pf_qm->ver == QM_HW_V3) > + return (pci_resource_len(vdev->pdev, index) >> 1); > + > + return pci_resource_len(vdev->pdev, index); > +} > + > static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev, > size_t count, loff_t *ppos, > size_t *new_count) > @@ -1260,8 +1282,9 @@ static int hisi_acc_pci_rw_access_check(struct > vfio_device *core_vdev, > > if (index == VFIO_PCI_BAR2_REGION_INDEX) { > loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; > - resource_size_t end = pci_resource_len(vdev->pdev, index) / > 2; > + resource_size_t end; > > + end = hisi_acc_get_resource_len(vdev, index); > /* Check if access is for migration control region */ > if (pos >= end) > return -EINVAL; > @@ -1282,8 +1305,9 @@ static int hisi_acc_vfio_pci_mmap(struct > vfio_device *core_vdev, > index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT); > if (index == VFIO_PCI_BAR2_REGION_INDEX) { > u64 req_len, pgoff, req_start; > - resource_size_t end = pci_resource_len(vdev->pdev, index) / > 2; > + resource_size_t end; > > + end = hisi_acc_get_resource_len(vdev, index); > req_len = vma->vm_end - vma->vm_start; > pgoff = vma->vm_pgoff & > ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); > @@ -1330,7 +1354,6 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > if (cmd == VFIO_DEVICE_GET_REGION_INFO) { > struct vfio_pci_core_device *vdev = > container_of(core_vdev, struct vfio_pci_core_device, > vdev); > - struct pci_dev *pdev = vdev->pdev; > struct vfio_region_info info; > unsigned long minsz; > > @@ -1345,12 +1368,7 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > if (info.index == VFIO_PCI_BAR2_REGION_INDEX) { > info.offset = > VFIO_PCI_INDEX_TO_OFFSET(info.index); > > - /* > - * ACC VF dev BAR2 region consists of both > functional > - * register space and migration control register > space. > - * Report only the functional region to Guest. > - */ > - info.size = pci_resource_len(pdev, info.index) / 2; > + info.size = hisi_acc_get_resource_len(vdev, > info.index); > > info.flags = VFIO_REGION_INFO_FLAG_READ | > VFIO_REGION_INFO_FLAG_WRITE | > -- > 2.24.0

1 0

Re: [PATCH v4 3/3] migration: adapt to new migration configuration
by Shameerali Kolothum Thodi 24 Jun '25

24 Jun '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, June 10, 2025 7:33 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v4 3/3] migration: adapt to new migration configuration > > On the new hardware platform, the migration region has been > relocated from the VF to the PF. The driver must also be modified > accordingly to adapt to the new hardware device. > > Utilize the PF's I/O base directly on the new hardware platform, > and no mmap operation is required. If it is on an old platform, > the driver needs to be compatible with the old solution. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 166 ++++++++++++------ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + > 2 files changed, 120 insertions(+), 53 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index b16115f590fd..ab815fa4258c 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 > *addr) > return 0; > } > > +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) > { > struct device *dev = &qm->pdev->dev; > @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to read QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to read QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to write QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to write QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -522,6 +560,10 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return ret; > + > ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0); > if (ret) { > dev_err(dev, "set sqc failed\n"); > @@ -589,6 +631,10 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > vf_data->vf_qm_state = QM_READY; > hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > > + ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return ret; > + > ret = vf_qm_read_data(vf_qm, vf_data); > if (ret) > return ret; > @@ -1186,34 +1232,47 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > { > struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device; > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; > struct pci_dev *vf_dev = vdev->pdev; > > - /* > - * ACC VF dev BAR2 region consists of both functional register space > - * and migration control register space. For migration to work, we > - * need access to both. Hence, we map the entire BAR2 region here. > - * But unnecessarily exposing the migration BAR region to the Guest > - * has the potential to prevent/corrupt the Guest migration. Hence, > - * we restrict access to the migration control space from > - * Guest(Please see mmap/ioctl/read/write override functions). > - * > - * Please note that it is OK to expose the entire VF BAR if migration > - * is not supported or required as this cannot affect the ACC PF > - * configurations. > - * > - * Also the HiSilicon ACC VF devices supported by this driver on > - * HiSilicon hardware platforms are integrated end point devices > - * and the platform lacks the capability to perform any PCIe P2P > - * between these devices. > - */ > + if (pf_qm->ver == QM_HW_V3) { > + /* > + * ACC VF dev BAR2 region consists of both functional > register space > + * and migration control register space. For migration to > work, we > + * need access to both. Hence, we map the entire BAR2 > region here. > + * But unnecessarily exposing the migration BAR region to > the Guest > + * has the potential to prevent/corrupt the Guest migration. > Hence, > + * we restrict access to the migration control space from > + * Guest(Please see mmap/ioctl/read/write override > functions). > + * > + * Please note that it is OK to expose the entire VF BAR if > migration > + * is not supported or required as this cannot affect the ACC > PF > + * configurations. > + * > + * Also the HiSilicon ACC VF devices supported by this driver > on > + * HiSilicon hardware platforms are integrated end point > devices > + * and the platform lacks the capability to perform any PCIe > P2P > + * between these devices. > + */ > > - vf_qm->io_base = > - ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > - pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > - if (!vf_qm->io_base) > - return -EIO; > + vf_qm->io_base = > + ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > + pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > + if (!vf_qm->io_base) > + return -EIO; > > - vf_qm->fun_type = QM_HW_VF; > + vf_qm->fun_type = QM_HW_VF; > + vf_qm->ver = pf_qm->ver; > + } else { > + /* > + * On the new HW device, the migration function register is What is the "new HW device"? Please make it specific. Also I think you should check it specifically. One more question is what will happen if you try to migrate between a QM_HW_V3 <--> QM_HW_V4? Will it succeed or do we need to block it? Thanks, Shameer > placed > + * in the BAR2 configuration region of the PF, and each VF > device > + * occupies 8KB of configuration space. > + */ > + vf_qm->io_base = pf_qm->io_base + > QM_MIG_REGION_OFFSET + > + hisi_acc_vdev->vf_id * > QM_MIG_REGION_SIZE; > + vf_qm->fun_type = QM_HW_PF; > + } > vf_qm->pdev = vf_dev; > mutex_init(&vf_qm->mailbox_lock); > > @@ -1539,7 +1598,8 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > hisi_acc_vf_disable_fds(hisi_acc_vdev); > mutex_lock(&hisi_acc_vdev->open_mutex); > hisi_acc_vdev->dev_opened = false; > - iounmap(vf_qm->io_base); > + if (vf_qm->ver == QM_HW_V3) > + iounmap(vf_qm->io_base); > mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_close_device(core_vdev); > } > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 91002ceeebc1..348f8bb5b42c 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -59,6 +59,13 @@ > #define ACC_DEV_MAGIC_V1 0XCDCDCDCDFEEDAACC > #define ACC_DEV_MAGIC_V2 0xAACCFEEDDECADEDE > > +#define QM_MIG_REGION_OFFSET 0x180000 > +#define QM_MIG_REGION_SIZE 0x2000 > + > +#define QM_SUB_VERSION_ID 0x100210 > +#define QM_EQC_PF_DW0 0x1c00 > +#define QM_AEQC_PF_DW0 0x1c20 > + > struct acc_vf_data { > #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) > /* QM match information */ > -- > 2.24.0

1 0

Re: [PATCH v4 2/3] migration: qm updates BAR configuration
by Shameerali Kolothum Thodi 24 Jun '25

24 Jun '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, June 10, 2025 7:33 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v4 2/3] migration: qm updates BAR configuration > > On the new hardware platform, the configuration region for the > live migration function of the accelerator device is no longer > placed in the VF, but is instead placed in the PF. > > Therefore, the configuration region of the live migration function > needs to be opened when the QM driver is loaded. When the QM driver > is uninstalled, the driver needs to clear this configuration. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/crypto/hisilicon/qm.c | 29 +++++++++++++++++++++++++++++ > 1 file changed, 29 insertions(+) > > diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c > index d3f5d108b898..0a8888304e15 100644 > --- a/drivers/crypto/hisilicon/qm.c > +++ b/drivers/crypto/hisilicon/qm.c > @@ -242,6 +242,9 @@ > #define QM_QOS_MAX_CIR_U 6 > #define QM_AUTOSUSPEND_DELAY 3000 > > +#define QM_MIG_REGION_SEL 0x100198 > +#define QM_MIG_REGION_EN 0x1 > + > /* abnormal status value for stopping queue */ > #define QM_STOP_QUEUE_FAIL 1 > #define QM_DUMP_SQC_FAIL 3 > @@ -3004,11 +3007,36 @@ static void qm_put_pci_res(struct hisi_qm *qm) > pci_release_mem_regions(pdev); > } > > +static void hisi_mig_region_clear(struct hisi_qm *qm) > +{ > + u32 val; > + > + /* Clear migration region set of PF */ > + if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) { Is this going to be same for all future hardware's like OM_HW_V5, OM_HW_V6 etc? Otherwise it is better you make it specific to OM_HW_V4. I think the above checking is repeated throughout this series and there is no guarantee that future hardware will have the same changes only. So better make it specific. Thanks, Shameer > + val = readl(qm->io_base + QM_MIG_REGION_SEL); > + val &= ~BIT(0); > + writel(val, qm->io_base + QM_MIG_REGION_SEL); > + } > +} > + > +static void hisi_mig_region_enable(struct hisi_qm *qm) > +{ > + u32 val; > + > + /* Select migration region of PF */ > + if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) { > + val = readl(qm->io_base + QM_MIG_REGION_SEL); > + val |= QM_MIG_REGION_EN; > + writel(val, qm->io_base + QM_MIG_REGION_SEL); > + } > +} > + > static void hisi_qm_pci_uninit(struct hisi_qm *qm) > { > struct pci_dev *pdev = qm->pdev; > > pci_free_irq_vectors(pdev); > + hisi_mig_region_clear(qm); > qm_put_pci_res(qm); > pci_disable_device(pdev); > } > @@ -5630,6 +5658,7 @@ int hisi_qm_init(struct hisi_qm *qm) > goto err_free_qm_memory; > > qm_cmd_init(qm); > + hisi_mig_region_enable(qm); > > return 0; > > -- > 2.24.0

1 0

Re: [PATCH v4 1/3] migration: update BAR space size
by Shameerali Kolothum Thodi 24 Jun '25

24 Jun '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, June 10, 2025 7:33 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v4 1/3] migration: update BAR space size > > On the new hardware platform, the live migration configuration region > is moved from VF to PF. The VF's own configuration space is > restored to the complete 64KB, and there is no need to divide the > size of the BAR configuration space equally. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 36 ++++++++++++++----- > 1 file changed, 27 insertions(+), 9 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 2149f49aeec7..b16115f590fd 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -1250,6 +1250,28 @@ static struct hisi_qm *hisi_acc_get_pf_qm(struct > pci_dev *pdev) > return !IS_ERR(pf_qm) ? pf_qm : NULL; > } > > +static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev, > + unsigned int index) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > + hisi_acc_drvdata(vdev->pdev); > + > + /* > + * On the old HW_V3 device, the ACC VF device BAR2 > + * region encompasses both functional register space > + * and migration control register space. > + * only the functional region should be report to Guest. > + * > + * On the new HW device, the migration control register > + * has been moved to the PF device BAR2 region. > + * The VF device BAR2 is entirely functional register space. > + */ > + if (hisi_acc_vdev->pf_qm->ver == QM_HW_V3) > + return (pci_resource_len(vdev->pdev, index) >> 1); > + > + return pci_resource_len(vdev->pdev, index); > +} > + > static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev, > size_t count, loff_t *ppos, > size_t *new_count) > @@ -1260,8 +1282,9 @@ static int hisi_acc_pci_rw_access_check(struct > vfio_device *core_vdev, > > if (index == VFIO_PCI_BAR2_REGION_INDEX) { > loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; > - resource_size_t end = pci_resource_len(vdev->pdev, index) / > 2; > + resource_size_t end; > > + end = hisi_acc_get_resource_len(vdev, index); > /* Check if access is for migration control region */ > if (pos >= end) > return -EINVAL; > @@ -1282,8 +1305,9 @@ static int hisi_acc_vfio_pci_mmap(struct > vfio_device *core_vdev, > index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT); > if (index == VFIO_PCI_BAR2_REGION_INDEX) { > u64 req_len, pgoff, req_start; > - resource_size_t end = pci_resource_len(vdev->pdev, index) / > 2; > + resource_size_t end; > > + end = PAGE_ALIGN(hisi_acc_get_resource_len(vdev, index)); I think I have commented on this before. The above PAGE_ALIGN will change the behavior on HW_V3 with 64K PAGE_SIZE kernel. The end will become 64K which is not what we want on HW_V3. Could you please check that again. Thanks, Shameer > req_len = vma->vm_end - vma->vm_start; > pgoff = vma->vm_pgoff & > ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); > @@ -1330,7 +1354,6 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > if (cmd == VFIO_DEVICE_GET_REGION_INFO) { > struct vfio_pci_core_device *vdev = > container_of(core_vdev, struct vfio_pci_core_device, > vdev); > - struct pci_dev *pdev = vdev->pdev; > struct vfio_region_info info; > unsigned long minsz; > > @@ -1345,12 +1368,7 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > if (info.index == VFIO_PCI_BAR2_REGION_INDEX) { > info.offset = > VFIO_PCI_INDEX_TO_OFFSET(info.index); > > - /* > - * ACC VF dev BAR2 region consists of both > functional > - * register space and migration control register > space. > - * Report only the functional region to Guest. > - */ > - info.size = pci_resource_len(pdev, info.index) / 2; > + info.size = hisi_acc_get_resource_len(vdev, > info.index); > > info.flags = VFIO_REGION_INFO_FLAG_READ | > VFIO_REGION_INFO_FLAG_WRITE | > -- > 2.24.0

1 0

Re: [PATCH v7 6/6] hisi_acc_vfio_pci: update function return values.
by Shameerali Kolothum Thodi 15 Apr '25

15 Apr '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Friday, April 11, 2025 4:59 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v7 6/6] hisi_acc_vfio_pci: update function return values. > > In this driver file, many functions call sub-functions and use ret > to store the error code of the sub-functions. > However, instead of directly returning ret to the caller, they use a > converted error code, which prevents the end-user from clearly > understanding the root cause of the error. > Therefore, the code needs to be modified to directly return the error > code from the sub-functions. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 16 ++++++++-------- > 1 file changed, 8 insertions(+), 8 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index d12a350440d3..c63e302ac092 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -392,7 +392,7 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > ret = vf_qm_version_check(vf_data, dev); > if (ret) { > dev_err(dev, "failed to match ACC_DEV_MAGIC\n"); > - return -EINVAL; > + return ret; > } > > if (vf_data->dev_id != hisi_acc_vdev->vf_dev->device) { > @@ -404,7 +404,7 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > ret = qm_get_vft(vf_qm, &vf_qm->qp_base); > if (ret <= 0) { > dev_err(dev, "failed to get vft qp nums\n"); > - return -EINVAL; > + return ret; > } > > if (ret != vf_data->qp_num) { > @@ -501,7 +501,7 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > ret = qm_write_regs(qm, QM_VF_STATE, &vf_data->vf_qm_state, 1); > if (ret) { > dev_err(dev, "failed to write QM_VF_STATE\n"); > - return -EINVAL; > + return ret; > } > hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > > @@ -542,7 +542,7 @@ static int vf_qm_read_data(struct hisi_qm *vf_qm, > struct acc_vf_data *vf_data) > > ret = qm_get_regs(vf_qm, vf_data); > if (ret) > - return -EINVAL; > + return ret; > > /* Every reg is 32 bit, the dma address is 64 bit. */ > vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; > @@ -556,13 +556,13 @@ static int vf_qm_read_data(struct hisi_qm > *vf_qm, struct acc_vf_data *vf_data) > ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > if (ret) { > dev_err(dev, "failed to read SQC addr!\n"); > - return -EINVAL; > + return ret; > } > > ret = qm_get_cqc(vf_qm, &vf_data->cqc_dma); > if (ret) { > dev_err(dev, "failed to read CQC addr!\n"); > - return -EINVAL; > + return ret; > } > > return 0; > @@ -588,7 +588,7 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > > ret = vf_qm_read_data(vf_qm, vf_data); > if (ret) > - return -EINVAL; > + return ret; > > migf->total_length = sizeof(struct acc_vf_data); > /* Save eqc and aeqc interrupt information */ > @@ -1379,7 +1379,7 @@ static int hisi_acc_vf_debug_check(struct seq_file > *seq, struct vfio_device *vde > ret = qm_wait_dev_not_ready(vf_qm); > if (ret) { > seq_puts(seq, "VF device not ready!\n"); > - return -EBUSY; > + return ret; > } > > return 0; Any reason you avoided few other instances here? 1. qemu_set_regs() --> hisi_qm_wait_mb_ready() -->ret -EBUSY; 2. vf_qm_cache_wb() ret -EINVAL on -ETIMEOUT. With the above addressed, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v7 5/6] hisi_acc_vfio_pci: bugfix live migration function without VF device driver
by Shameerali Kolothum Thodi 15 Apr '25

15 Apr '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Friday, April 11, 2025 4:59 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v7 5/6] hisi_acc_vfio_pci: bugfix live migration function > without VF device driver > > If the VF device driver is not loaded in the Guest OS and we attempt to > perform device data migration, the address of the migrated data will > be NULL. > The live migration recovery operation on the destination side will > access a null address value, which will cause access errors. > > Therefore, live migration of VMs without added VF device drivers > does not require device data migration. > In addition, when the queue address data obtained by the destination > is empty, device queue recovery processing will not be performed. > > Fixes: b0eed085903e ("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > Reviewed-by: Shameer Kolothum > <shameerali.kolothum.thodi(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 22 +++++++++++++------ > 1 file changed, 15 insertions(+), 7 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index cadc82419dca..d12a350440d3 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -426,13 +426,6 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return -EINVAL; > } > > - ret = qm_write_regs(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, > 1); > - if (ret) { > - dev_err(dev, "failed to write QM_VF_STATE\n"); > - return ret; > - } > - > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > hisi_acc_vdev->match_done = true; > return 0; > } > @@ -498,6 +491,20 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > if (migf->total_length < sizeof(struct acc_vf_data)) > return -EINVAL; > > + if (!vf_data->eqe_dma || !vf_data->aeqe_dma || > + !vf_data->sqc_dma || !vf_data->cqc_dma) { > + dev_info(dev, "resume dma addr is NULL!\n"); > + hisi_acc_vdev->vf_qm_state = QM_NOT_READY; > + return 0; > + } I am still not fully understood why we need the above check. The only case as far as I can think of where this will happen is when the source VM Guest has not loaded the ACC driver. And we do take care of that already using vf_qm_state and checking total_length == QM_MATCH_SIZE. Have you seen this happening in any other scenario during your tests? Thanks, Shameer > + > + ret = qm_write_regs(qm, QM_VF_STATE, &vf_data->vf_qm_state, 1); > + if (ret) { > + dev_err(dev, "failed to write QM_VF_STATE\n"); > + return -EINVAL; > + } > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + > qm->eqe_dma = vf_data->eqe_dma; > qm->aeqe_dma = vf_data->aeqe_dma; > qm->sqc_dma = vf_data->sqc_dma; > @@ -1531,6 +1538,7 @@ static int hisi_acc_vfio_pci_migrn_init_dev(struct > vfio_device *core_vdev) > hisi_acc_vdev->vf_id = pci_iov_vf_id(pdev) + 1; > hisi_acc_vdev->pf_qm = pf_qm; > hisi_acc_vdev->vf_dev = pdev; > + hisi_acc_vdev->vf_qm_state = QM_NOT_READY; > mutex_init(&hisi_acc_vdev->state_mutex); > mutex_init(&hisi_acc_vdev->open_mutex); > > -- > 2.24.0

1 0

Re: [PATCH v5 5/5] hisi_acc_vfio_pci: bugfix live migration function without VF device driver
by Shameerali Kolothum Thodi 14 Mar '25

14 Mar '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, March 13, 2025 7:20 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v5 5/5] hisi_acc_vfio_pci: bugfix live migration function > without VF device driver > > If the VF device driver is not loaded in the Guest OS and we attempt to > perform device data migration, the address of the migrated data will > be NULL. > The live migration recovery operation on the destination side will > access a null address value, which will cause access errors. > > Therefore, live migration of VMs without added VF device drivers > does not require device data migration. > In addition, when the queue address data obtained by the destination > is empty, device queue recovery processing will not be performed. > > Fixes: b0eed085903e ("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 25 +++++++++++++------ > 1 file changed, 18 insertions(+), 7 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index cadc82419dca..44fa2d16bbcc 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -426,13 +426,6 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return -EINVAL; > } > > - ret = qm_write_regs(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, > 1); > - if (ret) { > - dev_err(dev, "failed to write QM_VF_STATE\n"); > - return ret; > - } > - > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > hisi_acc_vdev->match_done = true; > return 0; > } > @@ -498,6 +491,13 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > if (migf->total_length < sizeof(struct acc_vf_data)) > return -EINVAL; > > + ret = qm_write_regs(qm, QM_VF_STATE, &vf_data->vf_qm_state, 1); > + if (ret) { > + dev_err(dev, "failed to write QM_VF_STATE\n"); > + return -EINVAL; > + } > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + > qm->eqe_dma = vf_data->eqe_dma; > qm->aeqe_dma = vf_data->aeqe_dma; > qm->sqc_dma = vf_data->sqc_dma; > @@ -506,6 +506,12 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > qm->qp_base = vf_data->qp_base; > qm->qp_num = vf_data->qp_num; > > + if (!vf_data->eqe_dma || !vf_data->aeqe_dma || > + !vf_data->sqc_dma || !vf_data->cqc_dma) { > + dev_err(dev, "resume dma addr is NULL!\n"); > + return -EINVAL; > + } > + > ret = qm_set_regs(qm, vf_data); > if (ret) { > dev_err(dev, "set VF regs failed\n"); > @@ -726,8 +732,12 @@ static int hisi_acc_vf_load_state(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > { > struct device *dev = &hisi_acc_vdev->vf_dev->dev; > struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev- > >resuming_migf; > + struct acc_vf_data *vf_data = &migf->vf_data; > int ret; > > + if (vf_data->vf_qm_state != QM_READY) > + return 0; I don't think we need to check the above. In vf_qm_satte_save(), If vf_qm_state != QM_READY, we set the migf->total_length = QM_MATCH_SIZE. Hence it will return 0 in the below vf_qm_load_data() anyway. With that corrected, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer > + > /* Recover data to VF */ > ret = vf_qm_load_data(hisi_acc_vdev, migf); > if (ret) { > @@ -1531,6 +1541,7 @@ static int hisi_acc_vfio_pci_migrn_init_dev(struct > vfio_device *core_vdev) > hisi_acc_vdev->vf_id = pci_iov_vf_id(pdev) + 1; > hisi_acc_vdev->pf_qm = pf_qm; > hisi_acc_vdev->vf_dev = pdev; > + hisi_acc_vdev->vf_qm_state = QM_NOT_READY; > mutex_init(&hisi_acc_vdev->state_mutex); > mutex_init(&hisi_acc_vdev->open_mutex); > > -- > 2.24.0

1 0

Re: [PATCH v5 1/5] hisi_acc_vfio_pci: fix XQE dma address error
by Shameerali Kolothum Thodi 14 Mar '25

14 Mar '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, March 13, 2025 7:20 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v5 1/5] hisi_acc_vfio_pci: fix XQE dma address error > > The dma addresses of EQE and AEQE are wrong after migration and > results in guest kernel-mode encryption services failure. > Comparing the definition of hardware registers, we found that > there was an error when the data read from the register was > combined into an address. Therefore, the address combination > sequence needs to be corrected. > > Even after fixing the above problem, we still have an issue > where the Guest from an old kernel can get migrated to > new kernel and may result in wrong data. > > In order to ensure that the address is correct after migration, > if an old magic number is detected, the dma address needs to be > updated. > > Fixes: b0eed085903e ("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> LGTM, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 41 ++++++++++++++++--- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 14 ++++++- > 2 files changed, 47 insertions(+), 8 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 451c639299eb..304dbdfa0e95 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -350,6 +350,32 @@ static int vf_qm_func_stop(struct hisi_qm *qm) > return hisi_qm_mb(qm, QM_MB_CMD_PAUSE_QM, 0, 0, 0); > } > > +static int vf_qm_version_check(struct acc_vf_data *vf_data, struct device > *dev) > +{ > + switch (vf_data->acc_magic) { > + case ACC_DEV_MAGIC_V2: > + if (vf_data->major_ver != ACC_DRV_MAJOR_VER) { > + dev_info(dev, "migration driver version<%u.%u> not > match!\n", > + vf_data->major_ver, vf_data->minor_ver); > + return -EINVAL; > + } > + break; > + case ACC_DEV_MAGIC_V1: > + /* Correct dma address */ > + vf_data->eqe_dma = vf_data- > >qm_eqc_dw[QM_XQC_ADDR_HIGH]; > + vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > + vf_data->eqe_dma |= vf_data- > >qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > + vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > + break; > + default: > + return -EINVAL; > + } > + > + return 0; > +} > + > static int vf_qm_check_match(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > struct hisi_acc_vf_migration_file *migf) > { > @@ -363,7 +389,8 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > if (migf->total_length < QM_MATCH_SIZE || hisi_acc_vdev- > >match_done) > return 0; > > - if (vf_data->acc_magic != ACC_DEV_MAGIC) { > + ret = vf_qm_version_check(vf_data, dev); > + if (ret) { > dev_err(dev, "failed to match ACC_DEV_MAGIC\n"); > return -EINVAL; > } > @@ -418,7 +445,9 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > int vf_id = hisi_acc_vdev->vf_id; > int ret; > > - vf_data->acc_magic = ACC_DEV_MAGIC; > + vf_data->acc_magic = ACC_DEV_MAGIC_V2; > + vf_data->major_ver = ACC_DRV_MAJOR_VER; > + vf_data->minor_ver = ACC_DRV_MINOR_VER; > /* Save device id */ > vf_data->dev_id = hisi_acc_vdev->vf_dev->device; > > @@ -496,12 +525,12 @@ static int vf_qm_read_data(struct hisi_qm > *vf_qm, struct acc_vf_data *vf_data) > return -EINVAL; > > /* Every reg is 32 bit, the dma address is 64 bit. */ > - vf_data->eqe_dma = vf_data->qm_eqc_dw[1]; > + vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->eqe_dma |= vf_data->qm_eqc_dw[0]; > - vf_data->aeqe_dma = vf_data->qm_aeqc_dw[1]; > + vf_data->eqe_dma |= vf_data->qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[0]; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > > /* Through SQC_BT/CQC_BT to get sqc and cqc address */ > ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 245d7537b2bc..91002ceeebc1 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -39,6 +39,9 @@ > #define QM_REG_ADDR_OFFSET 0x0004 > > #define QM_XQC_ADDR_OFFSET 32U > +#define QM_XQC_ADDR_LOW 0x1 > +#define QM_XQC_ADDR_HIGH 0x2 > + > #define QM_VF_AEQ_INT_MASK 0x0004 > #define QM_VF_EQ_INT_MASK 0x000c > #define QM_IFC_INT_SOURCE_V 0x0020 > @@ -50,10 +53,15 @@ > #define QM_EQC_DW0 0X8000 > #define QM_AEQC_DW0 0X8020 > > +#define ACC_DRV_MAJOR_VER 1 > +#define ACC_DRV_MINOR_VER 0 > + > +#define ACC_DEV_MAGIC_V1 0XCDCDCDCDFEEDAACC > +#define ACC_DEV_MAGIC_V2 0xAACCFEEDDECADEDE > + > struct acc_vf_data { > #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) > /* QM match information */ > -#define ACC_DEV_MAGIC 0XCDCDCDCDFEEDAACC > u64 acc_magic; > u32 qp_num; > u32 dev_id; > @@ -61,7 +69,9 @@ struct acc_vf_data { > u32 qp_base; > u32 vf_qm_state; > /* QM reserved match information */ > - u32 qm_rsv_state[3]; > + u16 major_ver; > + u16 minor_ver; > + u32 qm_rsv_state[2]; > > /* QM RW regs */ > u32 aeq_int_mask; > -- > 2.24.0

1 0

Re: [PATCH v2 0/3] update live migration configuration region
by Shameerali Kolothum Thodi 04 Mar '25

04 Mar '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, March 4, 2025 12:05 PM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 0/3] update live migration configuration region > > On the new hardware platform, the configuration register space > of the live migration function is set on the PF, while on the > old platform, this part is placed on the VF. > > Change v1 -> v2 > Delete the vf_qm_state read operation in Pre_Copy If I understand correctly, previously this was discussed in your bug fix series here, https://lore.kernel.org/all/fa8cd8c1cdbe4849b445ffd8f4894515@huawei.com/ And why we are having it here in this new hardware support series now? Could we please move all the existing bug fixes in one series and support for new platform in another series, please. Thanks, Shameer

1 0

Re: [PATCH v4 5/5] hisi_acc_vfio_pci: bugfix live migration function without VF device driver
by Shameerali Kolothum Thodi 26 Feb '25

26 Feb '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, February 25, 2025 6:28 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v4 5/5] hisi_acc_vfio_pci: bugfix live migration function > without VF device driver > > If the driver of the VF device is not loaded in the Guest OS, > then perform device data migration. The migrated data address will > be NULL. May be rephrase: If the VF device driver is not loaded in the Guest OS and we attempt to perform device data migration, the address of the migrated data will be NULL. > The live migration recovery operation on the destination side will > access a null address value, which will cause access errors. > Therefore, live migration of VMs without added VF device drivers > does not require device data migration. > In addition, when the queue address data obtained by the destination > is empty, device queue recovery processing will not be performed. > > Fixes: b0eed085903e ("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 18 ++++++++++++++++++ > 1 file changed, 18 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 3f0bcd855839..77872fc4cd34 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -440,6 +440,7 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > struct acc_vf_data *vf_data) > { > struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > struct device *dev = &pf_qm->pdev->dev; > int vf_id = hisi_acc_vdev->vf_id; > int ret; > @@ -466,6 +467,13 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + /* Get VF driver insmod state */ > + ret = qm_read_regs(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, > 1); We already have qm_wait_dev_not_ready() function that checks the QM_VF_STATE. Why can't we use that here? Also we are getting this vf_qm_state already in vf_qm_state_save(). And you don't seem to check the vf_qm_state in vf_qm_check_match(). So why it is read early in this function? Thanks, Shameer > + if (ret) { > + dev_err(dev, "failed to read QM_VF_STATE!\n"); > + return ret; > + } > + > return 0; > } > > @@ -505,6 +513,12 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > qm->qp_base = vf_data->qp_base; > qm->qp_num = vf_data->qp_num; > > + if (!vf_data->eqe_dma || !vf_data->aeqe_dma || > + !vf_data->sqc_dma || !vf_data->cqc_dma) { > + dev_err(dev, "resume dma addr is NULL!\n"); > + return -EINVAL; > + } > + > ret = qm_set_regs(qm, vf_data); > if (ret) { > dev_err(dev, "set VF regs failed\n"); > @@ -727,6 +741,9 @@ static int hisi_acc_vf_load_state(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev- > >resuming_migf; > int ret; > > + if (hisi_acc_vdev->vf_qm_state != QM_READY) > + return 0; > + > /* Recover data to VF */ > ret = vf_qm_load_data(hisi_acc_vdev, migf); > if (ret) { > @@ -1530,6 +1547,7 @@ static int hisi_acc_vfio_pci_migrn_init_dev(struct > vfio_device *core_vdev) > hisi_acc_vdev->vf_id = pci_iov_vf_id(pdev) + 1; > hisi_acc_vdev->pf_qm = pf_qm; > hisi_acc_vdev->vf_dev = pdev; > + hisi_acc_vdev->vf_qm_state = QM_NOT_READY; > mutex_init(&hisi_acc_vdev->state_mutex); > mutex_init(&hisi_acc_vdev->open_mutex); > > -- > 2.24.0

1 0

Re: [PATCH v4 1/5] hisi_acc_vfio_pci: fix XQE dma address error
by Shameerali Kolothum Thodi 26 Feb '25

26 Feb '25

> -----Original Message----- > From: Alex Williamson <alex.williamson(a)redhat.com> > Sent: Wednesday, February 26, 2025 12:10 AM > To: liulongfang <liulongfang(a)huawei.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v4 1/5] hisi_acc_vfio_pci: fix XQE dma address error > > On Tue, 25 Feb 2025 14:27:53 +0800 > Longfang Liu <liulongfang(a)huawei.com> wrote: > > > The dma addresses of EQE and AEQE are wrong after migration and > > results in guest kernel-mode encryption services failure. > > Comparing the definition of hardware registers, we found that > > there was an error when the data read from the register was > > combined into an address. Therefore, the address combination > > sequence needs to be corrected. > > > > Even after fixing the above problem, we still have an issue > > where the Guest from an old kernel can get migrated to > > new kernel and may result in wrong data. > > > > In order to ensure that the address is correct after migration, > > if an old magic number is detected, the dma address needs to be > > updated. > > > > Fixes: b0eed085903e ("hisi_acc_vfio_pci: Add support for VFIO live > migration") > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > > --- > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 40 ++++++++++++++++--- > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 14 ++++++- > > 2 files changed, 46 insertions(+), 8 deletions(-) > > > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > index 451c639299eb..35316984089b 100644 > > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > @@ -350,6 +350,31 @@ static int vf_qm_func_stop(struct hisi_qm *qm) > > return hisi_qm_mb(qm, QM_MB_CMD_PAUSE_QM, 0, 0, 0); > > } > > > > +static int vf_qm_version_check(struct acc_vf_data *vf_data, struct device > *dev) > > +{ > > + switch (vf_data->acc_magic) { > > + case ACC_DEV_MAGIC_V2: > > + if (vf_data->major_ver < ACC_DRV_MAJOR_VER || > > + vf_data->minor_ver < ACC_DRV_MINOR_VER) > > + dev_info(dev, "migration driver version not > match!\n"); > > + return -EINVAL; > > + break; > > What's your major/minor update strategy? > > Note that minor_ver is a u16 and ACC_DRV_MINOR_VER is defined as 0, so > testing less than 0 against an unsigned is useless. > > Arguably testing major and minor independently is pretty useless too. > > You're defining what a future "old" driver version will accept, which > is very nearly anything, so any breaking change *again* requires a new > magic, so we're accomplishing very little here. > > Maybe you want to consider a strategy where you'd increment the major > number for a breaking change and minor for a compatible feature. In > that case you'd want to verify the major_ver matches > ACC_DRV_MAJOR_VER > exactly and minor_ver would be more of a feature level. Agree. I think the above check should be just major_ver != ACC_DRV_MAJOR_VER and we can make use of minor version whenever we need a specific handling for a feature support. Also I think it would be good to print the version info above in case of mismatch. > > > + case ACC_DEV_MAGIC_V1: > > + /* Correct dma address */ > > + vf_data->eqe_dma = vf_data- > >qm_eqc_dw[QM_XQC_ADDR_HIGH]; > > + vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > > + vf_data->eqe_dma |= vf_data- > >qm_eqc_dw[QM_XQC_ADDR_LOW]; > > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > > + vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > > + break; > > + default: > > + return -EINVAL; > > + } > > + > > + return 0; > > +} > > + > > static int vf_qm_check_match(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > > struct hisi_acc_vf_migration_file *migf) > > { > > @@ -363,7 +388,8 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > > if (migf->total_length < QM_MATCH_SIZE || hisi_acc_vdev- > >match_done) > > return 0; > > > > - if (vf_data->acc_magic != ACC_DEV_MAGIC) { > > + ret = vf_qm_version_check(vf_data, dev); > > + if (ret) { > > dev_err(dev, "failed to match ACC_DEV_MAGIC\n"); > > return -EINVAL; > > } > > @@ -418,7 +444,9 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > > int vf_id = hisi_acc_vdev->vf_id; > > int ret; > > > > - vf_data->acc_magic = ACC_DEV_MAGIC; > > + vf_data->acc_magic = ACC_DEV_MAGIC_V2; > > + vf_data->major_ver = ACC_DRV_MAR; > > + vf_data->minor_ver = ACC_DRV_MIN; > > Where are "MAR" and "MIN" defined? I can't see how this would even > compile. Thanks, Yes. Please make sure to do a compile test and run basic sanity tested even if you think the changes are minor. Chances are there that you will miss out things like this. Thanks, Shameer

1 0

Re: [PATCH 3/3] migration: adapt to new migration configuration
by Shameerali Kolothum Thodi 18 Feb '25

18 Feb '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, February 18, 2025 2:15 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH 3/3] migration: adapt to new migration configuration > > On the new hardware platform, the migration region is moved from > VF to PF. the driver also needs to be modified accordingly to adapt > to the new hardware device. > > Use PF's io base directly on the new hardware platform. and no mmap > operation is required. > If it is on an old platform, the driver needs to be compatible with > the old solution. I see that you have used "new hardware platform" everywhere and I think it is better we make it more specific, like QM_HW_V4? Also it is not clear to me, what happens if you try migration between QM_HW_V3 <--> QM_HW_V4? Is that going to be success or we are failing it somewhere? I see only the dev_id check in vf_qm_check_match(). Thanks, Shameer > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 165 ++++++++++++------ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + > 2 files changed, 119 insertions(+), 53 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 599905dbb707..cf5a807c2199 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 > *addr) > return 0; > } > > +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to read QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct acc_vf_data *vf_data) > +{ > + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &qm->pdev->dev; > + u32 eqc_addr, aeqc_addr; > + int ret; > + > + if (qm->ver == QM_HW_V3) { > + eqc_addr = QM_EQC_DW0; > + aeqc_addr = QM_AEQC_DW0; > + } else { > + eqc_addr = QM_EQC_PF_DW0; > + aeqc_addr = QM_AEQC_PF_DW0; > + } > + > + /* QM_EQC_DW has 7 regs */ > + ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_EQC_DW\n"); > + return ret; > + } > + > + /* QM_AEQC_DW has 7 regs */ > + ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); > + if (ret) { > + dev_err(dev, "failed to write QM_AEQC_DW\n"); > + return ret; > + } > + > + return 0; > +} > + > static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) > { > struct device *dev = &qm->pdev->dev; > @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to read QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to read QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -238,20 +290,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct > acc_vf_data *vf_data) > return ret; > } > > - /* QM_EQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); > - if (ret) { > - dev_err(dev, "failed to write QM_EQC_DW\n"); > - return ret; > - } > - > - /* QM_AEQC_DW has 7 regs */ > - ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, > 7); > - if (ret) { > - dev_err(dev, "failed to write QM_AEQC_DW\n"); > - return ret; > - } > - > return 0; > } > > @@ -470,6 +508,10 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return ret; > + > ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0); > if (ret) { > dev_err(dev, "set sqc failed\n"); > @@ -544,6 +586,10 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data); > + if (ret) > + return -EINVAL; > + > ret = vf_qm_read_data(vf_qm, vf_data); > if (ret) > return -EINVAL; > @@ -1131,34 +1177,46 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > { > struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device; > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; > struct pci_dev *vf_dev = vdev->pdev; > > - /* > - * ACC VF dev BAR2 region consists of both functional register space > - * and migration control register space. For migration to work, we > - * need access to both. Hence, we map the entire BAR2 region here. > - * But unnecessarily exposing the migration BAR region to the Guest > - * has the potential to prevent/corrupt the Guest migration. Hence, > - * we restrict access to the migration control space from > - * Guest(Please see mmap/ioctl/read/write override functions). > - * > - * Please note that it is OK to expose the entire VF BAR if migration > - * is not supported or required as this cannot affect the ACC PF > - * configurations. > - * > - * Also the HiSilicon ACC VF devices supported by this driver on > - * HiSilicon hardware platforms are integrated end point devices > - * and the platform lacks the capability to perform any PCIe P2P > - * between these devices. > - */ > + if (pf_qm->ver == QM_HW_V3) { > + /* > + * ACC VF dev BAR2 region consists of both functional > register space > + * and migration control register space. For migration to > work, we > + * need access to both. Hence, we map the entire BAR2 > region here. > + * But unnecessarily exposing the migration BAR region to > the Guest > + * has the potential to prevent/corrupt the Guest migration. > Hence, > + * we restrict access to the migration control space from > + * Guest(Please see mmap/ioctl/read/write override > functions). > + * > + * Please note that it is OK to expose the entire VF BAR if > migration > + * is not supported or required as this cannot affect the ACC > PF > + * configurations. > + * > + * Also the HiSilicon ACC VF devices supported by this driver > on > + * HiSilicon hardware platforms are integrated end point > devices > + * and the platform lacks the capability to perform any PCIe > P2P > + * between these devices. > + */ > > - vf_qm->io_base = > - ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > - pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > - if (!vf_qm->io_base) > - return -EIO; > + vf_qm->io_base = > + ioremap(pci_resource_start(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX), > + pci_resource_len(vf_dev, > VFIO_PCI_BAR2_REGION_INDEX)); > + if (!vf_qm->io_base) > + return -EIO; > > - vf_qm->fun_type = QM_HW_VF; > + vf_qm->fun_type = QM_HW_VF; > + vf_qm->ver = pf_qm->ver; > + } else { > + /* > + * In the new HW platform, the migration function register > space is in BAR2 of PF, > + * and each VF occupies 8KB address space. > + */ > + vf_qm->io_base = pf_qm->io_base + > QM_MIG_REGION_OFFSET + > + hisi_acc_vdev->vf_id * > QM_MIG_REGION_SIZE; > + vf_qm->fun_type = QM_HW_PF; > + } > vf_qm->pdev = vf_dev; > mutex_init(&vf_qm->mailbox_lock); > > @@ -1488,7 +1546,8 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > > mutex_lock(&hisi_acc_vdev->open_mutex); > hisi_acc_vdev->dev_opened = false; > - iounmap(vf_qm->io_base); > + if (vf_qm->ver == QM_HW_V3) > + iounmap(vf_qm->io_base); > mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_close_device(core_vdev); > } > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 245d7537b2bc..b01eb54525d3 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -50,6 +50,13 @@ > #define QM_EQC_DW0 0X8000 > #define QM_AEQC_DW0 0X8020 > > +#define QM_MIG_REGION_OFFSET 0x180000 > +#define QM_MIG_REGION_SIZE 0x2000 > + > +#define QM_SUB_VERSION_ID 0x100210 > +#define QM_EQC_PF_DW0 0x1c00 > +#define QM_AEQC_PF_DW0 0x1c20 > + > struct acc_vf_data { > #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) > /* QM match information */ > -- > 2.24.0

1 0

Re: [PATCH 1/3] migration: update BAR space size
by Shameerali Kolothum Thodi 18 Feb '25

18 Feb '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, February 18, 2025 2:15 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH 1/3] migration: update BAR space size > > On the new hardware platform, the live migration configuration region > is moved from VF to PF. The VF's own configuration space is > restored to the complete 64KB, and there is no need to divide the > size of the BAR configuration space equally. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 41 +++++++++++++++---- > 1 file changed, 32 insertions(+), 9 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 451c639299eb..599905dbb707 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -1195,6 +1195,33 @@ static struct hisi_qm *hisi_acc_get_pf_qm(struct > pci_dev *pdev) > return !IS_ERR(pf_qm) ? pf_qm : NULL; > } > > +static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev, > + unsigned int index) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > + hisi_acc_drvdata(vdev->pdev); > + > + /* > + * ACC VF dev 64KB BAR2 region consists of both functional > + * register space and migration control register space, each > + * uses 32KB BAR2 region, on the system with more than 64KB > + * page size, even if the migration control register space > + * is written by VM, it will only affects the VF. > + * > + * In order to support the live migration function in the > + * system with a page size above 64KB, the driver needs > + * to ensure that the VF region size is aligned with the > + * system page size. I didn't get this. Are you referring to kernel with 64K page size? And this is for new hardware or QM_HW_V3 one? > + * > + * On the new hardware platform, the live migration control register > + * has been moved from VF to PF. > + */ > + if (hisi_acc_vdev->pf_qm->ver == QM_HW_V3) > + return (pci_resource_len(vdev->pdev, index) >> 1); > + > + return pci_resource_len(vdev->pdev, index); > +} > + > static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev, > size_t count, loff_t *ppos, > size_t *new_count) > @@ -1205,8 +1232,9 @@ static int hisi_acc_pci_rw_access_check(struct > vfio_device *core_vdev, > > if (index == VFIO_PCI_BAR2_REGION_INDEX) { > loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; > - resource_size_t end = pci_resource_len(vdev->pdev, index) / > 2; > + resource_size_t end; > > + end = hisi_acc_get_resource_len(vdev, index); > /* Check if access is for migration control region */ > if (pos >= end) > return -EINVAL; > @@ -1227,8 +1255,9 @@ static int hisi_acc_vfio_pci_mmap(struct > vfio_device *core_vdev, > index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT); > if (index == VFIO_PCI_BAR2_REGION_INDEX) { > u64 req_len, pgoff, req_start; > - resource_size_t end = pci_resource_len(vdev->pdev, index) / > 2; > + resource_size_t end; > > + end = PAGE_ALIGN(hisi_acc_get_resource_len(vdev, index)); So here, the whole BAR2 will be mapped to Guest in case of QM_HW_V3 && 64K kernel as well, right? Thanks, Shameer > req_len = vma->vm_end - vma->vm_start; > pgoff = vma->vm_pgoff & > ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); > @@ -1275,7 +1304,6 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > if (cmd == VFIO_DEVICE_GET_REGION_INFO) { > struct vfio_pci_core_device *vdev = > container_of(core_vdev, struct vfio_pci_core_device, > vdev); > - struct pci_dev *pdev = vdev->pdev; > struct vfio_region_info info; > unsigned long minsz; > > @@ -1290,12 +1318,7 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > if (info.index == VFIO_PCI_BAR2_REGION_INDEX) { > info.offset = > VFIO_PCI_INDEX_TO_OFFSET(info.index); > > - /* > - * ACC VF dev BAR2 region consists of both > functional > - * register space and migration control register > space. > - * Report only the functional region to Guest. > - */ > - info.size = pci_resource_len(pdev, info.index) / 2; > + info.size = hisi_acc_get_resource_len(vdev, > info.index); > > info.flags = VFIO_REGION_INFO_FLAG_READ | > VFIO_REGION_INFO_FLAG_WRITE | > -- > 2.24.0

1 0

Re: [PATCH v2 1/5] hisi_acc_vfio_pci: fix XQE dma address error
by Shameerali Kolothum Thodi 14 Jan '25

14 Jan '25

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, January 14, 2025 3:18 AM > To: Alex Williamson <alex.williamson(a)redhat.com> > Cc: Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; jgg(a)nvidia.com; Jonathan > Cameron <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v2 1/5] hisi_acc_vfio_pci: fix XQE dma address error > [...] > > @@ -418,7 +440,7 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > > int vf_id = hisi_acc_vdev->vf_id; > > int ret; > > > > - vf_data->acc_magic = ACC_DEV_MAGIC; > > + vf_data->acc_magic = ACC_DEV_MAGIC_V2; > > /* Save device id */ > > vf_data->dev_id = hisi_acc_vdev->vf_dev->device; > > > > Thanks, > > Alex > > > >>> > >>> As for the compatibility issues between old and new versions in the > >>> future, we do not need to reserve version numbers to deal with them > >>> now. Because before encountering specific problems, our design may > be redundant. > >> > >> A magic value + version number would prevent the need to replace the > >> magic value every time an issue is encountered, which I think was > >> also Shameer's commit, which is not addressed by forcing the > >> formatting of a portion of the magic value. None of what you're > >> trying to do here precludes a new data structure for the new magic > >> value or defining the padding fields for different use cases. > >> Thanks, > >> > >> Alex > > > > If we want to use the original magic number, we also need to add the major > and minor version numbers. It does not cause compatibility issues and can > only reuse the original u64 memory space. > > This is also Shameerali's previous plan. Do you accept this plan? The suggestion here is to improve my previous plan.. ie, instead of overloading the V2 MAGIC with version info, introduce version(major:minor) separately. Something like, Rename old MAGIC as MAGIC_V1 Introduce new MAGIC as MAGIC_V2 Repurpose any padding or reserved fields in struct vf_data for major:minor version fields. The idea of introducing these major:minor is for future use where instead of coming up with a new MAGIC data every time we can increment the version for bug fixes and features if required. Hope this is clear now. Thanks, Shameer

1 0

Re: [PATCH v2 0/5] bugfix some driver issues
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

Hi Longfang, > -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 0/5] bugfix some driver issues > > As the test scenarios for the live migration function become > more and more extensive. Some previously undiscovered driver > issues were found. > Update and fix through this patchset. > A changes history will be beneficial to understand what changed from v1. Also I have a request.... As part of this series, could you please include the below fix to this driver as well as we do have the same potential problem. https://lore.kernel.org/all/20241021123843.42979-1-giovanni.cabiddu@intel.c… Thanks, Shameer > Longfang Liu (5): > hisi_acc_vfio_pci: fix XQE dma address error > hisi_acc_vfio_pci: add eq and aeq interruption restore > hisi_acc_vfio_pci: bugfix cache write-back issue > hisi_acc_vfio_pci: bugfix the problem of uninstalling driver > hisi_acc_vfio_pci: bugfix live migration function without VF device > driver > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 83 ++++++++++++++++--- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 9 +- > 2 files changed, 78 insertions(+), 14 deletions(-) > > -- > 2.24.0

1 0

Re: [PATCH v2 5/5] hisi_acc_vfio_pci: bugfix live migration function without VF device driver
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 5/5] hisi_acc_vfio_pci: bugfix live migration function > without VF device driver > > If the driver of the VF device is not loaded in the Guest OS, > then perform device data migration. The migrated data address will > be NULL. > The live migration recovery operation on the destination side will > access a null address value, which will cause access errors. > > Therefore, live migration of VMs without added VF device drivers > does not require device data migration. > In addition, when the queue address data obtained by the destination > is empty, device queue recovery processing will not be performed. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> Why this doesn't need a Fixes tag? > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 18 ++++++++++++++++++ > 1 file changed, 18 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 8d9e07ebf4fd..9a5f7e9bc695 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -436,6 +436,7 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > struct acc_vf_data *vf_data) > { > struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > struct device *dev = &pf_qm->pdev->dev; > int vf_id = hisi_acc_vdev->vf_id; > int ret; > @@ -460,6 +461,13 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return ret; > } > > + /* Get VF driver insmod state */ > + ret = qm_read_regs(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, > 1); > + if (ret) { > + dev_err(dev, "failed to read QM_VF_STATE!\n"); > + return ret; > + } > + > return 0; > } > > @@ -499,6 +507,12 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > qm->qp_base = vf_data->qp_base; > qm->qp_num = vf_data->qp_num; > > + if (!vf_data->eqe_dma || !vf_data->aeqe_dma || > + !vf_data->sqc_dma || !vf_data->cqc_dma) { > + dev_err(dev, "resume dma addr is NULL!\n"); > + return -EINVAL; > + } > + So this is to cover the corner case where the Guest has loaded the driver (QM_READY set) but not configured the DMA address? When this will happen? I thought we are setting QM_READY in guest after all configurations. Thanks, Shameer

1 0

Re: [PATCH v2 4/5] hisi_acc_vfio_pci: bugfix the problem of uninstalling driver
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 4/5] hisi_acc_vfio_pci: bugfix the problem of uninstalling > driver > > In a live migration scenario. If the number of VFs at the destination is > greater than the source, the recovery operation will fail and qemu will not > be able to complete the process and exit after shutting down the device FD. > > This will cause the driver to be unable to be unloaded normally due to > abnormal reference counting of the live migration driver caused by the > abnormal closing operation of fd. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index c057c0e24693..8d9e07ebf4fd 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -1501,6 +1501,7 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + hisi_acc_vf_disable_fds(hisi_acc_vdev); > mutex_lock(&hisi_acc_vdev->open_mutex); > hisi_acc_vdev->dev_opened = false; > iounmap(vf_qm->io_base); Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v2 3/5] hisi_acc_vfio_pci: bugfix cache write-back issue
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 3/5] hisi_acc_vfio_pci: bugfix cache write-back issue > > At present, cache write-back is placed in the device data > copy stage after stopping the device operation. > Writing back to the cache at this stage will cause the data > obtained by the cache to be written back to be empty. > > In order to ensure that the cache data is written back > successfully, the data needs to be written back into the > stop device stage. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 14 +++++++------- > 1 file changed, 7 insertions(+), 7 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 4c8f1ae5b636..c057c0e24693 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -559,7 +559,6 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > { > struct acc_vf_data *vf_data = &migf->vf_data; > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > - struct device *dev = &vf_qm->pdev->dev; > int ret; > > if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > @@ -573,12 +572,6 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > vf_data->vf_qm_state = QM_READY; > hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > > - ret = vf_qm_cache_wb(vf_qm); > - if (ret) { > - dev_err(dev, "failed to writeback QM Cache!\n"); > - return ret; > - } > - > ret = vf_qm_read_data(vf_qm, vf_data); > if (ret) > return -EINVAL; > @@ -1005,6 +998,13 @@ static int hisi_acc_vf_stop_device(struct > hisi_acc_vf_core_device *hisi_acc_vdev > dev_err(dev, "failed to check QM INT state!\n"); > return ret; > } > + > + ret = vf_qm_cache_wb(vf_qm); > + if (ret) { > + dev_err(dev, "failed to writeback QM cache!\n"); > + return ret; > + } > + > return 0; > } Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v2 2/5] hisi_acc_vfio_pci: add eq and aeq interruption restore
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 2/5] hisi_acc_vfio_pci: add eq and aeq interruption > restore > > In order to ensure that the task packets of the accelerator > device are not lost during the migration process, it is necessary > to send an EQ and AEQ command to the device after the live migration > is completed and to update the completion position of the task queue. > > Let the device recheck the completed tasks data and if there are > uncollected packets, device resend a task completion interrupt > to the software. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 8518efea3a52..4c8f1ae5b636 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -463,6 +463,19 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return 0; > } > > +static void vf_qm_xeqc_save(struct hisi_qm *qm, > + struct hisi_acc_vf_migration_file *migf) > +{ > + struct acc_vf_data *vf_data = &migf->vf_data; > + u16 eq_head, aeq_head; > + > + eq_head = vf_data->qm_eqc_dw[0] & 0xFFFF; > + qm_db(qm, 0, QM_DOORBELL_CMD_EQ, eq_head, 0); > + > + aeq_head = vf_data->qm_aeqc_dw[0] & 0xFFFF; > + qm_db(qm, 0, QM_DOORBELL_CMD_AEQ, aeq_head, 0); > +} > + > static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev, > struct hisi_acc_vf_migration_file *migf) > { > @@ -571,6 +584,9 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return -EINVAL; > > migf->total_length = sizeof(struct acc_vf_data); > + /* Save eqc and aeqc interrupt information */ > + vf_qm_xeqc_save(vf_qm, migf); > + > return 0; > } Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v2 1/5] hisi_acc_vfio_pci: fix XQE dma address error
by Shameerali Kolothum Thodi 19 Dec '24

19 Dec '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, December 19, 2024 9:18 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 1/5] hisi_acc_vfio_pci: fix XQE dma address error > > The dma addresses of EQE and AEQE are wrong after migration and > results in guest kernel-mode encryption services failure. > Comparing the definition of hardware registers, we found that > there was an error when the data read from the register was > combined into an address. Therefore, the address combination > sequence needs to be corrected. > > Even after fixing the above problem, we still have an issue > where the Guest from an old kernel can get migrated to > new kernel and may result in wrong data. > > In order to ensure that the address is correct after migration, > if an old magic number is detected, the dma address needs to be > updated. > > Fixes:b0eed085903e("hisi_acc_vfio_pci: Add support for VFIO live > migration") > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 34 +++++++++++++++---- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 9 ++++- > 2 files changed, 36 insertions(+), 7 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 451c639299eb..8518efea3a52 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -350,6 +350,27 @@ static int vf_qm_func_stop(struct hisi_qm *qm) > return hisi_qm_mb(qm, QM_MB_CMD_PAUSE_QM, 0, 0, 0); > } > > +static int vf_qm_magic_check(struct acc_vf_data *vf_data) > +{ > + switch (vf_data->acc_magic) { > + case ACC_DEV_MAGIC_V2: > + break; > + case ACC_DEV_MAGIC_V1: > + /* Correct dma address */ > + vf_data->eqe_dma = vf_data- > >qm_eqc_dw[QM_XQC_ADDR_HIGH]; > + vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > + vf_data->eqe_dma |= vf_data- > >qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > + vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > + break; > + default: > + return -EINVAL; > + } > + > + return 0; > +} > + > static int vf_qm_check_match(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > struct hisi_acc_vf_migration_file *migf) > { > @@ -363,7 +384,8 @@ static int vf_qm_check_match(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > if (migf->total_length < QM_MATCH_SIZE || hisi_acc_vdev- > >match_done) > return 0; > > - if (vf_data->acc_magic != ACC_DEV_MAGIC) { > + ret = vf_qm_magic_check(vf_data); > + if (ret) { > dev_err(dev, "failed to match ACC_DEV_MAGIC\n"); > return -EINVAL; > } > @@ -418,7 +440,7 @@ static int vf_qm_get_match_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > int vf_id = hisi_acc_vdev->vf_id; > int ret; > > - vf_data->acc_magic = ACC_DEV_MAGIC; > + vf_data->acc_magic = ACC_DEV_MAGIC_V2; > /* Save device id */ > vf_data->dev_id = hisi_acc_vdev->vf_dev->device; > > @@ -496,12 +518,12 @@ static int vf_qm_read_data(struct hisi_qm > *vf_qm, struct acc_vf_data *vf_data) > return -EINVAL; > > /* Every reg is 32 bit, the dma address is 64 bit. */ > - vf_data->eqe_dma = vf_data->qm_eqc_dw[1]; > + vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->eqe_dma |= vf_data->qm_eqc_dw[0]; > - vf_data->aeqe_dma = vf_data->qm_aeqc_dw[1]; > + vf_data->eqe_dma |= vf_data->qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[0]; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > > /* Through SQC_BT/CQC_BT to get sqc and cqc address */ > ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 245d7537b2bc..2afce68f5a34 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -39,6 +39,9 @@ > #define QM_REG_ADDR_OFFSET 0x0004 > > #define QM_XQC_ADDR_OFFSET 32U > +#define QM_XQC_ADDR_LOW 0x1 > +#define QM_XQC_ADDR_HIGH 0x2 > + > #define QM_VF_AEQ_INT_MASK 0x0004 > #define QM_VF_EQ_INT_MASK 0x000c > #define QM_IFC_INT_SOURCE_V 0x0020 > @@ -50,10 +53,14 @@ > #define QM_EQC_DW0 0X8000 > #define QM_AEQC_DW0 0X8020 > > +enum acc_magic_num { > + ACC_DEV_MAGIC_V1 = 0XCDCDCDCDFEEDAACC, > + ACC_DEV_MAGIC_V2 = 0xAACCFEEDDECADEDE, I think we have discussed this before that having some kind of version info embed into magic_num will be beneficial going forward. ie, may be use the last 4 bytes for denoting version. ACC_DEV_MAGIC_V2 = 0xAACCFEEDDECA0002 The reason being, otherwise we have to come up with a random magic each time when a fix like this is required in future. Thanks, Shameer

1 0

Re: [PATCH v15 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 13 Nov '24

13 Nov '24

> -----Original Message----- > From: Alex Williamson <alex.williamson(a)redhat.com> > Sent: Tuesday, November 12, 2024 9:51 PM > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi(a)huawei.com> > Cc: liulongfang <liulongfang(a)huawei.com>; jgg(a)nvidia.com; Jonathan > Cameron <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v15 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > On Tue, 12 Nov 2024 08:40:03 +0000 > Shameerali Kolothum Thodi <shameerali.kolothum.thodi(a)huawei.com> > wrote: > > > > -----Original Message----- > > > From: liulongfang <liulongfang(a)huawei.com> > > > Sent: Tuesday, November 12, 2024 7:33 AM > > > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > > > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > > > <jonathan.cameron(a)huawei.com> > > > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > > > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > > > Subject: [PATCH v15 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > > > migration driver > > > > > > > > > +static void hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device > > > *hisi_acc_vdev) > > > +{ > > > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > > > + struct hisi_acc_vf_migration_file *migf = NULL; > > > + struct dentry *vfio_dev_migration = NULL; > > > + struct dentry *vfio_hisi_acc = NULL; > > > > Nit, I think we can get rid of these NULL initializations. > > Yup, all three are unnecessary. > > > If you have time, please consider respin (sorry, missed this in earlier > reviews.) > > If that's the only comment, I can fix that on commit if you want to add > an ack/review conditional on that change. Thanks, Thanks Alex. With the above nits addressed, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Shameer

1 0

Re: [PATCH] acpi: Fix hed module initialization order when it is built-in
by Jonathan Cameron 13 Nov '24

13 Nov '24

On Tue, 12 Nov 2024 19:22:27 +0800 Xiaofei Tan <tanxiaofei(a)huawei.com> wrote: > When the module hed is built-in, the init order is determined by > Makefile order. That order violates expectations. Because the module > hed init is behind evged. RAS records can't be handled in the > special time window that evged has initialized while hed not. > If the number of such RAS records is more than the APEI HEST error > source number, the HEST resources could be occupied all, and then > could affect subsequent RAS error reporting. Looks good but I'd like a comment in the makefile to cut down on risk of this breaking again. > > Signed-off-by: Xiaofei Tan <tanxiaofei(a)huawei.com> > --- > drivers/acpi/Makefile | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile > index 61ca4afe83dc..73a6b490d6a0 100644 > --- a/drivers/acpi/Makefile > +++ b/drivers/acpi/Makefile > @@ -14,6 +14,7 @@ tables.o: $(src)/../../include/$(CONFIG_ACPI_CUSTOM_DSDT_FILE) ; > endif > > obj-$(CONFIG_ACPI) += tables.o > +obj-$(CONFIG_ACPI_HED) += hed.o Add a comment on why it is up here. > > # > # ACPI Core Subsystem (Interpreter) > @@ -95,7 +96,6 @@ obj-$(CONFIG_ACPI_HOTPLUG_IOAPIC) += ioapic.o > obj-$(CONFIG_ACPI_BATTERY) += battery.o > obj-$(CONFIG_ACPI_SBS) += sbshc.o > obj-$(CONFIG_ACPI_SBS) += sbs.o > -obj-$(CONFIG_ACPI_HED) += hed.o > obj-$(CONFIG_ACPI_EC_DEBUGFS) += ec_sys.o > obj-$(CONFIG_ACPI_BGRT) += bgrt.o > obj-$(CONFIG_ACPI_CPPC_LIB) += cppc_acpi.o

1 0

Re: [PATCH v15 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 12 Nov '24

12 Nov '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, November 12, 2024 7:33 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v15 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > > +static void hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > + struct hisi_acc_vf_migration_file *migf = NULL; > + struct dentry *vfio_dev_migration = NULL; > + struct dentry *vfio_hisi_acc = NULL; Nit, I think we can get rid of these NULL initializations. If you have time, please consider respin (sorry, missed this in earlier reviews.) Thanks, Shameer

1 0

Re: [PATCH v12 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 05 Nov '24

05 Nov '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, November 5, 2024 3:53 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v12 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver Hi, Few minor comments below. Please don't re-spin just for these yet. Please wait for others to review as well. Thanks, Shameer > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index a8c53952d82e..7728c9745b9d 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -627,15 +627,30 @@ static void hisi_acc_vf_disable_fd(struct > hisi_acc_vf_migration_file *migf) > mutex_unlock(&migf->lock); > } > > +static void hisi_acc_debug_migf_copy(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > + struct hisi_acc_vf_migration_file *src_migf) Alignment should match open parenthesis here. > +{ > + struct hisi_acc_vf_migration_file *dst_migf = hisi_acc_vdev- > >debug_migf; > + > + if (!dst_migf) > + return; > + > + dst_migf->total_length = src_migf->total_length; > + memcpy(&dst_migf->vf_data, &src_migf->vf_data, > + sizeof(struct acc_vf_data)); Here too, alignment not correct. It is better to run, ./scripts/checkpatch --strict on these patches if you haven't done already. > +} > + > static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > { > if (hisi_acc_vdev->resuming_migf) { > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >resuming_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf); > fput(hisi_acc_vdev->resuming_migf->filp); > hisi_acc_vdev->resuming_migf = NULL; > } > > if (hisi_acc_vdev->saving_migf) { > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >saving_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf); > fput(hisi_acc_vdev->saving_migf->filp); > hisi_acc_vdev->saving_migf = NULL; > @@ -1294,6 +1309,129 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > return vfio_pci_core_ioctl(core_vdev, cmd, arg); > } > > +static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device > *vdev) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + int ret; > + > + lockdep_assert_held(&hisi_acc_vdev->open_mutex); > + /* > + * When the device is not opened, the io_base is not mapped. > + * The driver cannot perform device read and write operations. > + */ > + if (!hisi_acc_vdev->dev_opened) { > + seq_printf(seq, "device not opened!\n"); > + return -EINVAL; > + } > + > + ret = qm_wait_dev_not_ready(vf_qm); > + if (ret) { > + seq_printf(seq, "VF device not ready!\n"); > + return -EBUSY; > + } > + > + return 0; > +} > + > +static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = > dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + u64 value; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->open_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + return ret; > + } > + > + value = readl(vf_qm->io_base + QM_MB_CMD_SEND_BASE); > + if (value == QM_MB_CMD_NOT_READY) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + seq_printf(seq, "mailbox cmd channel not ready!\n"); > + return -EINVAL; > + } > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + seq_printf(seq, "mailbox cmd channel ready!\n"); > + > + return 0; > +} > + > +static int hisi_acc_vf_dev_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = > dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + struct acc_vf_data *vf_data; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->open_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + return ret; > + } > + > + mutex_lock(&hisi_acc_vdev->state_mutex); > + vf_data = kzalloc(sizeof(struct acc_vf_data), GFP_KERNEL); > + if (!vf_data) { > + ret = -ENOMEM; > + goto mutex_release; > + } > + > + vf_data->vf_qm_state = hisi_acc_vdev->vf_qm_state; > + ret = vf_qm_read_data(&hisi_acc_vdev->vf_qm, vf_data); > + if (ret) > + goto migf_err; > + > + seq_hex_dump(seq, "Dev Data:", DUMP_PREFIX_OFFSET, 16, 1, > + (const void *)vf_data, vf_data_sz, false); > + > + seq_printf(seq, > + "guest driver load: %u\n" > + "data size: %lu\n", > + hisi_acc_vdev->vf_qm_state, > + sizeof(struct acc_vf_data)); There was a suggestion to add a comment here to describe vf_qm_state better. May be something like, vf_qm_state here indicates whether the Guest has loaded the driver for the ACC VF device or not. > + > +migf_err: > + kfree(vf_data); > +mutex_release: > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + > + return ret; > +} > + > +static int hisi_acc_vf_migf_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = > dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev- > >debug_migf; > + > + /* Check whether the live migration operation has been performed > */ > + if (debug_migf->total_length < QM_MATCH_SIZE) { > + seq_printf(seq, "device not migrated!\n"); > + return -EAGAIN; > + } > + > + seq_hex_dump(seq, "Mig Data:", DUMP_PREFIX_OFFSET, 16, 1, > + (const void *)&debug_migf->vf_data, vf_data_sz, false); > + seq_printf(seq, "migrate data length: %lu\n", debug_migf- > >total_length); > + > + return 0; > +} > + > static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev) > { > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > @@ -1305,12 +1443,16 @@ static int hisi_acc_vfio_pci_open_device(struct > vfio_device *core_vdev) > return ret; > > if (core_vdev->mig_ops) { > + mutex_lock(&hisi_acc_vdev->open_mutex); > ret = hisi_acc_vf_qm_init(hisi_acc_vdev); > if (ret) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_disable(vdev); > return ret; > } > hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING; > + hisi_acc_vdev->dev_opened = true; > + mutex_unlock(&hisi_acc_vdev->open_mutex); > } > > vfio_pci_core_finish_enable(vdev); > @@ -1322,7 +1464,10 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + mutex_lock(&hisi_acc_vdev->open_mutex); > + hisi_acc_vdev->dev_opened = false; > iounmap(vf_qm->io_base); > + mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_close_device(core_vdev); > } > > @@ -1342,6 +1487,7 @@ static int hisi_acc_vfio_pci_migrn_init_dev(struct > vfio_device *core_vdev) > hisi_acc_vdev->pf_qm = pf_qm; > hisi_acc_vdev->vf_dev = pdev; > mutex_init(&hisi_acc_vdev->state_mutex); > + mutex_init(&hisi_acc_vdev->open_mutex); > > core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | > VFIO_MIGRATION_PRE_COPY; > core_vdev->mig_ops = &hisi_acc_vfio_pci_migrn_state_ops; > @@ -1387,6 +1533,48 @@ static const struct vfio_device_ops > hisi_acc_vfio_pci_ops = { > .detach_ioas = vfio_iommufd_physical_detach_ioas, > }; > > +static void hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > + struct dentry *vfio_dev_migration = NULL; > + struct dentry *vfio_hisi_acc = NULL; > + struct device *dev = vdev->dev; > + void *migf = NULL; > + > + if (!debugfs_initialized() || > + !IS_ENABLED(CONFIG_VFIO_DEBUGFS)) > + return; > + > + if (vdev->ops != &hisi_acc_vfio_pci_migrn_ops) > + return; > + > + vfio_dev_migration = debugfs_lookup("migration", vdev- > >debug_root); > + if (!vfio_dev_migration) { > + dev_err(dev, "failed to lookup migration debugfs file!\n"); > + return; > + } > + > + migf = kzalloc(sizeof(struct hisi_acc_vf_migration_file), > GFP_KERNEL); > + if (!migf) > + return; > + hisi_acc_vdev->debug_migf = migf; > + > + vfio_hisi_acc = debugfs_create_dir("hisi_acc", vfio_dev_migration); > + debugfs_create_devm_seqfile(dev, "dev_data", vfio_hisi_acc, > + hisi_acc_vf_dev_read); > + debugfs_create_devm_seqfile(dev, "migf_data", vfio_hisi_acc, > + hisi_acc_vf_migf_read); > + debugfs_create_devm_seqfile(dev, "cmd_state", vfio_hisi_acc, > + hisi_acc_vf_debug_cmd); > +} > + > +static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + /* If migrn_ops is not used, kfree(NULL) is valid */ The above comment is not required. Please remove. > + kfree(hisi_acc_vdev->debug_migf); > + hisi_acc_vdev->debug_migf = NULL; > +} > +

1 0

Re: [PATCH v11 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 04 Nov '24

04 Nov '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Monday, November 4, 2024 8:31 AM > To: Alex Williamson <alex.williamson(a)redhat.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v11 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver [...] > >>> + > >>> + seq_printf(seq, > >>> + "acc device:\n" > >>> + "guest driver load: %u\n" > >>> + "device opened: %d\n" > >>> + "migrate data length: %lu\n", > >>> + hisi_acc_vdev->vf_qm_state, > >>> + hisi_acc_vdev->dev_opened, > >>> + debug_migf->total_length); > >> > >> This debugfs entry is described as returning the data from the last > >> migration, but vf_qm_state and dev_opened are relative to the current > >> device/guest driver state. Both seem to have no relevance to the data > >> in debug_migf. > >> > > > > The benefit of dev_opened retention is that user can obtain the device > status > > during the cat migf_data operation. > > > > I will remove dev_opened in the next version. > And hisi_acc_vdev->vf_qm_state is changed to debug_migf- > >vf_data.vf_qm_state > Keep information about whether the device driver in the Guest OS is loaded > when live migration occurs. I think you already get that when you dump debug_migf->vf_data. So not required. Thanks, Shameer

1 0

Re: [PATCH v10 0/4] debugfs to hisilicon migration driver
by Shameerali Kolothum Thodi 25 Oct '24

25 Oct '24

> -----Original Message----- > From: Jason Gunthorpe <jgg(a)nvidia.com> > Sent: Thursday, October 24, 2024 4:28 PM > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi(a)huawei.com> > Cc: liulongfang <liulongfang(a)huawei.com>; alex.williamson(a)redhat.com; > Jonathan Cameron <jonathan.cameron(a)huawei.com>; > kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org > Subject: Re: [PATCH v10 0/4] debugfs to hisilicon migration driver > > On Thu, Oct 24, 2024 at 01:18:55PM +0000, Shameerali Kolothum Thodi > wrote: > > > Add a debugfs function to the hisilicon migration driver in VFIO to > > > provide intermediate state values and data during device migration. > > > > > > When the execution of live migration fails, the user can view the > > > status and data during the migration process separately from the > > > source and the destination, which is convenient for users to analyze > > > and locate problems. > > > > Could you please take another look at this series as it looks like almost > there. > > Why are we so keen to do this? Nobody else needed a complex debugfs for > their live migration? I don't think it is that complex debugfs. Longfang has found this very helpful in testing and debug with hardware. And hopefully this can be expanded in future with different hardware revisions. Thanks, Shameer

1 0

Re: [PATCH v10 0/4] debugfs to hisilicon migration driver
by Shameerali Kolothum Thodi 24 Oct '24

24 Oct '24

Hi Alex, > -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Wednesday, October 16, 2024 2:23 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v10 0/4] debugfs to hisilicon migration driver > > Add a debugfs function to the hisilicon migration driver in VFIO to > provide intermediate state values and data during device migration. > > When the execution of live migration fails, the user can view the > status and data during the migration process separately from the > source and the destination, which is convenient for users to analyze > and locate problems. Could you please take another look at this series as it looks like almost there. Thanks, Shameer

1 0

Re: [PATCH v10 4/4] Documentation: add debugfs description for hisi migration
by Shameerali Kolothum Thodi 24 Oct '24

24 Oct '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Wednesday, October 16, 2024 2:23 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v10 4/4] Documentation: add debugfs description for hisi > migration > > Add a debugfs document description file to help users understand > how to use the hisilicon accelerator live migration driver's > debugfs. > > Update the file paths that need to be maintained in MAINTAINERS > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> LGTM: Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer > --- > .../ABI/testing/debugfs-hisi-migration | 25 +++++++++++++++++++ > 1 file changed, 25 insertions(+) > create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration > > diff --git a/Documentation/ABI/testing/debugfs-hisi-migration > b/Documentation/ABI/testing/debugfs-hisi-migration > new file mode 100644 > index 000000000000..89e4fde5ec6a > --- /dev/null > +++ b/Documentation/ABI/testing/debugfs-hisi-migration > @@ -0,0 +1,25 @@ > +What: > /sys/kernel/debug/vfio/<device>/migration/hisi_acc/dev_data > +Date: Oct 2024 > +KernelVersion: 6.12 > +Contact: Longfang Liu <liulongfang(a)huawei.com> > +Description: Read the configuration data and some status data > + required for device live migration. These data include device > + status data, queue configuration data, some task > configuration > + data and device attribute data. The output format of the > data > + is defined by the live migration driver. > + > +What: > /sys/kernel/debug/vfio/<device>/migration/hisi_acc/migf_data > +Date: Oct 2024 > +KernelVersion: 6.12 > +Contact: Longfang Liu <liulongfang(a)huawei.com> > +Description: Read the data from the last completed live migration. > + This data includes the same device status data as in > "dev_data". > + The migf_data is the dev_data that is migrated. > + > +What: > /sys/kernel/debug/vfio/<device>/migration/hisi_acc/cmd_state > +Date: Oct 2024 > +KernelVersion: 6.12 > +Contact: Longfang Liu <liulongfang(a)huawei.com> > +Description: Used to obtain the device command sending and receiving > + channel status. Returns failure or success logs based on the > + results. > -- > 2.24.0

1 0

Re: [PATCH v10 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 24 Oct '24

24 Oct '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Wednesday, October 16, 2024 2:23 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v10 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver [..] > @@ -1342,6 +1538,7 @@ static int hisi_acc_vfio_pci_migrn_init_dev(struct > vfio_device *core_vdev) > hisi_acc_vdev->pf_qm = pf_qm; > hisi_acc_vdev->vf_dev = pdev; > mutex_init(&hisi_acc_vdev->state_mutex); > + mutex_init(&hisi_acc_vdev->open_mutex); > > core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | > VFIO_MIGRATION_PRE_COPY; > core_vdev->mig_ops = &hisi_acc_vfio_pci_migrn_state_ops; > @@ -1413,6 +1610,9 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev > *pdev, const struct pci_device > ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); > if (ret) > goto out_put_vdev; > + > + if (ops == &hisi_acc_vfio_pci_migrn_ops) > + hisi_acc_vfio_debug_init(hisi_acc_vdev); As commented earlier, the ops check can be moved to the debug_init() function and you can remove ops check for the debug_exit() below. You may have to rearrange the functions to avoid the compiler error you mentioned in previous version to do so. > return 0; > > out_put_vdev: > @@ -1423,8 +1623,11 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev > *pdev, const struct pci_device > static void hisi_acc_vfio_pci_remove(struct pci_dev *pdev) > { > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_drvdata(pdev); > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > > vfio_pci_core_unregister_device(&hisi_acc_vdev->core_device); > + if (vdev->ops == &hisi_acc_vfio_pci_migrn_ops) > + hisi_acc_vf_debugfs_exit(hisi_acc_vdev); As mentioned above remove the ops check here. With the above ones checked and fixed, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com>

1 0

Re: [PATCH v9 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 11 Oct '24

11 Oct '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Friday, September 13, 2024 10:55 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v9 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > On the debugfs framework of VFIO, if the CONFIG_VFIO_DEBUGFS macro is > enabled, the debug function is registered for the live migration driver > of the HiSilicon accelerator device. > > After registering the HiSilicon accelerator device on the debugfs > framework of live migration of vfio, a directory file "hisi_acc" > of debugfs is created, and then three debug function files are > created in this directory: > > vfio > | > +---<dev_name1> > | +---migration > | +--state > | +--hisi_acc > | +--dev_data > | +--migf_data > | +--cmd_state > | > +---<dev_name2> > +---migration > +--state > +--hisi_acc > +--dev_data > +--migf_data > +--cmd_state > > dev_data file: read device data that needs to be migrated from the > current device in real time > migf_data file: read the migration data of the last live migration > from the current driver. > cmd_state: used to get the cmd channel state for the device. > > +----------------+ +--------------+ +---------------+ > | migration dev | | src dev | | dst dev | > +-------+--------+ +------+-------+ +-------+-------+ > | | | > | +------v-------+ +-------v-------+ > | | saving_migf | | resuming_migf | > read | | file | | file | > | +------+-------+ +-------+-------+ > | | copy | > | +------------+----------+ > | | > +-------v--------+ +-------v--------+ > | data buffer | | debug_migf | > +-------+--------+ +-------+--------+ > | | > cat | cat | > +-------v--------+ +-------v--------+ > | dev_data | | migf_data | > +----------------+ +----------------+ > > When accessing debugfs, user can obtain the most recent status data > of the device through the "dev_data" file. It can read recent > complete status data of the device. If the current device is being > migrated, it will wait for it to complete. > The data for the last completed migration function will be stored > in debug_migf. Users can read it via "migf_data". > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 204 ++++++++++++++++++ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + > 2 files changed, 211 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index a8c53952d82e..da9f5b9e6c5b 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -627,15 +627,30 @@ static void hisi_acc_vf_disable_fd(struct > hisi_acc_vf_migration_file *migf) > mutex_unlock(&migf->lock); > } > > +static void hisi_acc_debug_migf_copy(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > + struct hisi_acc_vf_migration_file *src_migf) > +{ > + struct hisi_acc_vf_migration_file *dst_migf = hisi_acc_vdev- > >debug_migf; > + > + if (!dst_migf) > + return; > + > + dst_migf->total_length = src_migf->total_length; > + memcpy(&dst_migf->vf_data, &src_migf->vf_data, > + sizeof(struct acc_vf_data)); > +} > + > static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > { > if (hisi_acc_vdev->resuming_migf) { > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >resuming_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf); > fput(hisi_acc_vdev->resuming_migf->filp); > hisi_acc_vdev->resuming_migf = NULL; > } > > if (hisi_acc_vdev->saving_migf) { > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >saving_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf); > fput(hisi_acc_vdev->saving_migf->filp); > hisi_acc_vdev->saving_migf = NULL; > @@ -1294,6 +1309,181 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > return vfio_pci_core_ioctl(core_vdev, cmd, arg); > } > > +static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device > *vdev) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + int ret; > + > + lockdep_assert_held(&hisi_acc_vdev->open_mutex); > + /* > + * When the device is not opened, the io_base is not mapped. > + * The driver cannot perform device read and write operations. > + */ > + if (!hisi_acc_vdev->dev_opened) { > + seq_printf(seq, "device not opened!\n"); > + return -EINVAL; > + } > + > + ret = qm_wait_dev_not_ready(vf_qm); > + if (ret) { > + seq_printf(seq, "VF device not ready!\n"); > + return -EBUSY; > + } Still not very sure this vf_qm ready() check actually helps or not? What guarantee is there that the qm will stay Ready after this call? Any read/write afterwards will eventually fail if it is not ready later for some reason, right? Perhaps helps in early detection and bails out. > + return 0; > +} > + > +static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = > dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + u64 value; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->open_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + return ret; > + } > + > + value = readl(vf_qm->io_base + QM_MB_CMD_SEND_BASE); > + if (value == QM_MB_CMD_NOT_READY) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + seq_printf(seq, "mailbox cmd channel not ready!\n"); > + return -EINVAL; > + } > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + seq_printf(seq, "mailbox cmd channel ready!\n"); > + > + return 0; > +} > + > +static int hisi_acc_vf_dev_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = > dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + struct acc_vf_data *vf_data = NULL; > + int ret; > + > + vf_data = kzalloc(sizeof(struct acc_vf_data), GFP_KERNEL); > + if (!vf_data) > + return -ENOMEM;\ You could move the allocation after the below checks and to just before vf_qm_read_data(). > + > + mutex_lock(&hisi_acc_vdev->open_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + goto migf_err; > + } > + > + mutex_lock(&hisi_acc_vdev->state_mutex); > + vf_data->vf_qm_state = hisi_acc_vdev->vf_qm_state; > + ret = vf_qm_read_data(&hisi_acc_vdev->vf_qm, vf_data); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + mutex_unlock(&hisi_acc_vdev->state_mutex); I think it is better to unlock in the reverse order. Also probably you can move the unlocks to a goto area. > + goto migf_err; > + } > + > + mutex_unlock(&hisi_acc_vdev->open_mutex); > + mutex_unlock(&hisi_acc_vdev->state_mutex); Same as above. > + seq_hex_dump(seq, "Dev Data:", DUMP_PREFIX_OFFSET, 16, 1, > + (unsigned char *)vf_data, > + vf_data_sz, false); > + > + seq_printf(seq, > + "acc device:\n" > + "guest driver load: %u\n" > + "data size: %lu\n", > + hisi_acc_vdev->vf_qm_state, > + sizeof(struct acc_vf_data)); > + > +migf_err: > + kfree(vf_data); > + > + return ret; > +} > + > +static int hisi_acc_vf_migf_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = > dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev- > >debug_migf; > + > + /* Check whether the live migration operation has been performed > */ > + if (debug_migf->total_length < QM_MATCH_SIZE) { > + seq_printf(seq, "device not migrated!\n"); > + return -EAGAIN; > + } > + > + seq_hex_dump(seq, "Mig Data:", DUMP_PREFIX_OFFSET, 16, 1, > + (unsigned char *)&debug_migf->vf_data, > + vf_data_sz, false); > + > + seq_printf(seq, > + "acc device:\n" > + "guest driver load: %u\n" > + "device opened: %d\n" > + "migrate data length: %lu\n", > + hisi_acc_vdev->vf_qm_state, > + hisi_acc_vdev->dev_opened, > + debug_migf->total_length); > + > + return 0; > +} > + > +static void hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > + struct dentry *vfio_dev_migration = NULL; > + struct dentry *vfio_hisi_acc = NULL; > + struct device *dev = vdev->dev; > + void *migf = NULL; > + > + if (!debugfs_initialized() || > + !IS_ENABLED(CONFIG_VFIO_DEBUGFS)) > + return; > + > + vfio_dev_migration = debugfs_lookup("migration", vdev- > >debug_root); > + if (!vfio_dev_migration) { > + dev_err(dev, "failed to lookup migration debugfs file!\n"); > + return; > + } > + > + migf = kzalloc(sizeof(struct hisi_acc_vf_migration_file), > GFP_KERNEL); > + if (!migf) > + return; > + hisi_acc_vdev->debug_migf = migf; > + > + vfio_hisi_acc = debugfs_create_dir("hisi_acc", vfio_dev_migration); > + debugfs_create_devm_seqfile(dev, "dev_data", vfio_hisi_acc, > + hisi_acc_vf_dev_read); > + debugfs_create_devm_seqfile(dev, "migf_data", vfio_hisi_acc, > + hisi_acc_vf_migf_read); > + debugfs_create_devm_seqfile(dev, "cmd_state", vfio_hisi_acc, > + hisi_acc_vf_debug_cmd); > +} > + > +static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + if (hisi_acc_vdev->debug_migf) { > + kfree(hisi_acc_vdev->debug_migf); > + hisi_acc_vdev->debug_migf = NULL; > + } > +} > + > static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev) > { > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > @@ -1305,12 +1495,16 @@ static int hisi_acc_vfio_pci_open_device(struct > vfio_device *core_vdev) > return ret; > > if (core_vdev->mig_ops) { > + mutex_lock(&hisi_acc_vdev->open_mutex); > ret = hisi_acc_vf_qm_init(hisi_acc_vdev); > if (ret) { > + mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_disable(vdev); > return ret; > } > hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING; > + hisi_acc_vdev->dev_opened = true; > + mutex_unlock(&hisi_acc_vdev->open_mutex); > } > > vfio_pci_core_finish_enable(vdev); > @@ -1322,7 +1516,10 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + mutex_lock(&hisi_acc_vdev->open_mutex); > + hisi_acc_vdev->dev_opened = false; > iounmap(vf_qm->io_base); > + mutex_unlock(&hisi_acc_vdev->open_mutex); > vfio_pci_core_close_device(core_vdev); > } > > @@ -1342,6 +1539,7 @@ static int hisi_acc_vfio_pci_migrn_init_dev(struct > vfio_device *core_vdev) > hisi_acc_vdev->pf_qm = pf_qm; > hisi_acc_vdev->vf_dev = pdev; > mutex_init(&hisi_acc_vdev->state_mutex); > + mutex_init(&hisi_acc_vdev->open_mutex); > > core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | > VFIO_MIGRATION_PRE_COPY; > core_vdev->mig_ops = &hisi_acc_vfio_pci_migrn_state_ops; > @@ -1413,6 +1611,9 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev > *pdev, const struct pci_device > ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); > if (ret) > goto out_put_vdev; > + > + if (ops == &hisi_acc_vfio_pci_migrn_ops) > + hisi_acc_vfio_debug_init(hisi_acc_vdev); I think there was a comment earlier on this. Still not sure why it is not possible to move ops == &hisi_acc_vfio_pci_migrn_ops check inside hisi_acc_vfio_debug_init(). > return 0; > > out_put_vdev: > @@ -1423,8 +1624,11 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev > *pdev, const struct pci_device > static void hisi_acc_vfio_pci_remove(struct pci_dev *pdev) > { > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_drvdata(pdev); > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > > vfio_pci_core_unregister_device(&hisi_acc_vdev->core_device); > + if (vdev->ops == &hisi_acc_vfio_pci_migrn_ops) > + hisi_acc_vf_debugfs_exit(hisi_acc_vdev); Do we need to do vdev->ops == &hisi_acc_vfio_pci_migrn_ops check here? Since we are checking hisi_acc_vdev->debug_migf inside the exit function, which I think is only set when the ops == migrn_ops. Right? Thanks, Shameer

1 0

Re: [PATCH v8 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 14 Aug '24

14 Aug '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, August 6, 2024 1:29 PM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v8 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver ... > +static int hisi_acc_vf_dev_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + struct acc_vf_data *vf_data = NULL; > + int ret; > + > + vf_data = kzalloc(sizeof(struct acc_vf_data), GFP_KERNEL); > + if (!vf_data) > + return -ENOMEM; > + > + mutex_lock(&hisi_acc_vdev->state_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + goto migf_err; > + } > + > + vf_data->vf_qm_state = hisi_acc_vdev->vf_qm_state; > + ret = vf_qm_read_data(&hisi_acc_vdev->vf_qm, vf_data); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + goto migf_err; > + } > + > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + > + seq_hex_dump(seq, "Dev Data:", DUMP_PREFIX_OFFSET, 16, 1, > + (unsigned char *)vf_data, > + vf_data_sz, false); > + > + seq_printf(seq, > + "acc device:\n" > + "device ready: %u\n" > + "device opened: %d\n" > + "data size: %lu\n", > + hisi_acc_vdev->vf_qm_state, > + hisi_acc_vdev->dev_opened, I think the dev_opened will be always true if it reaches here and can be removed from here and from hisi_acc_vf_migf_read() as well. Please don't respin just for this. Let us wait for others to review this. Thanks, Shameer

1 0

Re: [PATCH v8 4/4] Documentation: add debugfs description for hisi migration
by Shameerali Kolothum Thodi 14 Aug '24

14 Aug '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, August 6, 2024 1:29 PM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v8 4/4] Documentation: add debugfs description for hisi > migration > > Add a debugfs document description file to help users understand > how to use the hisilicon accelerator live migration driver's > debugfs. > > Update the file paths that need to be maintained in MAINTAINERS > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> LGTM, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer

1 0

Re: [PATCH v7 2/4] hisi_acc_vfio_pci: create subfunction for data reading
by Shameerali Kolothum Thodi 05 Aug '24

05 Aug '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, July 30, 2024 1:15 PM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v7 2/4] hisi_acc_vfio_pci: create subfunction for data > reading > > This patch generates the code for the operation of reading data from > the device into a sub-function. > Then, it can be called during the device status data saving phase of > the live migration process and the device status data reading function > in debugfs. > Thereby reducing the redundant code of the driver. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- LGTM, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 54 +++++++++++-------- > 1 file changed, 33 insertions(+), 21 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 45351be8e270..a8c53952d82e 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -486,31 +486,11 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return 0; > } > > -static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, > - struct hisi_acc_vf_migration_file *migf) > +static int vf_qm_read_data(struct hisi_qm *vf_qm, struct acc_vf_data > *vf_data) > { > - struct acc_vf_data *vf_data = &migf->vf_data; > - struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > struct device *dev = &vf_qm->pdev->dev; > int ret; > > - if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > - /* Update state and return with match data */ > - vf_data->vf_qm_state = QM_NOT_READY; > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > - migf->total_length = QM_MATCH_SIZE; > - return 0; > - } > - > - vf_data->vf_qm_state = QM_READY; > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > - > - ret = vf_qm_cache_wb(vf_qm); > - if (ret) { > - dev_err(dev, "failed to writeback QM Cache!\n"); > - return ret; > - } > - > ret = qm_get_regs(vf_qm, vf_data); > if (ret) > return -EINVAL; > @@ -536,6 +516,38 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return -EINVAL; > } > > + return 0; > +} > + > +static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct hisi_acc_vf_migration_file *migf) > +{ > + struct acc_vf_data *vf_data = &migf->vf_data; > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + struct device *dev = &vf_qm->pdev->dev; > + int ret; > + > + if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > + /* Update state and return with match data */ > + vf_data->vf_qm_state = QM_NOT_READY; > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + migf->total_length = QM_MATCH_SIZE; > + return 0; > + } > + > + vf_data->vf_qm_state = QM_READY; > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + > + ret = vf_qm_cache_wb(vf_qm); > + if (ret) { > + dev_err(dev, "failed to writeback QM Cache!\n"); > + return ret; > + } > + > + ret = vf_qm_read_data(vf_qm, vf_data); > + if (ret) > + return -EINVAL; > + > migf->total_length = sizeof(struct acc_vf_data); > return 0; > } > -- > 2.24.0

1 0

Re: [PATCH v7 1/4] hisi_acc_vfio_pci: extract public functions for container_of
by Shameerali Kolothum Thodi 05 Aug '24

05 Aug '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, July 30, 2024 1:15 PM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v7 1/4] hisi_acc_vfio_pci: extract public functions for > container_of > > In the current driver, vdev is obtained from struct > hisi_acc_vf_core_device through the container_of function. > This method is used in many places in the driver. In order to > reduce this repetitive operation, It was extracted into > a public function. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- LGTM, Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> Thanks, Shameer > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 21 ++++++++++--------- > 1 file changed, 11 insertions(+), 10 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 9a3e97108ace..45351be8e270 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -630,6 +630,12 @@ static void hisi_acc_vf_disable_fds(struct > hisi_acc_vf_core_device *hisi_acc_vde > } > } > > +static struct hisi_acc_vf_core_device *hisi_acc_get_vf_dev(struct > vfio_device *vdev) > +{ > + return container_of(vdev, struct hisi_acc_vf_core_device, > + core_device.vdev); > +} > + > static void hisi_acc_vf_reset(struct hisi_acc_vf_core_device *hisi_acc_vdev) > { > hisi_acc_vdev->vf_qm_state = QM_NOT_READY; > @@ -1033,8 +1039,7 @@ static struct file * > hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev, > enum vfio_device_mig_state new_state) > { > - struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(vdev, > - struct hisi_acc_vf_core_device, core_device.vdev); > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > enum vfio_device_mig_state next_state; > struct file *res = NULL; > int ret; > @@ -1075,8 +1080,7 @@ static int > hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev, > enum vfio_device_mig_state *curr_state) > { > - struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(vdev, > - struct hisi_acc_vf_core_device, core_device.vdev); > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > > mutex_lock(&hisi_acc_vdev->state_mutex); > *curr_state = hisi_acc_vdev->mig_state; > @@ -1280,8 +1284,7 @@ static long hisi_acc_vfio_pci_ioctl(struct vfio_device > *core_vdev, unsigned int > > static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev) > { > - struct hisi_acc_vf_core_device *hisi_acc_vdev = > container_of(core_vdev, > - struct hisi_acc_vf_core_device, core_device.vdev); > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device; > int ret; > > @@ -1304,8 +1307,7 @@ static int hisi_acc_vfio_pci_open_device(struct > vfio_device *core_vdev) > > static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev) > { > - struct hisi_acc_vf_core_device *hisi_acc_vdev = > container_of(core_vdev, > - struct hisi_acc_vf_core_device, core_device.vdev); > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > iounmap(vf_qm->io_base); > @@ -1320,8 +1322,7 @@ static const struct vfio_migration_ops > hisi_acc_vfio_pci_migrn_state_ops = { > > static int hisi_acc_vfio_pci_migrn_init_dev(struct vfio_device *core_vdev) > { > - struct hisi_acc_vf_core_device *hisi_acc_vdev = > container_of(core_vdev, > - struct hisi_acc_vf_core_device, core_device.vdev); > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct pci_dev *pdev = to_pci_dev(core_vdev->dev); > struct hisi_qm *pf_qm = hisi_acc_get_pf_qm(pdev); > > -- > 2.24.0

1 0

Re: [PATCH v7 4/4] Documentation: add debugfs description for hisi migration
by Shameerali Kolothum Thodi 05 Aug '24

05 Aug '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, July 30, 2024 1:15 PM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v7 4/4] Documentation: add debugfs description for hisi > migration > > Add a debugfs document description file to help users understand > how to use the hisilicon accelerator live migration driver's > debugfs. > > Update the file paths that need to be maintained in MAINTAINERS > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../ABI/testing/debugfs-hisi-migration | 25 +++++++++++++++++++ > 1 file changed, 25 insertions(+) > create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration > > diff --git a/Documentation/ABI/testing/debugfs-hisi-migration > b/Documentation/ABI/testing/debugfs-hisi-migration > new file mode 100644 > index 000000000000..053f3ebba9b1 > --- /dev/null > +++ b/Documentation/ABI/testing/debugfs-hisi-migration > @@ -0,0 +1,25 @@ > +What: > /sys/kernel/debug/vfio/<device>/migration/hisi_acc/dev_data > +Date: Jul 2024 > +KernelVersion: 6.11 > +Contact: Longfang Liu <liulongfang(a)huawei.com> > +Description: Read the configuration data and some status data > + required for device live migration. These data include device > + status data, queue configuration data, some task > configuration > + data and device attribute data. The output format of the data > + is defined by the live migration driver. > + > +What: > /sys/kernel/debug/vfio/<device>/migration/hisi_acc/migf_data > +Date: Jul 2024 > +KernelVersion: 6.11 > +Contact: Longfang Liu <liulongfang(a)huawei.com> > +Description: Read the data from the last completed live migration. > + This data includes the same device status data as in > "dev_data". > + And some device status data after the migration is > completed. Actually what info is different from dev_data here? Only that it is the dev_data after a migration is attempted/completed, right? Thanks, Shameer > + > +What: > /sys/kernel/debug/vfio/<device>/migration/hisi_acc/cmd_state > +Date: Jul 2024 > +KernelVersion: 6.11 > +Contact: Longfang Liu <liulongfang(a)huawei.com> > +Description: Used to obtain the device command sending and receiving > + channel status. Returns failure or success logs based on the > + results. > -- > 2.24.0

1 0

Re: [PATCH v7 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 05 Aug '24

05 Aug '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, July 30, 2024 1:15 PM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v7 3/4] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > On the debugfs framework of VFIO, if the CONFIG_VFIO_DEBUGFS macro is > enabled, the debug function is registered for the live migration driver > of the HiSilicon accelerator device. > > After registering the HiSilicon accelerator device on the debugfs > framework of live migration of vfio, a directory file "hisi_acc" > of debugfs is created, and then three debug function files are > created in this directory: > > vfio > | > +---<dev_name1> > | +---migration > | +--state > | +--hisi_acc > | +--dev_data > | +--migf_data > | +--cmd_state > | > +---<dev_name2> > +---migration > +--state > +--hisi_acc > +--dev_data > +--migf_data > +--cmd_state > > dev_data file: read device data that needs to be migrated from the > current device in real time > migf_data file: read the migration data of the last live migration > from the current driver. > cmd_state: used to get the cmd channel state for the device. > > +----------------+ +--------------+ +---------------+ > | migration dev | | src dev | | dst dev | > +-------+--------+ +------+-------+ +-------+-------+ > | | | > | +------v-------+ +-------v-------+ > | | saving_migf | | resuming_migf | > read | | file | | file | > | +------+-------+ +-------+-------+ > | | copy | > | +------------+----------+ > | | > +-------v--------+ +-------v--------+ > | data buffer | | debug_migf | > +-------+--------+ +-------+--------+ > | | > cat | cat | > +-------v--------+ +-------v--------+ > | dev_data | | migf_data | > +----------------+ +----------------+ > > When accessing debugfs, user can obtain the most recent status data > of the device through the "dev_data" file. It can read recent > complete status data of the device. If the current device is being > migrated, it will wait for it to complete. > The data for the last completed migration function will be stored > in debug_migf. Users can read it via "migf_data". > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 220 ++++++++++++++++++ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 6 + > 2 files changed, 226 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index a8c53952d82e..ae8946901e73 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -627,15 +627,31 @@ static void hisi_acc_vf_disable_fd(struct > hisi_acc_vf_migration_file *migf) > mutex_unlock(&migf->lock); > } > > +static void hisi_acc_debug_migf_copy(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > + struct hisi_acc_vf_migration_file *src_migf) > +{ > + struct hisi_acc_vf_migration_file *dst_migf = hisi_acc_vdev- > >debug_migf; > + > + if (!dst_migf) > + return; > + > + dst_migf->disabled = true; This is always set to true, then why bother printing this as part of migf_data below?. See also comments on "disabled" below. Is there any value in giving this info to user? > + dst_migf->total_length = src_migf->total_length; > + memcpy(&dst_migf->vf_data, &src_migf->vf_data, > + sizeof(struct acc_vf_data)); > +} > + > static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > { > if (hisi_acc_vdev->resuming_migf) { > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >resuming_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf); > fput(hisi_acc_vdev->resuming_migf->filp); > hisi_acc_vdev->resuming_migf = NULL; > } > > if (hisi_acc_vdev->saving_migf) { > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >saving_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf); > fput(hisi_acc_vdev->saving_migf->filp); > hisi_acc_vdev->saving_migf = NULL; > @@ -1294,6 +1310,201 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > return vfio_pci_core_ioctl(core_vdev, cmd, arg); > } > > +static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device > *vdev) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + int ret; > + > + if (!vdev->mig_ops) { > + seq_printf(seq, "%s\n", "device does not support live > migration!\n"); > + return -EINVAL; > + } > + > + /* > + * When the device is not opened, the io_base is not mapped. > + * The driver cannot perform device read and write operations. > + */ I think it is better you make sure the lock is held in this function before checking this. Use lockdep_assert_held(lock). > + if (!hisi_acc_vdev->dev_opened) { > + seq_printf(seq, "%s\n", "device not opened!\n"); > + return -EINVAL; > + } > + > + ret = qm_wait_dev_not_ready(vf_qm); > + if (ret) { > + seq_printf(seq, "%s\n", "VF device not ready!\n"); > + return -EBUSY; > + } > + > + return 0; > +} > + > +static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + u64 value; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->state_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + return ret; > + } > + > + value = readl(vf_qm->io_base + QM_MB_CMD_SEND_BASE); > + if (value == QM_MB_CMD_NOT_READY) { > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + seq_printf(seq, "mailbox cmd channel not ready!\n"); > + return -EINVAL; > + } > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + seq_printf(seq, "mailbox cmd channel ready!\n"); > + > + return 0; > +} > + > +static int hisi_acc_vf_dev_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + struct acc_vf_data *vf_data = NULL; > + bool migf_state; > + int ret; > + > + vf_data = kzalloc(sizeof(struct acc_vf_data), GFP_KERNEL); > + if (!vf_data) > + return -ENOMEM; > + > + mutex_lock(&hisi_acc_vdev->state_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + goto migf_err; > + } > + > + vf_data->vf_qm_state = hisi_acc_vdev->vf_qm_state; > + ret = vf_qm_read_data(&hisi_acc_vdev->vf_qm, vf_data); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + goto migf_err; > + } > + > + if (hisi_acc_vdev->resuming_migf) > + migf_state = hisi_acc_vdev->resuming_migf->disabled; > + else if (hisi_acc_vdev->saving_migf) > + migf_state = hisi_acc_vdev->saving_migf->disabled; > + else > + migf_state = true; I am still not sure what information we are getting from this "disabled". The value "true" means the migf file is released, isn't it? How is that equivalent to report as "data valid" below? Also "migf_state" is misleading here. Thanks, Shameer > + mutex_unlock(&hisi_acc_vdev->state_mutex); > + > + seq_hex_dump(seq, "Dev Data:", DUMP_PREFIX_OFFSET, 16, 1, > + (unsigned char *)vf_data, > + vf_data_sz, false); > + > + seq_printf(seq, > + "acc device:\n" > + "device ready: %u\n" > + "device opened: %d\n" > + "data valid: %d\n" > + "data size: %lu\n", > + hisi_acc_vdev->vf_qm_state, > + hisi_acc_vdev->dev_opened, > + migf_state, > + sizeof(struct acc_vf_data)); > + > +migf_err: > + kfree(vf_data); > + > + return ret; > +} > + > +static int hisi_acc_vf_migf_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev- > >debug_migf; > + > + /* Check whether the live migration operation has been performed > */ > + if (debug_migf->total_length < QM_MATCH_SIZE) { > + seq_printf(seq, "%s\n", "device not migrated!\n"); > + return -EAGAIN; > + } > + > + seq_hex_dump(seq, "Mig Data:", DUMP_PREFIX_OFFSET, 16, 1, > + (unsigned char *)&debug_migf->vf_data, > + vf_data_sz, false); > + > + seq_printf(seq, > + "acc device:\n" > + "device ready: %u\n" > + "device opened: %d\n" > + "data valid: %d\n" > + "data size: %lu\n", > + hisi_acc_vdev->vf_qm_state, > + hisi_acc_vdev->dev_opened, > + debug_migf->disabled, > + debug_migf->total_length); > + > + return 0; > +} > + > +static int hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > + struct dentry *vfio_dev_migration = NULL; > + struct dentry *vfio_hisi_acc = NULL; > + struct device *dev = vdev->dev; > + void *migf = NULL; > + > + if (!debugfs_initialized() || > + !IS_ENABLED(CONFIG_VFIO_DEBUGFS)) > + return 0; > + > + migf = kzalloc(sizeof(struct hisi_acc_vf_migration_file), GFP_KERNEL); > + if (!migf) > + return -ENOMEM; > + hisi_acc_vdev->debug_migf = migf; > + > + vfio_dev_migration = debugfs_lookup("migration", vdev- > >debug_root); > + if (!vfio_dev_migration) { > + kfree(migf); > + hisi_acc_vdev->debug_migf = NULL; > + dev_err(dev, "failed to lookup migration debugfs file!\n"); > + return -ENODEV; > + } > + > + vfio_hisi_acc = debugfs_create_dir("hisi_acc", vfio_dev_migration); > + debugfs_create_devm_seqfile(dev, "dev_data", vfio_hisi_acc, > + hisi_acc_vf_dev_read); > + debugfs_create_devm_seqfile(dev, "migf_data", vfio_hisi_acc, > + hisi_acc_vf_migf_read); > + debugfs_create_devm_seqfile(dev, "cmd_state", vfio_hisi_acc, > + hisi_acc_vf_debug_cmd); > + > + return 0; > +} > + > +static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + if (!debugfs_initialized() || > + !IS_ENABLED(CONFIG_VFIO_DEBUGFS)) > + return; > + > + if (hisi_acc_vdev->debug_migf) > + kfree(hisi_acc_vdev->debug_migf); > +} > + > static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev) > { > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > @@ -1311,9 +1522,11 @@ static int hisi_acc_vfio_pci_open_device(struct > vfio_device *core_vdev) > return ret; > } > hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING; > + hisi_acc_vdev->dev_opened = true; > } > > vfio_pci_core_finish_enable(vdev); > + > return 0; > } > > @@ -1322,7 +1535,10 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + mutex_lock(&hisi_acc_vdev->state_mutex); > + hisi_acc_vdev->dev_opened = false; > iounmap(vf_qm->io_base); > + mutex_unlock(&hisi_acc_vdev->state_mutex); > vfio_pci_core_close_device(core_vdev); > } > > @@ -1413,6 +1629,9 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev > *pdev, const struct pci_device > ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); > if (ret) > goto out_put_vdev; > + > + if (ops == &hisi_acc_vfio_pci_migrn_ops) > + hisi_acc_vfio_debug_init(hisi_acc_vdev); > return 0; > > out_put_vdev: > @@ -1425,6 +1644,7 @@ static void hisi_acc_vfio_pci_remove(struct pci_dev > *pdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_drvdata(pdev); > > vfio_pci_core_unregister_device(&hisi_acc_vdev->core_device); > + hisi_acc_vf_debugfs_exit(hisi_acc_vdev); > vfio_put_device(&hisi_acc_vdev->core_device.vdev); > } > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 5bab46602fad..f86f3b88b09e 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -32,6 +32,7 @@ > #define QM_SQC_VFT_BASE_MASK_V2 GENMASK(15, 0) > #define QM_SQC_VFT_NUM_SHIFT_V2 45 > #define QM_SQC_VFT_NUM_MASK_V2 GENMASK(9, 0) > +#define QM_MB_CMD_NOT_READY 0xffffffff > > /* RW regs */ > #define QM_REGS_MAX_LEN 7 > @@ -111,5 +112,10 @@ struct hisi_acc_vf_core_device { > int vf_id; > struct hisi_acc_vf_migration_file *resuming_migf; > struct hisi_acc_vf_migration_file *saving_migf; > + > + /* To make sure the device is opened */ > + bool dev_opened; > + /* To save migration data */ > + struct hisi_acc_vf_migration_file *debug_migf; > }; > #endif /* HISI_ACC_VFIO_PCI_H */ > -- > 2.24.0

1 0

Re: [PATCH] soc: hisilicon: Support memory repair driver on Kunpeng SoC
by Jonathan Cameron 19 Jul '24

19 Jul '24

On Tue, 9 Jul 2024 17:34:18 +0800 Xiaofei Tan <tanxiaofei(a)huawei.com> wrote: > Kunpeng SoC support several memory repair capability. Normally, > such repairs are done in firmware startup stage or triggered by > BMC(Baseboard Management Controller) and done in firmware in > runtime stage. OS can't perceive such capability. > > Some support online repair, while others can't. One reason of not > supporting online repair is that the memory is in use and the > hardware does not support backpressure on the primary access channel. > > In order to support more online memory repair, we seek solutions > from OS. That is try to isolate the memory page first, and do repair, > and de-isloate after repair done. > > This patch support online ACLS(Adapter Cache Line Sparing) repair, and > may add other repair such as PPR in future. > > Signed-off-by: Xiaofei Tan <tanxiaofei(a)huawei.com> As you know PPR is on the list of features we are proposing to cover with the 'new' EDAC stuff / RAS control that Shiju is working on so as to bring a unified interface to CXL and anything else (e.g. this) that needs it. (I happen to know there is a proposal from Micron to discuss PPR specifically at LPC). I'm not keen to see an upstream solution for a specific device that we then have to move over later. More significantly what stops the user of this shooting themselves in the foot? How do we know the memory is offline when repair is attempted? That was the bit that I was least sure about for the CXL version (and why it needs to be kernel mediated rather than just made a userspace problem) If this is something custom to keep openeuler happy in the short term them fair enough, but we don't want lots of (OS in the loop) PPR solutions. I did raise an open question in one of those threads on whether anyone currently cares about poison repair on a running system. Interest so far is minor so it maybe challenging to convince people of this except that it might be used on initial boot to remove lines known bad from previous boots etc before onlining the memory. If we are considering a DAX / ACPI Specific Purpose Memory model for the memory we are considering repairing things get easier as can do repair after tearing down the mappings. Various comments inline. May well overlap with earlier reviews though! > --- > drivers/soc/hisilicon/Kconfig | 10 + > drivers/soc/hisilicon/Makefile | 1 + > drivers/soc/hisilicon/hisi_mem_ras.c | 553 +++++++++++++++++++++++++++ > drivers/soc/hisilicon/hisi_mem_ras.h | 84 ++++ Needs Documentation/ABI/* > 4 files changed, 648 insertions(+) > create mode 100644 drivers/soc/hisilicon/hisi_mem_ras.c > create mode 100644 drivers/soc/hisilicon/hisi_mem_ras.h > > diff --git a/drivers/soc/hisilicon/Kconfig b/drivers/soc/hisilicon/Kconfig > index ae8e55a776e0..983cafc38e5c 100644 > --- a/drivers/soc/hisilicon/Kconfig > +++ b/drivers/soc/hisilicon/Kconfig > @@ -44,4 +44,14 @@ config HISI_HBMCACHE > To compile the driver as a module, choose M here: > the module will be called hisi_hbmcache. > > +config HISI_MEM_RAS > + tristate "Add support for HISI Memory repair" > + depends on ACPI > + help > + Add Memory repair support for Hisilicon device, which can be used to query > + memory repair cap and repair hardware error in memory devices. This feature > + need to work with hardware firmwares. I'd drop the lat sentence. It's a detail the kernel doesn't need. > + > + If not sure say no. > + > endmenu > diff --git a/drivers/soc/hisilicon/Makefile b/drivers/soc/hisilicon/Makefile > index 4ef8823250e8..b339a67bcd77 100644 > --- a/drivers/soc/hisilicon/Makefile > +++ b/drivers/soc/hisilicon/Makefile > @@ -6,3 +6,4 @@ obj-$(CONFIG_KUNPENG_HCCS) += kunpeng_hccs.o > obj-$(CONFIG_HISI_HBMDEV) += hisi_hbmdev.o > obj-$(CONFIG_HISI_HBMCACHE) += hisi_hbmcache.o > obj-$(CONFIG_ARM64_PBHA) += pbha.o > +obj-$(CONFIG_HISI_MEM_RAS) += hisi_mem_ras.o > diff --git a/drivers/soc/hisilicon/hisi_mem_ras.c b/drivers/soc/hisilicon/hisi_mem_ras.c > new file mode 100644 > index 000000000000..a29c41ccdba1 > --- /dev/null > +++ b/drivers/soc/hisilicon/hisi_mem_ras.c > @@ -0,0 +1,553 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright (C) Huawei Technologies Co., Ltd. 2024. All rights reserved. > + */ > + > +#include <linux/kobject.h> > +#include <linux/module.h> > +#include <linux/acpi.h> > +#include <linux/iopoll.h> > +#include <linux/platform_device.h> > +#include <linux/mm.h> > +#include <acpi/pcc.h> > + > +#include "hisi_mem_ras.h" > +#define DRV_NAME "hisi_mem_ras" > + > +#define HISI_MEM_PCC_CMD_WAIT_RETRIES_NUM 500ULL > +#define HISI_MEM_POLL_STATUS_TIME_INTERVAL_US 3 > + > +static struct hisi_mem_ras *mras; > + > +static int hisi_mem_get_paddr_range(struct hisi_mem_dev *hdev) > +{ > + struct platform_device *pdev = hdev->pdev; > + struct acpi_device *adev = ACPI_COMPANION(&pdev->dev); > + struct hisi_mem_register_ctx ctx = {0}; Might as well set dev here. > + acpi_handle handle = adev->handle; > + acpi_status status; > + > + if (!acpi_has_method(handle, METHOD_NAME__CRS)) { > + dev_err(&pdev->dev, "No _CRS method.\n"); > + return -ENODEV; > + } > + > + ctx.dev = &pdev->dev; > + status = acpi_walk_resources(handle, METHOD_NAME__CRS, > + hisi_mem_get_paddr_range_cb, &ctx); > + if (ACPI_FAILURE(status)) > + return ctx.err; > + > + hdev->paddr_min = ctx.paddr_min; > + hdev->paddr_max = ctx.paddr_max; > + return 0; > +} > + > + > +static int hisi_mem_register_pcc_channel(struct hisi_mem_dev *hdev) > +{ > + struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; > + struct mbox_client *cl = &cl_info->client; > + struct pcc_mbox_chan *pcc_chan; > + struct platform_device *pdev = hdev->pdev; > + int rc; > + > + cl->dev = &pdev->dev; > + cl->tx_block = false; > + cl->knows_txdone = true; > + cl->tx_done = hisi_mem_chan_tx_done; > + cl->rx_callback = hdev->verspec_data->rx_callback; > + init_completion(&cl_info->done); > + > + pcc_chan = pcc_mbox_request_channel(cl, hdev->chan_id); > + if (IS_ERR(pcc_chan)) { > + dev_err(&pdev->dev, "PPC channel request failed.\n"); > + rc = -ENODEV; > + goto out; return dev_err_probe(&pdev->dev, -ENODEV,...); > + } > + cl_info->pcc_chan = pcc_chan; Maybe worth some __free() magic and delaying setting this until all error paths are done then using a no_free_ptr() to steal the pointer from the autocleanup. Will let you do direct returns and use dev_err_probe() giving cleaner result. > + cl_info->mbox_chan = pcc_chan->mchan; > + > + /* > + * pcc_chan->latency is just a nominal value. In reality the remote > + * processor could be much slower to reply. So add an arbitrary amount > + * of wait on top of nominal. > + */ > + cl_info->deadline_us = > + HISI_MEM_PCC_CMD_WAIT_RETRIES_NUM * pcc_chan->latency; > + if (!hdev->verspec_data->has_txdone_irq && > + cl_info->mbox_chan->mbox->txdone_irq) { > + dev_err(&pdev->dev, "PCC IRQ in PCCT is enabled.\n"); > + rc = -EINVAL; > + goto err_mbx_channel_free; > + } else if (hdev->verspec_data->has_txdone_irq && > + !cl_info->mbox_chan->mbox->txdone_irq) { > + dev_err(&pdev->dev, "PCC IRQ in PCCT isn't supported.\n"); > + rc = -EINVAL; > + goto err_mbx_channel_free; > + } > + > + if (pcc_chan->shmem_base_addr) { > + cl_info->pcc_comm_addr = ioremap(pcc_chan->shmem_base_addr, > + pcc_chan->shmem_size); > + if (!cl_info->pcc_comm_addr) { > + dev_err(&pdev->dev, "Failed to ioremap PCC communication region for channel-%u.\n", > + hdev->chan_id); > + rc = -ENOMEM; > + goto err_mbx_channel_free; > + } > + } > + > + return 0; > + > +err_mbx_channel_free: > + pcc_mbox_free_channel(cl_info->pcc_chan); > +out: Never have an out label that just returns. Just return inline as it is easier to read. > + return rc; > +} > + > + > +static int hisi_mem_wait_cmd_complete_by_poll(struct hisi_mem_dev *hdev) > +{ > + struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; > + struct acpi_pcct_shared_memory __iomem *comm_base = > + cl_info->pcc_comm_addr; That's an ugly formatting. Just one tab more than line above before cl_info->... > + struct platform_device *pdev = hdev->pdev; > + u16 status; > + int ret; > + > + /* > + * Poll PCC status register every 3us(delay_us) for maximum of > + * deadline_us(timeout_us) until PCC command complete bit is set(cond) > + */ > + ret = readw_poll_timeout(&comm_base->status, status, > + status & PCC_STATUS_CMD_COMPLETE, > + HISI_MEM_POLL_STATUS_TIME_INTERVAL_US, > + cl_info->deadline_us); > + if (unlikely(ret)) > + dev_err(&pdev->dev, "poll PCC status failed, ret = %d.\n", ret); > + > + return ret; > +} > + > +static int hisi_mem_wait_cmd_complete_by_irq(struct hisi_mem_dev *hdev) > +{ > + struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; > + struct platform_device *pdev = hdev->pdev; > + > + if (!wait_for_completion_timeout(&cl_info->done, > + usecs_to_jiffies(cl_info->deadline_us))) { > + dev_err(&pdev->dev, "PCC command executed timeout!\n"); > + return -ETIMEDOUT; > + } > + > + return 0; > +} > + > +static inline void hisi_mem_fill_pcc_shared_mem_region(struct hisi_mem_dev *hdev, > + u8 cmd, > + struct hisi_mem_desc *desc, Given this is the request, why use the union rather than struct hisi_mem_req_desc? > + void __iomem *comm_space, > + u16 space_size) > +{ > + struct acpi_pcct_shared_memory tmp = { > + .signature = PCC_SIGNATURE | hdev->chan_id, > + .command = cmd, > + .status = 0, I'd not bother intializing status. C will do it for you and it's a fairly sensible default. > + }; > + > + memcpy_toio(hdev->cl_info.pcc_comm_addr, (void *)&tmp, Should never need to cast to or from an void * > + sizeof(struct acpi_pcct_shared_memory)); sizeof (*tmp) > + > + /* Copy the message to the PCC comm space */ > + memcpy_toio(comm_space, (void *)desc, space_size); If desc is less than the size of the memspace this is going to copy in extra. Maybe that's prevented somewhere but it's not obvious to me. > +} > + > +static inline void hisi_mem_fill_ext_pcc_shared_mem_region(struct hisi_mem_dev *hdev, > + u8 cmd, > + struct hisi_mem_desc *desc, > + void __iomem *comm_space, > + u16 space_size) > +{ > + struct acpi_pcct_ext_pcc_shared_memory tmp = { > + .signature = PCC_SIGNATURE | hdev->chan_id, > + .flags = PCC_CMD_COMPLETION_NOTIFY, > + .length = HISI_MEM_PCC_SHARE_MEM_BYTES, > + .command = cmd, > + }; > + > + memcpy_toio(hdev->cl_info.pcc_comm_addr, (void *)&tmp, Should be no point in void * casts. > + sizeof(struct acpi_pcct_ext_pcc_shared_memory)); sizeof (*tmp) > + > + /* Copy the message to the PCC comm space */ > + memcpy_toio(comm_space, (void *)desc, space_size); > +} > + > +static const struct hisi_mem_verspecific_data hisi04b1_verspec_data = { > + .rx_callback = NULL, > + .wait_cmd_complete = hisi_mem_wait_cmd_complete_by_poll, > + .fill_pcc_shared_mem = hisi_mem_fill_pcc_shared_mem_region, > + .shared_mem_size = sizeof(struct acpi_pcct_shared_memory), > + .has_txdone_irq = false, > +}; > + > +static const struct hisi_mem_verspecific_data hisi04b2_verspec_data = { > + .rx_callback = hisi_mem_pcc_rx_callback, > + .wait_cmd_complete = hisi_mem_wait_cmd_complete_by_irq, > + .fill_pcc_shared_mem = hisi_mem_fill_ext_pcc_shared_mem_region, > + .shared_mem_size = sizeof(struct acpi_pcct_ext_pcc_shared_memory), > + .has_txdone_irq = true, > +}; > + > +static int hisi_mem_pcc_cmd_send(struct hisi_mem_dev *hdev, u8 cmd, > + struct hisi_mem_desc *desc) The operation isn't done in place, so I'd split this into request and response. They can use same data if that actually makes sense. I'm not sure it does though. > +{ > + const struct hisi_mem_verspecific_data *verspec_data = hdev->verspec_data; > + struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; > + struct platform_device *pdev = hdev->pdev; > + void __iomem *comm_space; > + u16 space_size; > + int ret; > + > + comm_space = cl_info->pcc_comm_addr + verspec_data->shared_mem_size; > + space_size = HISI_MEM_PCC_SHARE_MEM_BYTES - verspec_data->shared_mem_size; > + verspec_data->fill_pcc_shared_mem(hdev, cmd, desc, > + comm_space, space_size); > + if (verspec_data->has_txdone_irq) > + reinit_completion(&cl_info->done); > + > + /* Ring doorbell */ > + ret = mbox_send_message(cl_info->mbox_chan, &cmd); > + if (ret < 0) { > + dev_err(&pdev->dev, "Send PCC mbox message failed, ret = %d.\n", > + ret); > + goto end; > + } > + > + ret = verspec_data->wait_cmd_complete(hdev); > + if (ret) > + goto end; > + > + /* Copy response data */ > + memcpy_fromio((void *)desc, comm_space, space_size); Should be no need to cast to void * > + > +end: > + if (verspec_data->has_txdone_irq) > + mbox_chan_txdone(cl_info->mbox_chan, ret); > + else > + mbox_client_txdone(cl_info->mbox_chan, ret); > + return ret; > +} > + > +int hisi_mem_do_acls_query(struct hisi_mem_dev *hdev, int *status, u64 paddr) > +{ > + struct platform_device *pdev = hdev->pdev; > + struct hisi_mem_desc desc; > + int ret; > + > + memset(&desc, 0, sizeof(desc)); > + desc.req.paddr = paddr; > + desc.req.subcmd = HISI_MEM_RAS_QUERY_CAP; > + ret = hisi_mem_pcc_cmd_send(hdev, HISI_MEM_RAS_ACLS, &desc); Similar comments to below. > + if (ret) { > + dev_err(&pdev->dev, "do acls query failed, ret = %d.\n", ret); > + return ret; > + } > + > + *status = desc.rsp.retStatus; > + return 0; > +} > + > +static ssize_t acls_query_store(struct kobject *kobj, > + struct kobj_attribute *attr, > + const char *buf, size_t count) > +{ > + struct hisi_mem_dev *hdev, *tmp; > + int status, ret; > + u64 paddr; > + > + if (kstrtoull(buf, 16, &paddr)) > + return -EINVAL; > + > + list_for_each_entry_safe(hdev, tmp, &mras->list, list) { > + struct platform_device *pdev = hdev->pdev; > + > + if (paddr < hdev->paddr_min || paddr > hdev->paddr_max) > + continue; > + > + ret = hisi_mem_do_acls_query(hdev, &status, paddr); > + if (ret) > + return ret; > + > + if (status == HISI_MEM_ACLS_OK) { > + return count; As below, I'd rather see errors out of line and good path inline. if (status == HISI_MEM_ACLS_QUERY_NO_RES) { } else if (status != HSI_MEM_ACLS_OK) } return count; It's a minor thing but if people are reviewing a lot of code this convention saves them thinking about what is good path and what is bad. > + } else if (status == HISI_MEM_ACLS_QUERY_NO_RES) { > + dev_info(&pdev->dev, "acls query no resource left.\n"); > + return -ENOSPC; > + } > + > + dev_err(&pdev->dev, "acls query error code %d.\n", status); > + return -EIO; > + } > + > + return -ENODEV; > +} > + > +static struct kobj_attribute acls_query_store_attribute = > + __ATTR(acls_query, 0200, NULL, acls_query_store); > + > +int hisi_mem_do_acls_repair(struct hisi_mem_dev *hdev, int *status, u64 paddr) > +{ > + struct hisi_mem_desc desc; > + struct platform_device *pdev; > + int ret; > + > + memset(&desc, 0, sizeof(desc)); > + desc.req.paddr = paddr; > + desc.req.subcmd = HISI_MEM_RAS_DO_REPAIR; struct hisi_mem_desc desc = { .req = { .paddr = paddr, .subcmd = HISI_MEM_RAS_DO_REPAIR, }, }; Probably better than meset unless the descriptor has holes that we are relying on filling with zero. If so add a comment. > + ret = hisi_mem_pcc_cmd_send(hdev, HISI_MEM_RAS_ACLS, &desc); As below, I'd use separate storage for req and rsp so we can type them all the way through. They are tiny so who cares about overhead ;) > + if (ret) { > + dev_err(&pdev->dev, "do acls repair failed, ret = %d.\n", ret); > + return ret; > + } > + > + *status = desc.rsp.retStatus; > + return 0; > +} > + > +static ssize_t acls_repair_store(struct kobject *kobj, > + struct kobj_attribute *attr, > + const char *buf, size_t count) > +{ > + struct hisi_mem_dev *hdev, *tmp; > + u64 paddr, ret; > + int status; > + > + if (kstrtoull(buf, 16, &paddr)) > + return -EINVAL; > + > + list_for_each_entry_safe(hdev, tmp, &mras->list, list) { > + struct platform_device *pdev = hdev->pdev; > + > + if (paddr < hdev->paddr_min || paddr > hdev->paddr_max) > + continue; Hmm. So this is why the unified control. I suspect the question that an upstream proposal would ask is why can't your instances all have their own controls + padd_min/max exposed in sysfs then issue repair on the right ones only. A generic interfaces is going to include info on granularity as well. How big is the repair? > + > + ret = hisi_mem_do_acls_repair(hdev, &status, paddr); > + if (ret) > + return ret; > + > + if (status == HISI_MEM_ACLS_OK) > + return count; Flip this so error condition out of line not the good one. > + > + dev_err(&pdev->dev, "acls repair error code %d.\n", status); > + return -EIO; > + } > + > + return -ENODEV; > +} > +static struct kobj_attribute acls_repair_store_attribute = > + __ATTR(acls_repair, 0200, NULL, acls_repair_store); > + > +static struct attribute *acls_attrs[] = { > + &acls_query_store_attribute.attr, > + &acls_repair_store_attribute.attr, > + NULL, No comma after null terminators. > +}; > + > +static struct attribute_group acls_attr_group = { > + .attrs = acls_attrs, > +}; AS these are associated with init / exit not probe / remove, move them down to make that association more obvious > + > +static int hisi_mem_ras_probe(struct platform_device *pdev) > +{ > + struct hisi_mem_dev *hdev; > + int ret = -ENOMEM; > + > + hdev = kzalloc(sizeof(struct hisi_mem_dev), GFP_KERNEL); > + if (!hdev) > + return -ENOMEM; > + > + hdev->pdev = pdev; > + ret = hisi_mem_get_pcc_chan_id(hdev); > + if (ret) > + goto free_hdev; > + > + ret = hisi_mem_register_pcc_channel(hdev); This lot all needs tearing down in remove.. > + if (ret) > + goto free_hdev; > + > + platform_set_drvdata(pdev, hdev); > + list_add_tail(&hdev->list, &mras->list); > + hdev->verspec_data = &hisi04b2_verspec_data; > + return 0; > + > +free_hdev: > + kfree(hdev); > + return ret; > +} > + > +static int hisi_mem_ras_remove(struct platform_device *pdev) > +{ > + struct hisi_mem_dev *hdev = platform_get_drvdata(pdev); > + > + platform_set_drvdata(pdev, NULL); Should be no need to do that. > + list_del(&hdev->list); Use a devm_add_action_or_reset() for this. > + > + return 0; > +} > + > +static const struct acpi_device_id hisi_mem_acpi_match[] = { > + { "HISI0FF1", 0}, > + { }, No need for comma on null terminator. > +}; > +MODULE_DEVICE_TABLE(acpi, hisi_mem_acpi_match); > + > +static struct platform_driver hisi_mem_ras_driver = { > + .probe = hisi_mem_ras_probe, > + .remove = hisi_mem_ras_remove, > + .driver = { > + .name = "hisi_mem_ras", > + .acpi_match_table = hisi_mem_acpi_match, > + }, > +}; > + > +static int __init hisi_mem_ras_init(void) > +{ > + int ret; > + > + mras = kzalloc(sizeof(struct hisi_mem_ras), GFP_KERNEL); sizeof(*mras) > + if (!mras) > + return -ENOMEM; > + > + mras->acls_kobj = kobject_create_and_add("acls", kernel_kobj); > + if (!mras->acls_kobj) { > + kfree(mras); > + return -ENOMEM; > + } > + > + ret = sysfs_create_group(mras->acls_kobj, &acls_attr_group); > + if (ret) { > + mras->acls_kobj = NULL; > + kobject_put(mras->acls_kobj); Put something you just made NULL? Use a goto for this stuff so it's consistent. > + return ret; > + } This is unusual to see. I guess ABI docs will make it clear what the intent is. Gut feeling is it won't be upstreamable though. If the aim is bringing the various PPR control instances together then that's what the RAS feature control proposal is all about. > + > + ret = platform_driver_register(&hisi_mem_ras_driver); > + if (ret) { > + sysfs_remove_group(mras->acls_kobj, &acls_attr_group); > + kobject_put(mras->acls_kobj); > + kfree(mras); > + return ret; > + } > + > + INIT_LIST_HEAD(&mras->list); > + > + return 0; > +} > +module_init(hisi_mem_ras_init); > + > +static void __exit hisi_mem_ras_exit(void) > +{ > + platform_driver_unregister(&hisi_mem_ras_driver); > + sysfs_remove_group(mras->acls_kobj, &acls_attr_group); > + kobject_put(mras->acls_kobj); > + kfree(mras); > +} > +module_exit(hisi_mem_ras_exit); > + > +MODULE_LICENSE("GPL v2"); > +MODULE_AUTHOR("Xiaofei Tan <tanxiaofei(a)huawei.com>"); > +MODULE_DESCRIPTION("HISILICON Memory RAS driver"); > +MODULE_ALIAS("platform:" DRV_NAME); > diff --git a/drivers/soc/hisilicon/hisi_mem_ras.h b/drivers/soc/hisilicon/hisi_mem_ras.h > new file mode 100644 > index 000000000000..a9acbb0c1012 > --- /dev/null > +++ b/drivers/soc/hisilicon/hisi_mem_ras.h > @@ -0,0 +1,84 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright (C) Huawei Technologies Co., Ltd. 2023. All rights reserved. If posting somewhere, update the year. > + */ > + > +#ifndef _HISI_MEM_RAS_H > +#define _HISI_MEM_RAS_H > + > +struct hisi_mem_ras { > + struct list_head list; > + struct kobject *acls_kobj; > +}; > + > +struct hisi_mem_mbox_client_info { > + struct mbox_client client; > + struct mbox_chan *mbox_chan; > + struct pcc_mbox_chan *pcc_chan; > + u64 deadline_us; > + void __iomem *pcc_comm_addr; > + struct completion done; > +}; > + > +struct hisi_mem_desc; > +struct hisi_mem_dev; > + > +struct hisi_mem_verspecific_data { > + void (*rx_callback)(struct mbox_client *cl, void *mssg); > + int (*wait_cmd_complete)(struct hisi_mem_dev *hdev); > + void (*fill_pcc_shared_mem)(struct hisi_mem_dev *hdev, > + u8 cmd, struct hisi_mem_desc *desc, As above, I'm not convinced a fill should take the union. You only fill with requests. > + void __iomem *comm_space, > + u16 space_size); > + u16 shared_mem_size; > + bool has_txdone_irq; > +}; > + > +struct hisi_mem_dev { > + struct list_head list; > + struct platform_device *pdev; > + //struct acpi_device *acpi_dev; Tidy that up. > + const struct hisi_mem_verspecific_data *verspec_data; > + /* device capabilities from firmware, like hisi_mem_CAPS_xxx. */ > + u64 caps; > + u8 chan_id; > + u64 paddr_min; > + u64 paddr_max; > + struct mutex lock; > + struct hisi_mem_mbox_client_info cl_info; > +}; > + > +#define HISI_MEM_PCC_SHARE_MEM_BYTES 128 > +enum hisi_mem_ras_cmd_type { > + HISI_MEM_RAS_ACLS = 0x4, > + HISI_MEM_RAS_PPR = 0X5, > +}; > + > +enum hisi_mem_ras_subcmd_type { > + HISI_MEM_RAS_QUERY_CAP = 0x0, > + HISI_MEM_RAS_DO_REPAIR = 0x1, > +}; > + > +struct hisi_mem_req_desc { > + u64 paddr; > + u8 subcmd; > + u8 rsv[3]; > +}; > + > +#define HISI_MEM_ACLS_OK 0 > +#define HISI_MEM_ACLS_QUERY_NO_RES 1 > +#define HISI_MEM_ACLS_REPAIR_FAIL 2 > +#define HISI_MEM_ACLS_EINVAL 3 > +struct hisi_mem_rsp_desc { > + u8 retStatus; /* 0: success, other: failure */ > + u8 rsv[7]; > +}; > + > +struct hisi_mem_desc { > + union { > + struct hisi_mem_req_desc req; > + struct hisi_mem_rsp_desc rsp; > + }; > +}; > + > +#endif

1 0

Re: [PATCH] soc: hisilicon: Support memory repair driver on Kunpeng SoC
by Shiju Jose 18 Jul '24

18 Jul '24

Hi Tanxiaofei, Some more comments inline. >-----Original Message----- >From: tanxiaofei <tanxiaofei(a)huawei.com> >Sent: 09 July 2024 10:34 >To: Shiju Jose <shiju.jose(a)huawei.com>; Jonathan Cameron ><jonathan.cameron(a)huawei.com>; Mauro Carvalho Chehab ><M.Chehab(a)huawei.com>; Roberto Sassu <roberto.sassu(a)huawei.com>; >Guohanjun (Hanjun Guo) <guohanjun(a)huawei.com> >Cc: linuxarm(a)openeuler.org; tanxiaofei <tanxiaofei(a)huawei.com> >Subject: [PATCH] soc: hisilicon: Support memory repair driver on Kunpeng SoC > >Kunpeng SoC support several memory repair capability. Normally, such repairs >are done in firmware startup stage or triggered by BMC(Baseboard >Management Controller) and done in firmware in runtime stage. OS can't >perceive such capability. > >Some support online repair, while others can't. One reason of not supporting >online repair is that the memory is in use and the hardware does not support >backpressure on the primary access channel. > >In order to support more online memory repair, we seek solutions from OS. That >is try to isolate the memory page first, and do repair, and de-isloate after repair >done. > >This patch support online ACLS(Adapter Cache Line Sparing) repair, and may add >other repair such as PPR in future. > >Signed-off-by: Xiaofei Tan <tanxiaofei(a)huawei.com> >--- > drivers/soc/hisilicon/Kconfig | 10 + > drivers/soc/hisilicon/Makefile | 1 + > drivers/soc/hisilicon/hisi_mem_ras.c | 553 +++++++++++++++++++++++++++ >drivers/soc/hisilicon/hisi_mem_ras.h | 84 ++++ > 4 files changed, 648 insertions(+) > create mode 100644 drivers/soc/hisilicon/hisi_mem_ras.c > create mode 100644 drivers/soc/hisilicon/hisi_mem_ras.h > >diff --git a/drivers/soc/hisilicon/Kconfig b/drivers/soc/hisilicon/Kconfig index >ae8e55a776e0..983cafc38e5c 100644 >--- a/drivers/soc/hisilicon/Kconfig >+++ b/drivers/soc/hisilicon/Kconfig >@@ -44,4 +44,14 @@ config HISI_HBMCACHE > To compile the driver as a module, choose M here: > the module will be called hisi_hbmcache. > >+config HISI_MEM_RAS >+ tristate "Add support for HISI Memory repair" >+ depends on ACPI >+ help >+ Add Memory repair support for Hisilicon device, which can be used to >query >+ memory repair cap and repair hardware error in memory devices. This >feature >+ need to work with hardware firmwares. >+ >+ If not sure say no. >+ > endmenu >diff --git a/drivers/soc/hisilicon/Makefile b/drivers/soc/hisilicon/Makefile index >4ef8823250e8..b339a67bcd77 100644 >--- a/drivers/soc/hisilicon/Makefile >+++ b/drivers/soc/hisilicon/Makefile >@@ -6,3 +6,4 @@ obj-$(CONFIG_KUNPENG_HCCS) += kunpeng_hccs.o > obj-$(CONFIG_HISI_HBMDEV) += hisi_hbmdev.o > obj-$(CONFIG_HISI_HBMCACHE) += hisi_hbmcache.o > obj-$(CONFIG_ARM64_PBHA) += pbha.o >+obj-$(CONFIG_HISI_MEM_RAS) += hisi_mem_ras.o >diff --git a/drivers/soc/hisilicon/hisi_mem_ras.c >b/drivers/soc/hisilicon/hisi_mem_ras.c >new file mode 100644 >index 000000000000..a29c41ccdba1 >--- /dev/null >+++ b/drivers/soc/hisilicon/hisi_mem_ras.c >@@ -0,0 +1,553 @@ >+// SPDX-License-Identifier: GPL-2.0 >+/* >+ * Copyright (C) Huawei Technologies Co., Ltd. 2024. All rights reserved. >+ */ >+ >+#include <linux/kobject.h> >+#include <linux/module.h> >+#include <linux/acpi.h> >+#include <linux/iopoll.h> >+#include <linux/platform_device.h> >+#include <linux/mm.h> >+#include <acpi/pcc.h> >+ >+#include "hisi_mem_ras.h" >+#define DRV_NAME "hisi_mem_ras" >+ >+#define HISI_MEM_PCC_CMD_WAIT_RETRIES_NUM 500ULL >+#define HISI_MEM_POLL_STATUS_TIME_INTERVAL_US 3 >+ >+static struct hisi_mem_ras *mras; >+ >+struct hisi_mem_register_ctx { >+ struct device *dev; >+ u8 chan_id; >+ int err; >+ u64 paddr_min; >+ u64 paddr_max; >+}; >+ >+static acpi_status hisi_mem_get_chan_id_cb(struct acpi_resource *ares, >+ void *context) >+{ >+ struct acpi_resource_generic_register *reg; >+ struct hisi_mem_register_ctx *ctx = context; >+ >+ if (ares->type != ACPI_RESOURCE_TYPE_GENERIC_REGISTER) >+ return AE_OK; >+ >+ reg = &ares->data.generic_reg; >+ if (reg->space_id != ACPI_ADR_SPACE_PLATFORM_COMM) { >+ dev_err(ctx->dev, "Bad register resource.\n"); >+ ctx->err = -EINVAL; >+ return AE_ERROR; >+ } >+ ctx->chan_id = reg->access_size; Passing PCC channel ID in register access size may not be correct. >+ >+ return AE_OK; >+} >+ >+static int hisi_mem_get_pcc_chan_id(struct hisi_mem_dev *hdev) { >+ struct platform_device *pdev = hdev->pdev; >+ struct acpi_device *adev = ACPI_COMPANION(&pdev->dev); >+ struct hisi_mem_register_ctx ctx = {0}; >+ acpi_handle handle = adev->handle; >+ acpi_status status; >+ >+ if (!acpi_has_method(handle, METHOD_NAME__CRS)) { >+ dev_err(&pdev->dev, "No _CRS method.\n"); >+ return -ENODEV; >+ } >+ >+ ctx.dev = &pdev->dev; >+ status = acpi_walk_resources(handle, METHOD_NAME__CRS, >+ hisi_mem_get_chan_id_cb, &ctx); >+ if (ACPI_FAILURE(status)) >+ return ctx.err; >+ Please add following here as hdev->chan_id is not set. hdev->chan_id = ctx->chan_id; >+ return 0; >+} >+ >+static acpi_status hisi_mem_get_paddr_range_cb(struct acpi_resource *ares, >+ void *context) >+{ >+ struct hisi_mem_register_ctx *ctx = context; >+ struct acpi_resource_address64 *addr64; >+ >+ if (ares->type != ACPI_RESOURCE_TYPE_ADDRESS64) >+ return AE_OK; >+ >+ addr64 = &ares->data.address64; >+ ctx->paddr_min = addr64->address.minimum; >+ ctx->paddr_max = addr64->address.maximum; >+ >+ return AE_OK; >+} >+ >+static int hisi_mem_get_paddr_range(struct hisi_mem_dev *hdev) { >+ struct platform_device *pdev = hdev->pdev; >+ struct acpi_device *adev = ACPI_COMPANION(&pdev->dev); >+ struct hisi_mem_register_ctx ctx = {0}; >+ acpi_handle handle = adev->handle; >+ acpi_status status; >+ >+ if (!acpi_has_method(handle, METHOD_NAME__CRS)) { >+ dev_err(&pdev->dev, "No _CRS method.\n"); >+ return -ENODEV; >+ } >+ >+ ctx.dev = &pdev->dev; >+ status = acpi_walk_resources(handle, METHOD_NAME__CRS, >+ hisi_mem_get_paddr_range_cb, &ctx); >+ if (ACPI_FAILURE(status)) >+ return ctx.err; >+ >+ hdev->paddr_min = ctx.paddr_min; >+ hdev->paddr_max = ctx.paddr_max; >+ return 0; >+} >+ >+static void hisi_mem_chan_tx_done(struct mbox_client *cl, void *msg, >+int ret) { >+ if (ret < 0) >+ pr_debug("TX did not complete: CMD sent:0x%x, ret:%d\n", >+ *(u8 *)msg, ret); >+ else >+ pr_debug("TX completed. CMD sent:0x%x, ret:%d\n", >+ *(u8 *)msg, ret); >+} >+ >+static void hisi_mem_pcc_rx_callback(struct mbox_client *cl, void >+*mssg) { >+ struct hisi_mem_mbox_client_info *cl_info = >+ container_of(cl, struct hisi_mem_mbox_client_info, >client); >+ >+ complete(&cl_info->done); >+} >+ >+static void hisi_mem_unregister_pcc_channel(struct hisi_mem_dev *hdev) >+{ >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ >+ if (cl_info->pcc_comm_addr) >+ iounmap(cl_info->pcc_comm_addr); >+ pcc_mbox_free_channel(hdev->cl_info.pcc_chan); >+} >+ >+static int hisi_mem_register_pcc_channel(struct hisi_mem_dev *hdev) { >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct mbox_client *cl = &cl_info->client; >+ struct pcc_mbox_chan *pcc_chan; >+ struct platform_device *pdev = hdev->pdev; >+ int rc; >+ >+ cl->dev = &pdev->dev; >+ cl->tx_block = false; >+ cl->knows_txdone = true; >+ cl->tx_done = hisi_mem_chan_tx_done; >+ cl->rx_callback = hdev->verspec_data->rx_callback; >+ init_completion(&cl_info->done); >+ >+ pcc_chan = pcc_mbox_request_channel(cl, hdev->chan_id); Where the hdev->chan_id is being set? I think it is the same ctx->chan_id set in the hisi_mem_get_chan_id_cb() ? >+ if (IS_ERR(pcc_chan)) { >+ dev_err(&pdev->dev, "PPC channel request failed.\n"); PPC -> PCC >+ rc = -ENODEV; >+ goto out; >+ } >+ cl_info->pcc_chan = pcc_chan; >+ cl_info->mbox_chan = pcc_chan->mchan; >+ >+ /* >+ * pcc_chan->latency is just a nominal value. In reality the remote >+ * processor could be much slower to reply. So add an arbitrary amount >+ * of wait on top of nominal. >+ */ >+ cl_info->deadline_us = >+ HISI_MEM_PCC_CMD_WAIT_RETRIES_NUM * >pcc_chan->latency; >+ if (!hdev->verspec_data->has_txdone_irq && >+ cl_info->mbox_chan->mbox->txdone_irq) { >+ dev_err(&pdev->dev, "PCC IRQ in PCCT is enabled.\n"); >+ rc = -EINVAL; >+ goto err_mbx_channel_free; >+ } else if (hdev->verspec_data->has_txdone_irq && >+ !cl_info->mbox_chan->mbox->txdone_irq) { >+ dev_err(&pdev->dev, "PCC IRQ in PCCT isn't supported.\n"); >+ rc = -EINVAL; >+ goto err_mbx_channel_free; >+ } >+ >+ if (pcc_chan->shmem_base_addr) { >+ cl_info->pcc_comm_addr = ioremap(pcc_chan- >>shmem_base_addr, >+ pcc_chan->shmem_size); >+ if (!cl_info->pcc_comm_addr) { >+ dev_err(&pdev->dev, "Failed to ioremap PCC >communication region for channel-%u.\n", >+ hdev->chan_id); >+ rc = -ENOMEM; >+ goto err_mbx_channel_free; >+ } >+ } >+ >+ return 0; >+ >+err_mbx_channel_free: >+ pcc_mbox_free_channel(cl_info->pcc_chan); >+out: >+ return rc; >+} >+ >+ >+static int hisi_mem_wait_cmd_complete_by_poll(struct hisi_mem_dev >+*hdev) { >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct acpi_pcct_shared_memory __iomem *comm_base = >+ cl_info- >>pcc_comm_addr; >+ struct platform_device *pdev = hdev->pdev; >+ u16 status; >+ int ret; >+ >+ /* >+ * Poll PCC status register every 3us(delay_us) for maximum of >+ * deadline_us(timeout_us) until PCC command complete bit is set(cond) >+ */ >+ ret = readw_poll_timeout(&comm_base->status, status, >+ status & PCC_STATUS_CMD_COMPLETE, >+ HISI_MEM_POLL_STATUS_TIME_INTERVAL_US, >+ cl_info->deadline_us); >+ if (unlikely(ret)) >+ dev_err(&pdev->dev, "poll PCC status failed, ret = %d.\n", ret); >+ >+ return ret; >+} >+ >+static int hisi_mem_wait_cmd_complete_by_irq(struct hisi_mem_dev *hdev) >+{ >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct platform_device *pdev = hdev->pdev; >+ >+ if (!wait_for_completion_timeout(&cl_info->done, >+ usecs_to_jiffies(cl_info->deadline_us))) { >+ dev_err(&pdev->dev, "PCC command executed timeout!\n"); >+ return -ETIMEDOUT; >+ } >+ >+ return 0; >+} >+ >+static inline void hisi_mem_fill_pcc_shared_mem_region(struct hisi_mem_dev >*hdev, >+ u8 cmd, >+ struct hisi_mem_desc *desc, >+ void __iomem *comm_space, >+ u16 space_size) >+{ >+ struct acpi_pcct_shared_memory tmp = { >+ .signature = PCC_SIGNATURE | hdev->chan_id, >+ .command = cmd, >+ .status = 0, >+ }; >+ >+ memcpy_toio(hdev->cl_info.pcc_comm_addr, (void *)&tmp, >+ sizeof(struct acpi_pcct_shared_memory)); >+ >+ /* Copy the message to the PCC comm space */ >+ memcpy_toio(comm_space, (void *)desc, space_size); } >+ >+static inline void hisi_mem_fill_ext_pcc_shared_mem_region(struct >hisi_mem_dev *hdev, >+ u8 cmd, >+ struct hisi_mem_desc >*desc, >+ void __iomem >*comm_space, >+ u16 space_size) >+{ >+ struct acpi_pcct_ext_pcc_shared_memory tmp = { >+ .signature = PCC_SIGNATURE | hdev->chan_id, >+ .flags = PCC_CMD_COMPLETION_NOTIFY, >+ .length = HISI_MEM_PCC_SHARE_MEM_BYTES, >+ .command = cmd, >+ }; >+ >+ memcpy_toio(hdev->cl_info.pcc_comm_addr, (void *)&tmp, >+ sizeof(struct acpi_pcct_ext_pcc_shared_memory)); >+ >+ /* Copy the message to the PCC comm space */ >+ memcpy_toio(comm_space, (void *)desc, space_size); } >+ >+static const struct hisi_mem_verspecific_data hisi04b1_verspec_data = { >+ .rx_callback = NULL, >+ .wait_cmd_complete = hisi_mem_wait_cmd_complete_by_poll, >+ .fill_pcc_shared_mem = hisi_mem_fill_pcc_shared_mem_region, >+ .shared_mem_size = sizeof(struct acpi_pcct_shared_memory), >+ .has_txdone_irq = false, >+}; >+ >+static const struct hisi_mem_verspecific_data hisi04b2_verspec_data = { >+ .rx_callback = hisi_mem_pcc_rx_callback, >+ .wait_cmd_complete = hisi_mem_wait_cmd_complete_by_irq, >+ .fill_pcc_shared_mem = hisi_mem_fill_ext_pcc_shared_mem_region, >+ .shared_mem_size = sizeof(struct acpi_pcct_ext_pcc_shared_memory), >+ .has_txdone_irq = true, >+}; >+ >+static int hisi_mem_pcc_cmd_send(struct hisi_mem_dev *hdev, u8 cmd, >+ struct hisi_mem_desc *desc) >+{ >+ const struct hisi_mem_verspecific_data *verspec_data = hdev- >>verspec_data; >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct platform_device *pdev = hdev->pdev; >+ void __iomem *comm_space; >+ u16 space_size; >+ int ret; >+ >+ comm_space = cl_info->pcc_comm_addr + verspec_data- >>shared_mem_size; >+ space_size = HISI_MEM_PCC_SHARE_MEM_BYTES - verspec_data- >>shared_mem_size; >+ verspec_data->fill_pcc_shared_mem(hdev, cmd, desc, >+ comm_space, space_size); >+ if (verspec_data->has_txdone_irq) >+ reinit_completion(&cl_info->done); >+ >+ /* Ring doorbell */ >+ ret = mbox_send_message(cl_info->mbox_chan, &cmd); >+ if (ret < 0) { >+ dev_err(&pdev->dev, "Send PCC mbox message failed, ret = >%d.\n", >+ ret); >+ goto end; >+ } >+ >+ ret = verspec_data->wait_cmd_complete(hdev); >+ if (ret) >+ goto end; >+ >+ /* Copy response data */ >+ memcpy_fromio((void *)desc, comm_space, space_size); >+ >+end: >+ if (verspec_data->has_txdone_irq) >+ mbox_chan_txdone(cl_info->mbox_chan, ret); >+ else >+ mbox_client_txdone(cl_info->mbox_chan, ret); >+ return ret; >+} >+ >+int hisi_mem_do_acls_query(struct hisi_mem_dev *hdev, int *status, u64 >+paddr) { >+ struct platform_device *pdev = hdev->pdev; >+ struct hisi_mem_desc desc; >+ int ret; >+ >+ memset(&desc, 0, sizeof(desc)); >+ desc.req.paddr = paddr; >+ desc.req.subcmd = HISI_MEM_RAS_QUERY_CAP; >+ ret = hisi_mem_pcc_cmd_send(hdev, HISI_MEM_RAS_ACLS, &desc); >+ if (ret) { >+ dev_err(&pdev->dev, "do acls query failed, ret = %d.\n", ret); >+ return ret; >+ } >+ >+ *status = desc.rsp.retStatus; >+ return 0; >+} >+ >+static ssize_t acls_query_store(struct kobject *kobj, >+ struct kobj_attribute *attr, >+ const char *buf, size_t count) >+{ >+ struct hisi_mem_dev *hdev, *tmp; >+ int status, ret; >+ u64 paddr; >+ >+ if (kstrtoull(buf, 16, &paddr)) >+ return -EINVAL; >+ >+ list_for_each_entry_safe(hdev, tmp, &mras->list, list) { >+ struct platform_device *pdev = hdev->pdev; >+ >+ if (paddr < hdev->paddr_min || paddr > hdev->paddr_max) >+ continue; >+ >+ ret = hisi_mem_do_acls_query(hdev, &status, paddr); >+ if (ret) >+ return ret; >+ >+ if (status == HISI_MEM_ACLS_OK) { >+ return count; >+ } else if (status == HISI_MEM_ACLS_QUERY_NO_RES) { >+ dev_info(&pdev->dev, "acls query no resource left.\n"); >+ return -ENOSPC; >+ } >+ >+ dev_err(&pdev->dev, "acls query error code %d.\n", status); >+ return -EIO; >+ } >+ >+ return -ENODEV; >+} >+ >+static struct kobj_attribute acls_query_store_attribute = >+ __ATTR(acls_query, 0200, NULL, acls_query_store); >+ >+int hisi_mem_do_acls_repair(struct hisi_mem_dev *hdev, int *status, u64 >+paddr) { >+ struct hisi_mem_desc desc; >+ struct platform_device *pdev; >+ int ret; >+ >+ memset(&desc, 0, sizeof(desc)); >+ desc.req.paddr = paddr; >+ desc.req.subcmd = HISI_MEM_RAS_DO_REPAIR; >+ ret = hisi_mem_pcc_cmd_send(hdev, HISI_MEM_RAS_ACLS, &desc); >+ if (ret) { >+ dev_err(&pdev->dev, "do acls repair failed, ret = %d.\n", ret); >+ return ret; >+ } >+ >+ *status = desc.rsp.retStatus; >+ return 0; >+} >+ >+static ssize_t acls_repair_store(struct kobject *kobj, >+ struct kobj_attribute *attr, >+ const char *buf, size_t count) >+{ >+ struct hisi_mem_dev *hdev, *tmp; >+ u64 paddr, ret; >+ int status; >+ >+ if (kstrtoull(buf, 16, &paddr)) >+ return -EINVAL; >+ >+ list_for_each_entry_safe(hdev, tmp, &mras->list, list) { >+ struct platform_device *pdev = hdev->pdev; >+ >+ if (paddr < hdev->paddr_min || paddr > hdev->paddr_max) >+ continue; >+ >+ ret = hisi_mem_do_acls_repair(hdev, &status, paddr); >+ if (ret) >+ return ret; >+ >+ if (status == HISI_MEM_ACLS_OK) >+ return count; >+ >+ dev_err(&pdev->dev, "acls repair error code %d.\n", status); >+ return -EIO; >+ } >+ >+ return -ENODEV; >+} >+static struct kobj_attribute acls_repair_store_attribute = >+ __ATTR(acls_repair, 0200, NULL, acls_repair_store); >+ >+static struct attribute *acls_attrs[] = { >+ &acls_query_store_attribute.attr, >+ &acls_repair_store_attribute.attr, >+ NULL, >+}; >+ >+static struct attribute_group acls_attr_group = { >+ .attrs = acls_attrs, >+}; >+ >+static int hisi_mem_ras_probe(struct platform_device *pdev) { >+ struct hisi_mem_dev *hdev; >+ int ret = -ENOMEM; >+ >+ hdev = kzalloc(sizeof(struct hisi_mem_dev), GFP_KERNEL); >+ if (!hdev) >+ return -ENOMEM; >+ >+ hdev->pdev = pdev; >+ ret = hisi_mem_get_pcc_chan_id(hdev); >+ if (ret) >+ goto free_hdev; >+ >+ ret = hisi_mem_register_pcc_channel(hdev); >+ if (ret) >+ goto free_hdev; >+ >+ platform_set_drvdata(pdev, hdev); >+ list_add_tail(&hdev->list, &mras->list); >+ hdev->verspec_data = &hisi04b2_verspec_data; 1. hdev->verspec_data is being accessed in the above hisi_mem_register_pcc_channel() function. Need to set hdev->verspec_data before calling hisi_mem_register_pcc_channel()? 2. Both hisi04b1_verspec_data and hisi04b2_verspec_data are defined, seems hisi04b1_verspec_data not used. >+ return 0; >+ >+free_hdev: >+ kfree(hdev); >+ return ret; >+} >+ >+static int hisi_mem_ras_remove(struct platform_device *pdev) { >+ struct hisi_mem_dev *hdev = platform_get_drvdata(pdev); >+ >+ platform_set_drvdata(pdev, NULL); >+ list_del(&hdev->list); >+ >+ return 0; >+} >+ >+static const struct acpi_device_id hisi_mem_acpi_match[] = { >+ { "HISI0FF1", 0}, >+ { }, >+}; >+MODULE_DEVICE_TABLE(acpi, hisi_mem_acpi_match); >+ >+static struct platform_driver hisi_mem_ras_driver = { >+ .probe = hisi_mem_ras_probe, >+ .remove = hisi_mem_ras_remove, >+ .driver = { >+ .name = "hisi_mem_ras", >+ .acpi_match_table = hisi_mem_acpi_match, >+ }, >+}; >+ >+static int __init hisi_mem_ras_init(void) { >+ int ret; >+ >+ mras = kzalloc(sizeof(struct hisi_mem_ras), GFP_KERNEL); >+ if (!mras) >+ return -ENOMEM; >+ >+ mras->acls_kobj = kobject_create_and_add("acls", kernel_kobj); >+ if (!mras->acls_kobj) { >+ kfree(mras); >+ return -ENOMEM; >+ } >+ >+ ret = sysfs_create_group(mras->acls_kobj, &acls_attr_group); >+ if (ret) { >+ mras->acls_kobj = NULL; This should be done after below kobject_put. >+ kobject_put(mras->acls_kobj); >+ return ret; >+ } >+ >+ ret = platform_driver_register(&hisi_mem_ras_driver); >+ if (ret) { >+ sysfs_remove_group(mras->acls_kobj, &acls_attr_group); >+ kobject_put(mras->acls_kobj); >+ kfree(mras); >+ return ret; >+ } >+ >+ INIT_LIST_HEAD(&mras->list); >+ >+ return 0; >+} >+module_init(hisi_mem_ras_init); >+ >+static void __exit hisi_mem_ras_exit(void) { >+ platform_driver_unregister(&hisi_mem_ras_driver); >+ sysfs_remove_group(mras->acls_kobj, &acls_attr_group); >+ kobject_put(mras->acls_kobj); >+ kfree(mras); >+} >+module_exit(hisi_mem_ras_exit); >+ >+MODULE_LICENSE("GPL v2"); >+MODULE_AUTHOR("Xiaofei Tan <tanxiaofei(a)huawei.com>"); >+MODULE_DESCRIPTION("HISILICON Memory RAS driver"); >+MODULE_ALIAS("platform:" DRV_NAME); >diff --git a/drivers/soc/hisilicon/hisi_mem_ras.h >b/drivers/soc/hisilicon/hisi_mem_ras.h >new file mode 100644 >index 000000000000..a9acbb0c1012 >--- /dev/null >+++ b/drivers/soc/hisilicon/hisi_mem_ras.h >@@ -0,0 +1,84 @@ >+/* SPDX-License-Identifier: GPL-2.0 */ >+/* >+ * Copyright (C) Huawei Technologies Co., Ltd. 2023. All rights reserved. >+ */ >+ >+#ifndef _HISI_MEM_RAS_H >+#define _HISI_MEM_RAS_H >+ >+struct hisi_mem_ras { >+ struct list_head list; >+ struct kobject *acls_kobj; >+}; >+ >+struct hisi_mem_mbox_client_info { >+ struct mbox_client client; >+ struct mbox_chan *mbox_chan; >+ struct pcc_mbox_chan *pcc_chan; >+ u64 deadline_us; >+ void __iomem *pcc_comm_addr; >+ struct completion done; >+}; >+ >+struct hisi_mem_desc; >+struct hisi_mem_dev; >+ >+struct hisi_mem_verspecific_data { >+ void (*rx_callback)(struct mbox_client *cl, void *mssg); >+ int (*wait_cmd_complete)(struct hisi_mem_dev *hdev); >+ void (*fill_pcc_shared_mem)(struct hisi_mem_dev *hdev, >+ u8 cmd, struct hisi_mem_desc *desc, >+ void __iomem *comm_space, >+ u16 space_size); >+ u16 shared_mem_size; >+ bool has_txdone_irq; >+}; >+ >+struct hisi_mem_dev { >+ struct list_head list; >+ struct platform_device *pdev; >+ //struct acpi_device *acpi_dev; >+ const struct hisi_mem_verspecific_data *verspec_data; >+ /* device capabilities from firmware, like hisi_mem_CAPS_xxx. */ >+ u64 caps; >+ u8 chan_id; >+ u64 paddr_min; >+ u64 paddr_max; >+ struct mutex lock; >+ struct hisi_mem_mbox_client_info cl_info; }; >+ >+#define HISI_MEM_PCC_SHARE_MEM_BYTES 128 >+enum hisi_mem_ras_cmd_type { >+ HISI_MEM_RAS_ACLS = 0x4, >+ HISI_MEM_RAS_PPR = 0X5, >+}; >+ >+enum hisi_mem_ras_subcmd_type { >+ HISI_MEM_RAS_QUERY_CAP = 0x0, >+ HISI_MEM_RAS_DO_REPAIR = 0x1, >+}; >+ >+struct hisi_mem_req_desc { >+ u64 paddr; >+ u8 subcmd; >+ u8 rsv[3]; >+}; >+ >+#define HISI_MEM_ACLS_OK 0 >+#define HISI_MEM_ACLS_QUERY_NO_RES 1 >+#define HISI_MEM_ACLS_REPAIR_FAIL 2 >+#define HISI_MEM_ACLS_EINVAL 3 >+struct hisi_mem_rsp_desc { >+ u8 retStatus; /* 0: success, other: failure */ >+ u8 rsv[7]; >+}; >+ >+struct hisi_mem_desc { >+ union { >+ struct hisi_mem_req_desc req; >+ struct hisi_mem_rsp_desc rsp; >+ }; >+}; >+ >+#endif >-- >2.33.0 Thanks, Shiju

1 0

Re: [PATCH] soc: hisilicon: Support memory repair driver on Kunpeng SoC
by Shiju Jose 10 Jul '24

10 Jul '24

Hi Tanxiaofei, Please find few comments inline after quick look. I will go through in more detail bit later. Thanks, Shiju >-----Original Message----- >From: tanxiaofei <tanxiaofei(a)huawei.com> >Sent: 09 July 2024 10:34 >To: Shiju Jose <shiju.jose(a)huawei.com>; Jonathan Cameron ><jonathan.cameron(a)huawei.com>; Mauro Carvalho Chehab ><M.Chehab(a)huawei.com>; Roberto Sassu <roberto.sassu(a)huawei.com>; >Guohanjun (Hanjun Guo) <guohanjun(a)huawei.com> >Cc: linuxarm(a)openeuler.org; tanxiaofei <tanxiaofei(a)huawei.com> >Subject: [PATCH] soc: hisilicon: Support memory repair driver on Kunpeng SoC > >Kunpeng SoC support several memory repair capability. Normally, such repairs >are done in firmware startup stage or triggered by BMC(Baseboard >Management Controller) and done in firmware in runtime stage. OS can't >perceive such capability. > >Some support online repair, while others can't. One reason of not supporting >online repair is that the memory is in use and the hardware does not support >backpressure on the primary access channel. > >In order to support more online memory repair, we seek solutions from OS. That >is try to isolate the memory page first, and do repair, and de-isloate after repair >done. > >This patch support online ACLS(Adapter Cache Line Sparing) repair, and may add >other repair such as PPR in future. > >Signed-off-by: Xiaofei Tan <tanxiaofei(a)huawei.com> >--- > drivers/soc/hisilicon/Kconfig | 10 + > drivers/soc/hisilicon/Makefile | 1 + > drivers/soc/hisilicon/hisi_mem_ras.c | 553 +++++++++++++++++++++++++++ >drivers/soc/hisilicon/hisi_mem_ras.h | 84 ++++ > 4 files changed, 648 insertions(+) > create mode 100644 drivers/soc/hisilicon/hisi_mem_ras.c > create mode 100644 drivers/soc/hisilicon/hisi_mem_ras.h > >diff --git a/drivers/soc/hisilicon/Kconfig b/drivers/soc/hisilicon/Kconfig index >ae8e55a776e0..983cafc38e5c 100644 >--- a/drivers/soc/hisilicon/Kconfig >+++ b/drivers/soc/hisilicon/Kconfig >@@ -44,4 +44,14 @@ config HISI_HBMCACHE > To compile the driver as a module, choose M here: > the module will be called hisi_hbmcache. > >+config HISI_MEM_RAS >+ tristate "Add support for HISI Memory repair" >+ depends on ACPI >+ help >+ Add Memory repair support for Hisilicon device, which can be used to >query >+ memory repair cap and repair hardware error in memory devices. This >feature >+ need to work with hardware firmwares. >+ >+ If not sure say no. >+ > endmenu >diff --git a/drivers/soc/hisilicon/Makefile b/drivers/soc/hisilicon/Makefile index >4ef8823250e8..b339a67bcd77 100644 >--- a/drivers/soc/hisilicon/Makefile >+++ b/drivers/soc/hisilicon/Makefile >@@ -6,3 +6,4 @@ obj-$(CONFIG_KUNPENG_HCCS) += kunpeng_hccs.o > obj-$(CONFIG_HISI_HBMDEV) += hisi_hbmdev.o > obj-$(CONFIG_HISI_HBMCACHE) += hisi_hbmcache.o > obj-$(CONFIG_ARM64_PBHA) += pbha.o >+obj-$(CONFIG_HISI_MEM_RAS) += hisi_mem_ras.o >diff --git a/drivers/soc/hisilicon/hisi_mem_ras.c >b/drivers/soc/hisilicon/hisi_mem_ras.c >new file mode 100644 >index 000000000000..a29c41ccdba1 >--- /dev/null >+++ b/drivers/soc/hisilicon/hisi_mem_ras.c >@@ -0,0 +1,553 @@ >+// SPDX-License-Identifier: GPL-2.0 >+/* >+ * Copyright (C) Huawei Technologies Co., Ltd. 2024. All rights reserved. >+ */ >+ >+#include <linux/kobject.h> >+#include <linux/module.h> >+#include <linux/acpi.h> >+#include <linux/iopoll.h> >+#include <linux/platform_device.h> >+#include <linux/mm.h> >+#include <acpi/pcc.h> >+ >+#include "hisi_mem_ras.h" >+#define DRV_NAME "hisi_mem_ras" >+ >+#define HISI_MEM_PCC_CMD_WAIT_RETRIES_NUM 500ULL >+#define HISI_MEM_POLL_STATUS_TIME_INTERVAL_US 3 >+ >+static struct hisi_mem_ras *mras; >+ >+struct hisi_mem_register_ctx { >+ struct device *dev; >+ u8 chan_id; >+ int err; >+ u64 paddr_min; >+ u64 paddr_max; >+}; >+ >+static acpi_status hisi_mem_get_chan_id_cb(struct acpi_resource *ares, >+ void *context) >+{ >+ struct acpi_resource_generic_register *reg; >+ struct hisi_mem_register_ctx *ctx = context; >+ >+ if (ares->type != ACPI_RESOURCE_TYPE_GENERIC_REGISTER) >+ return AE_OK; >+ >+ reg = &ares->data.generic_reg; >+ if (reg->space_id != ACPI_ADR_SPACE_PLATFORM_COMM) { >+ dev_err(ctx->dev, "Bad register resource.\n"); >+ ctx->err = -EINVAL; >+ return AE_ERROR; >+ } >+ ctx->chan_id = reg->access_size; >+ >+ return AE_OK; >+} >+ >+static int hisi_mem_get_pcc_chan_id(struct hisi_mem_dev *hdev) { >+ struct platform_device *pdev = hdev->pdev; >+ struct acpi_device *adev = ACPI_COMPANION(&pdev->dev); >+ struct hisi_mem_register_ctx ctx = {0}; >+ acpi_handle handle = adev->handle; >+ acpi_status status; >+ >+ if (!acpi_has_method(handle, METHOD_NAME__CRS)) { >+ dev_err(&pdev->dev, "No _CRS method.\n"); >+ return -ENODEV; >+ } >+ >+ ctx.dev = &pdev->dev; >+ status = acpi_walk_resources(handle, METHOD_NAME__CRS, >+ hisi_mem_get_chan_id_cb, &ctx); >+ if (ACPI_FAILURE(status)) >+ return ctx.err; >+ >+ return 0; >+} >+ >+static acpi_status hisi_mem_get_paddr_range_cb(struct acpi_resource *ares, >+ void *context) >+{ >+ struct hisi_mem_register_ctx *ctx = context; >+ struct acpi_resource_address64 *addr64; >+ >+ if (ares->type != ACPI_RESOURCE_TYPE_ADDRESS64) >+ return AE_OK; >+ >+ addr64 = &ares->data.address64; >+ ctx->paddr_min = addr64->address.minimum; >+ ctx->paddr_max = addr64->address.maximum; >+ >+ return AE_OK; >+} >+ >+static int hisi_mem_get_paddr_range(struct hisi_mem_dev *hdev) { >+ struct platform_device *pdev = hdev->pdev; >+ struct acpi_device *adev = ACPI_COMPANION(&pdev->dev); >+ struct hisi_mem_register_ctx ctx = {0}; >+ acpi_handle handle = adev->handle; >+ acpi_status status; >+ >+ if (!acpi_has_method(handle, METHOD_NAME__CRS)) { >+ dev_err(&pdev->dev, "No _CRS method.\n"); >+ return -ENODEV; >+ } >+ >+ ctx.dev = &pdev->dev; >+ status = acpi_walk_resources(handle, METHOD_NAME__CRS, >+ hisi_mem_get_paddr_range_cb, &ctx); >+ if (ACPI_FAILURE(status)) >+ return ctx.err; >+ >+ hdev->paddr_min = ctx.paddr_min; >+ hdev->paddr_max = ctx.paddr_max; >+ return 0; >+} >+ >+static void hisi_mem_chan_tx_done(struct mbox_client *cl, void *msg, >+int ret) { >+ if (ret < 0) >+ pr_debug("TX did not complete: CMD sent:0x%x, ret:%d\n", >+ *(u8 *)msg, ret); >+ else >+ pr_debug("TX completed. CMD sent:0x%x, ret:%d\n", >+ *(u8 *)msg, ret); >+} >+ >+static void hisi_mem_pcc_rx_callback(struct mbox_client *cl, void >+*mssg) { >+ struct hisi_mem_mbox_client_info *cl_info = >+ container_of(cl, struct hisi_mem_mbox_client_info, >client); >+ >+ complete(&cl_info->done); >+} >+ >+static void hisi_mem_unregister_pcc_channel(struct hisi_mem_dev *hdev) This function is not used. >+{ >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ >+ if (cl_info->pcc_comm_addr) >+ iounmap(cl_info->pcc_comm_addr); >+ pcc_mbox_free_channel(hdev->cl_info.pcc_chan); >+} >+ >+static int hisi_mem_register_pcc_channel(struct hisi_mem_dev *hdev) { >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct mbox_client *cl = &cl_info->client; >+ struct pcc_mbox_chan *pcc_chan; >+ struct platform_device *pdev = hdev->pdev; >+ int rc; >+ >+ cl->dev = &pdev->dev; >+ cl->tx_block = false; >+ cl->knows_txdone = true; >+ cl->tx_done = hisi_mem_chan_tx_done; >+ cl->rx_callback = hdev->verspec_data->rx_callback; >+ init_completion(&cl_info->done); >+ >+ pcc_chan = pcc_mbox_request_channel(cl, hdev->chan_id); >+ if (IS_ERR(pcc_chan)) { >+ dev_err(&pdev->dev, "PPC channel request failed.\n"); >+ rc = -ENODEV; >+ goto out; >+ } >+ cl_info->pcc_chan = pcc_chan; >+ cl_info->mbox_chan = pcc_chan->mchan; >+ >+ /* >+ * pcc_chan->latency is just a nominal value. In reality the remote >+ * processor could be much slower to reply. So add an arbitrary amount >+ * of wait on top of nominal. >+ */ >+ cl_info->deadline_us = >+ HISI_MEM_PCC_CMD_WAIT_RETRIES_NUM * >pcc_chan->latency; >+ if (!hdev->verspec_data->has_txdone_irq && >+ cl_info->mbox_chan->mbox->txdone_irq) { >+ dev_err(&pdev->dev, "PCC IRQ in PCCT is enabled.\n"); >+ rc = -EINVAL; >+ goto err_mbx_channel_free; >+ } else if (hdev->verspec_data->has_txdone_irq && >+ !cl_info->mbox_chan->mbox->txdone_irq) { >+ dev_err(&pdev->dev, "PCC IRQ in PCCT isn't supported.\n"); >+ rc = -EINVAL; >+ goto err_mbx_channel_free; >+ } >+ >+ if (pcc_chan->shmem_base_addr) { >+ cl_info->pcc_comm_addr = ioremap(pcc_chan- >>shmem_base_addr, >+ pcc_chan->shmem_size); >+ if (!cl_info->pcc_comm_addr) { >+ dev_err(&pdev->dev, "Failed to ioremap PCC >communication region for channel-%u.\n", >+ hdev->chan_id); >+ rc = -ENOMEM; >+ goto err_mbx_channel_free; >+ } >+ } >+ >+ return 0; >+ >+err_mbx_channel_free: >+ pcc_mbox_free_channel(cl_info->pcc_chan); >+out: >+ return rc; >+} >+ >+ >+static int hisi_mem_wait_cmd_complete_by_poll(struct hisi_mem_dev >+*hdev) { >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct acpi_pcct_shared_memory __iomem *comm_base = >+ cl_info- >>pcc_comm_addr; >+ struct platform_device *pdev = hdev->pdev; >+ u16 status; >+ int ret; >+ >+ /* >+ * Poll PCC status register every 3us(delay_us) for maximum of >+ * deadline_us(timeout_us) until PCC command complete bit is set(cond) >+ */ >+ ret = readw_poll_timeout(&comm_base->status, status, >+ status & PCC_STATUS_CMD_COMPLETE, >+ HISI_MEM_POLL_STATUS_TIME_INTERVAL_US, >+ cl_info->deadline_us); >+ if (unlikely(ret)) >+ dev_err(&pdev->dev, "poll PCC status failed, ret = %d.\n", ret); >+ >+ return ret; >+} >+ >+static int hisi_mem_wait_cmd_complete_by_irq(struct hisi_mem_dev *hdev) >+{ >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct platform_device *pdev = hdev->pdev; >+ >+ if (!wait_for_completion_timeout(&cl_info->done, >+ usecs_to_jiffies(cl_info->deadline_us))) { >+ dev_err(&pdev->dev, "PCC command executed timeout!\n"); >+ return -ETIMEDOUT; >+ } >+ >+ return 0; >+} >+ >+static inline void hisi_mem_fill_pcc_shared_mem_region(struct hisi_mem_dev >*hdev, >+ u8 cmd, >+ struct hisi_mem_desc *desc, >+ void __iomem *comm_space, >+ u16 space_size) >+{ >+ struct acpi_pcct_shared_memory tmp = { >+ .signature = PCC_SIGNATURE | hdev->chan_id, >+ .command = cmd, >+ .status = 0, >+ }; >+ >+ memcpy_toio(hdev->cl_info.pcc_comm_addr, (void *)&tmp, >+ sizeof(struct acpi_pcct_shared_memory)); >+ >+ /* Copy the message to the PCC comm space */ >+ memcpy_toio(comm_space, (void *)desc, space_size); } >+ >+static inline void hisi_mem_fill_ext_pcc_shared_mem_region(struct >hisi_mem_dev *hdev, >+ u8 cmd, >+ struct hisi_mem_desc >*desc, >+ void __iomem >*comm_space, >+ u16 space_size) >+{ >+ struct acpi_pcct_ext_pcc_shared_memory tmp = { >+ .signature = PCC_SIGNATURE | hdev->chan_id, >+ .flags = PCC_CMD_COMPLETION_NOTIFY, >+ .length = HISI_MEM_PCC_SHARE_MEM_BYTES, >+ .command = cmd, >+ }; >+ >+ memcpy_toio(hdev->cl_info.pcc_comm_addr, (void *)&tmp, >+ sizeof(struct acpi_pcct_ext_pcc_shared_memory)); >+ >+ /* Copy the message to the PCC comm space */ >+ memcpy_toio(comm_space, (void *)desc, space_size); } >+ >+static const struct hisi_mem_verspecific_data hisi04b1_verspec_data = { >+ .rx_callback = NULL, >+ .wait_cmd_complete = hisi_mem_wait_cmd_complete_by_poll, >+ .fill_pcc_shared_mem = hisi_mem_fill_pcc_shared_mem_region, >+ .shared_mem_size = sizeof(struct acpi_pcct_shared_memory), >+ .has_txdone_irq = false, >+}; >+ >+static const struct hisi_mem_verspecific_data hisi04b2_verspec_data = { >+ .rx_callback = hisi_mem_pcc_rx_callback, >+ .wait_cmd_complete = hisi_mem_wait_cmd_complete_by_irq, >+ .fill_pcc_shared_mem = hisi_mem_fill_ext_pcc_shared_mem_region, >+ .shared_mem_size = sizeof(struct acpi_pcct_ext_pcc_shared_memory), >+ .has_txdone_irq = true, >+}; >+ >+static int hisi_mem_pcc_cmd_send(struct hisi_mem_dev *hdev, u8 cmd, >+ struct hisi_mem_desc *desc) >+{ >+ const struct hisi_mem_verspecific_data *verspec_data = hdev- >>verspec_data; >+ struct hisi_mem_mbox_client_info *cl_info = &hdev->cl_info; >+ struct platform_device *pdev = hdev->pdev; >+ void __iomem *comm_space; >+ u16 space_size; >+ int ret; >+ >+ comm_space = cl_info->pcc_comm_addr + verspec_data- >>shared_mem_size; >+ space_size = HISI_MEM_PCC_SHARE_MEM_BYTES - verspec_data- >>shared_mem_size; >+ verspec_data->fill_pcc_shared_mem(hdev, cmd, desc, >+ comm_space, space_size); >+ if (verspec_data->has_txdone_irq) >+ reinit_completion(&cl_info->done); >+ >+ /* Ring doorbell */ >+ ret = mbox_send_message(cl_info->mbox_chan, &cmd); >+ if (ret < 0) { >+ dev_err(&pdev->dev, "Send PCC mbox message failed, ret = >%d.\n", >+ ret); >+ goto end; >+ } >+ >+ ret = verspec_data->wait_cmd_complete(hdev); >+ if (ret) >+ goto end; >+ >+ /* Copy response data */ >+ memcpy_fromio((void *)desc, comm_space, space_size); >+ >+end: >+ if (verspec_data->has_txdone_irq) >+ mbox_chan_txdone(cl_info->mbox_chan, ret); >+ else >+ mbox_client_txdone(cl_info->mbox_chan, ret); >+ return ret; >+} >+ >+int hisi_mem_do_acls_query(struct hisi_mem_dev *hdev, int *status, u64 >+paddr) { static function? >+ struct platform_device *pdev = hdev->pdev; >+ struct hisi_mem_desc desc; >+ int ret; >+ >+ memset(&desc, 0, sizeof(desc)); >+ desc.req.paddr = paddr; >+ desc.req.subcmd = HISI_MEM_RAS_QUERY_CAP; >+ ret = hisi_mem_pcc_cmd_send(hdev, HISI_MEM_RAS_ACLS, &desc); >+ if (ret) { >+ dev_err(&pdev->dev, "do acls query failed, ret = %d.\n", ret); >+ return ret; >+ } >+ >+ *status = desc.rsp.retStatus; >+ return 0; >+} >+ >+static ssize_t acls_query_store(struct kobject *kobj, >+ struct kobj_attribute *attr, >+ const char *buf, size_t count) >+{ >+ struct hisi_mem_dev *hdev, *tmp; >+ int status, ret; >+ u64 paddr; >+ >+ if (kstrtoull(buf, 16, &paddr)) >+ return -EINVAL; >+ >+ list_for_each_entry_safe(hdev, tmp, &mras->list, list) { >+ struct platform_device *pdev = hdev->pdev; >+ >+ if (paddr < hdev->paddr_min || paddr > hdev->paddr_max) >+ continue; >+ >+ ret = hisi_mem_do_acls_query(hdev, &status, paddr); >+ if (ret) >+ return ret; >+ >+ if (status == HISI_MEM_ACLS_OK) { >+ return count; >+ } else if (status == HISI_MEM_ACLS_QUERY_NO_RES) { >+ dev_info(&pdev->dev, "acls query no resource left.\n"); >+ return -ENOSPC; >+ } >+ >+ dev_err(&pdev->dev, "acls query error code %d.\n", status); >+ return -EIO; >+ } >+ >+ return -ENODEV; >+} >+ >+static struct kobj_attribute acls_query_store_attribute = >+ __ATTR(acls_query, 0200, NULL, acls_query_store); 0200 for write? 0444 if query is read only. May be we can use DEVICE_ATTR_RO/ DEVICE_ATTR_RW/ DEVICE_ATTR_WO instead? >+ >+int hisi_mem_do_acls_repair(struct hisi_mem_dev *hdev, int *status, u64 >+paddr) { static function? >+ struct hisi_mem_desc desc; >+ struct platform_device *pdev; >+ int ret; >+ >+ memset(&desc, 0, sizeof(desc)); >+ desc.req.paddr = paddr; >+ desc.req.subcmd = HISI_MEM_RAS_DO_REPAIR; >+ ret = hisi_mem_pcc_cmd_send(hdev, HISI_MEM_RAS_ACLS, &desc); >+ if (ret) { >+ dev_err(&pdev->dev, "do acls repair failed, ret = %d.\n", ret); >+ return ret; >+ } >+ >+ *status = desc.rsp.retStatus; >+ return 0; >+} >+ >+static ssize_t acls_repair_store(struct kobject *kobj, >+ struct kobj_attribute *attr, >+ const char *buf, size_t count) >+{ >+ struct hisi_mem_dev *hdev, *tmp; >+ u64 paddr, ret; >+ int status; >+ >+ if (kstrtoull(buf, 16, &paddr)) >+ return -EINVAL; >+ >+ list_for_each_entry_safe(hdev, tmp, &mras->list, list) { >+ struct platform_device *pdev = hdev->pdev; >+ >+ if (paddr < hdev->paddr_min || paddr > hdev->paddr_max) >+ continue; >+ >+ ret = hisi_mem_do_acls_repair(hdev, &status, paddr); >+ if (ret) >+ return ret; >+ >+ if (status == HISI_MEM_ACLS_OK) >+ return count; >+ >+ dev_err(&pdev->dev, "acls repair error code %d.\n", status); >+ return -EIO; >+ } >+ >+ return -ENODEV; >+} >+static struct kobj_attribute acls_repair_store_attribute = >+ __ATTR(acls_repair, 0200, NULL, acls_repair_store); >+ >+static struct attribute *acls_attrs[] = { >+ &acls_query_store_attribute.attr, >+ &acls_repair_store_attribute.attr, >+ NULL, >+}; >+ >+static struct attribute_group acls_attr_group = { >+ .attrs = acls_attrs, >+}; >+ >+static int hisi_mem_ras_probe(struct platform_device *pdev) { >+ struct hisi_mem_dev *hdev; >+ int ret = -ENOMEM; May not require to initialize here. >+ >+ hdev = kzalloc(sizeof(struct hisi_mem_dev), GFP_KERNEL); >+ if (!hdev) >+ return -ENOMEM; >+ >+ hdev->pdev = pdev; >+ ret = hisi_mem_get_pcc_chan_id(hdev); >+ if (ret) >+ goto free_hdev; >+ >+ ret = hisi_mem_register_pcc_channel(hdev); >+ if (ret) >+ goto free_hdev; >+ >+ platform_set_drvdata(pdev, hdev); >+ list_add_tail(&hdev->list, &mras->list); >+ hdev->verspec_data = &hisi04b2_verspec_data; >+ return 0; >+ >+free_hdev: >+ kfree(hdev); >+ return ret; >+} >+ >+static int hisi_mem_ras_remove(struct platform_device *pdev) { >+ struct hisi_mem_dev *hdev = platform_get_drvdata(pdev); >+ >+ platform_set_drvdata(pdev, NULL); >+ list_del(&hdev->list); >+ >+ return 0; >+} >+ >+static const struct acpi_device_id hisi_mem_acpi_match[] = { >+ { "HISI0FF1", 0}, >+ { }, >+}; >+MODULE_DEVICE_TABLE(acpi, hisi_mem_acpi_match); >+ >+static struct platform_driver hisi_mem_ras_driver = { >+ .probe = hisi_mem_ras_probe, >+ .remove = hisi_mem_ras_remove, >+ .driver = { >+ .name = "hisi_mem_ras", >+ .acpi_match_table = hisi_mem_acpi_match, >+ }, >+}; >+ >+static int __init hisi_mem_ras_init(void) { >+ int ret; >+ >+ mras = kzalloc(sizeof(struct hisi_mem_ras), GFP_KERNEL); >+ if (!mras) >+ return -ENOMEM; >+ >+ mras->acls_kobj = kobject_create_and_add("acls", kernel_kobj); >+ if (!mras->acls_kobj) { >+ kfree(mras); >+ return -ENOMEM; >+ } >+ >+ ret = sysfs_create_group(mras->acls_kobj, &acls_attr_group); >+ if (ret) { >+ mras->acls_kobj = NULL; >+ kobject_put(mras->acls_kobj); kfree(mras); missing here. >+ return ret; >+ } >+ >+ ret = platform_driver_register(&hisi_mem_ras_driver); >+ if (ret) { >+ sysfs_remove_group(mras->acls_kobj, &acls_attr_group); >+ kobject_put(mras->acls_kobj); >+ kfree(mras); >+ return ret; >+ } >+ >+ INIT_LIST_HEAD(&mras->list); >+ >+ return 0; >+} >+module_init(hisi_mem_ras_init); >+ >+static void __exit hisi_mem_ras_exit(void) { >+ platform_driver_unregister(&hisi_mem_ras_driver); >+ sysfs_remove_group(mras->acls_kobj, &acls_attr_group); >+ kobject_put(mras->acls_kobj); >+ kfree(mras); >+} >+module_exit(hisi_mem_ras_exit); >+ >+MODULE_LICENSE("GPL v2"); >+MODULE_AUTHOR("Xiaofei Tan <tanxiaofei(a)huawei.com>"); >+MODULE_DESCRIPTION("HISILICON Memory RAS driver"); >+MODULE_ALIAS("platform:" DRV_NAME); >diff --git a/drivers/soc/hisilicon/hisi_mem_ras.h >b/drivers/soc/hisilicon/hisi_mem_ras.h >new file mode 100644 >index 000000000000..a9acbb0c1012 >--- /dev/null >+++ b/drivers/soc/hisilicon/hisi_mem_ras.h >@@ -0,0 +1,84 @@ >+/* SPDX-License-Identifier: GPL-2.0 */ >+/* >+ * Copyright (C) Huawei Technologies Co., Ltd. 2023. All rights reserved. >+ */ >+ >+#ifndef _HISI_MEM_RAS_H >+#define _HISI_MEM_RAS_H >+ >+struct hisi_mem_ras { >+ struct list_head list; >+ struct kobject *acls_kobj; >+}; >+ >+struct hisi_mem_mbox_client_info { >+ struct mbox_client client; >+ struct mbox_chan *mbox_chan; >+ struct pcc_mbox_chan *pcc_chan; >+ u64 deadline_us; >+ void __iomem *pcc_comm_addr; >+ struct completion done; >+}; >+ >+struct hisi_mem_desc; >+struct hisi_mem_dev; >+ >+struct hisi_mem_verspecific_data { >+ void (*rx_callback)(struct mbox_client *cl, void *mssg); >+ int (*wait_cmd_complete)(struct hisi_mem_dev *hdev); >+ void (*fill_pcc_shared_mem)(struct hisi_mem_dev *hdev, >+ u8 cmd, struct hisi_mem_desc *desc, >+ void __iomem *comm_space, >+ u16 space_size); >+ u16 shared_mem_size; >+ bool has_txdone_irq; >+}; >+ >+struct hisi_mem_dev { >+ struct list_head list; >+ struct platform_device *pdev; >+ //struct acpi_device *acpi_dev; Not used. >+ const struct hisi_mem_verspecific_data *verspec_data; >+ /* device capabilities from firmware, like hisi_mem_CAPS_xxx. */ >+ u64 caps; >+ u8 chan_id; >+ u64 paddr_min; >+ u64 paddr_max; >+ struct mutex lock; >+ struct hisi_mem_mbox_client_info cl_info; }; >+ >+#define HISI_MEM_PCC_SHARE_MEM_BYTES 128 >+enum hisi_mem_ras_cmd_type { >+ HISI_MEM_RAS_ACLS = 0x4, >+ HISI_MEM_RAS_PPR = 0X5, >+}; >+ >+enum hisi_mem_ras_subcmd_type { >+ HISI_MEM_RAS_QUERY_CAP = 0x0, >+ HISI_MEM_RAS_DO_REPAIR = 0x1, >+}; >+ >+struct hisi_mem_req_desc { >+ u64 paddr; >+ u8 subcmd; >+ u8 rsv[3]; >+}; >+ >+#define HISI_MEM_ACLS_OK 0 >+#define HISI_MEM_ACLS_QUERY_NO_RES 1 >+#define HISI_MEM_ACLS_REPAIR_FAIL 2 >+#define HISI_MEM_ACLS_EINVAL 3 >+struct hisi_mem_rsp_desc { >+ u8 retStatus; /* 0: success, other: failure */ >+ u8 rsv[7]; >+}; >+ >+struct hisi_mem_desc { >+ union { >+ struct hisi_mem_req_desc req; >+ struct hisi_mem_rsp_desc rsp; >+ }; >+}; >+ >+#endif >-- >2.33.0

1 0

Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of the XQC address
by Shameerali Kolothum Thodi 13 May '24

13 May '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Monday, May 13, 2024 9:16 AM > To: Alex Williamson <alex.williamson(a)redhat.com>; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com> > Cc: jgg(a)nvidia.com; Jonathan Cameron <jonathan.cameron(a)huawei.com>; > kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of > the XQC address > > On 2024/5/9 22:29, Alex Williamson wrote: > > On Thu, 9 May 2024 09:37:51 +0000 > > Shameerali Kolothum Thodi <shameerali.kolothum.thodi(a)huawei.com> > wrote: > > > >>> -----Original Message----- > >>> From: Alex Williamson <alex.williamson(a)redhat.com> > >>> Sent: Wednesday, May 8, 2024 7:00 PM > >>> To: liulongfang <liulongfang(a)huawei.com> > >>> Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > >>> <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > >>> <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > >>> kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > >>> Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register > location of > >>> the XQC address > >> > >> [...] > >> > >>>> HiSilicon accelerator equipment can perform general services after > >>> completing live migration. > >>>> This kind of business is executed through the user mode driver and only > >>> needs to use SQE and CQE. > >>>> > >>>> At the same time, this device can also perform kernel-mode services in > the > >>> VM through the crypto > >>>> subsystem. This kind of service requires the use of EQE. > >>>> > >>>> Finally, if the device is abnormal, the driver needs to perform a device > >>> reset, and AEQE needs to > >>>> be used in this case. > >>>> > >>>> Therefore, a complete device live migration function needs to ensure > that > >>> device functions are > >>>> normal in all these scenarios. > >>>> Therefore, this data still needs to be migrated. > >>> > >>> Ok, I had jumped to an in-kernel host driver in reference to "kernel > >>> mode" rather than a guest kernel. Migrating with bad data only affects > >>> the current configuration of the device, reloading a guest driver to > >>> update these registers or a reset of the device would allow proper > >>> operation of the device, correct? > >> > >> Yes, after talking to Longfang, the device RAS will trigger a reset and > >> would function after reset. > >> > >>> > >>> But I think this still isn't really a complete solution, we know > >>> there's a bug in the migration data stream, so not only would we fix > >>> the data stream, but I think we should also take measures to prevent > >>> loading a known bad data stream. AIUI migration of this device while > >>> running in kernel mode (ie. a kernel driver within a guest VM) is > >>> broken. Therefore, the least we can do in a new kernel, knowing that > >>> there was previously a bug in the migration data stream, is to fail to > >>> load that migration data because it risks this scenario where the > >>> device is broken after migration. Shouldn't we then also increment a > >>> migration version field in the data stream to block migrations that > >>> risk this breakage, or barring that, change the magic data field to > >>> prevent the migration? Thanks, > >> > >> Ok. We could add a new ACC_DEV_MAGIC_V2 and prevent the migration > >> in vf_qm_check_match(). The only concern here is that, it will completely > >> block old kernel to new kernel migration and since we can recover the > >> device after the reset whether it is too restrictive or not. > > > > What's the impact to the running driver, kernel or userspace, if the > > device is reset? Migration is intended to be effectively transparent > > If the device is reset, the user's task needs to be restarted. > If an exception has been detected, the best way is not to migrate. > > > to the driver. If the driver stalls and needs to reset the device, > > what has the migration driver accomplished versus an offline migration? > > > > If there's a way to detect from the migration data if the device is > > running in kernel mode or user mode then you could potentially accept > > and send v1 magic conditional that the device is in user mode and > > require v2 magic for any migration where the device is in kernel mode. > > This all adds complication though and seems like it has corner cases > > where we might allow migration to an old kernel that might trap the > > device there if the use case changes. > > > > The driver does not support checking whether the device is running in > kernel mode or user mode. > Moreover, the device supports user-mode services and kernel-mode services > to run at the same time. > > > Essentially it comes down to what should the migration experience be > > and while restricting old->new and new->old migration is undesirable, > > it seems old->old migration is effectively already broken anyway. As > > you consider a v2 magic, perhaps consider how the migration data > > structure might be improved overall to better handle new features and > > bugs. Thanks, > > > > We discussed a plan: > Update ACC_DEV_MAGIC to ACC_DEV_MAGIC_VERSION and configure its > last byte > as version information: > > /* QM match information, last byte is version number */ > #define ACC_DEV_MAGIC_VERSION 0XACCDEVFEEDCAFE01 Oops..cant have V there. But the idea is replace magic with last byte as version info which can be used in future for handling bugs/features etc. Thanks, Shameer

1 0

Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of the XQC address
by Shameerali Kolothum Thodi 09 May '24

09 May '24

> -----Original Message----- > From: Alex Williamson <alex.williamson(a)redhat.com> > Sent: Wednesday, May 8, 2024 7:00 PM > To: liulongfang <liulongfang(a)huawei.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of > the XQC address [...] > > HiSilicon accelerator equipment can perform general services after > completing live migration. > > This kind of business is executed through the user mode driver and only > needs to use SQE and CQE. > > > > At the same time, this device can also perform kernel-mode services in the > VM through the crypto > > subsystem. This kind of service requires the use of EQE. > > > > Finally, if the device is abnormal, the driver needs to perform a device > reset, and AEQE needs to > > be used in this case. > > > > Therefore, a complete device live migration function needs to ensure that > device functions are > > normal in all these scenarios. > > Therefore, this data still needs to be migrated. > > Ok, I had jumped to an in-kernel host driver in reference to "kernel > mode" rather than a guest kernel. Migrating with bad data only affects > the current configuration of the device, reloading a guest driver to > update these registers or a reset of the device would allow proper > operation of the device, correct? Yes, after talking to Longfang, the device RAS will trigger a reset and would function after reset. > > But I think this still isn't really a complete solution, we know > there's a bug in the migration data stream, so not only would we fix > the data stream, but I think we should also take measures to prevent > loading a known bad data stream. AIUI migration of this device while > running in kernel mode (ie. a kernel driver within a guest VM) is > broken. Therefore, the least we can do in a new kernel, knowing that > there was previously a bug in the migration data stream, is to fail to > load that migration data because it risks this scenario where the > device is broken after migration. Shouldn't we then also increment a > migration version field in the data stream to block migrations that > risk this breakage, or barring that, change the magic data field to > prevent the migration? Thanks, Ok. We could add a new ACC_DEV_MAGIC_V2 and prevent the migration in vf_qm_check_match(). The only concern here is that, it will completely block old kernel to new kernel migration and since we can recover the device after the reset whether it is too restrictive or not. Thanks, Shameer

1 0

Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of the XQC address
by Shameerali Kolothum Thodi 07 May '24

07 May '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Tuesday, May 7, 2024 9:29 AM > To: Alex Williamson <alex.williamson(a)redhat.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 2/5] hisi_acc_vfio_pci: modify the register location of > the XQC address > > On 2024/5/4 0:11, Alex Williamson wrote: > > On Thu, 25 Apr 2024 21:23:19 +0800 > > Longfang Liu <liulongfang(a)huawei.com> wrote: > > > >> According to the latest hardware register specification. The DMA > >> addresses of EQE and AEQE are not at the front of their respective > >> register groups, but start from the second. > >> So, previously fetching the value starting from the first register > >> would result in an incorrect address. > >> > >> Therefore, the register location from which the address is obtained > >> needs to be modified. > > > > How does this affect migration? Has it ever worked? Does this make > > The general HiSilicon accelerator task will only use SQE and CQE. > EQE is only used when user running kernel mode task and uses interrupt mode. > AEQE is only used when user running task exceptions occur and software reset > is required. > > The DMA addresses of these four queues are written to the device by the device > driver through the mailbox command during driver initialization. > The DMA addresses of EQE and AEQE are migrated through the device register. > > EQE and AEQE are not used in general task, after the live migration is > completed, > this DMA address error will not be found. until we added a new kernel-mode test > case > that we discovered that this address was abnormal. > > > the migration data incompatible? > > > > This address only affects the kernel mode interrupt mode task function and > device > exception recovery function. > They do not affect live migration functionality > > > Fixes: ??? > > OK! Hi, Could you please add the Fixes tag and resend this separately if there are no outstanding comments on this. This is not related to the debugfs support anyway. Thanks, Shameer > Thanks. > Longfang. > >> Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > >> --- > >> drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 8 ++++---- > >> drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 3 +++ > >> 2 files changed, 7 insertions(+), 4 deletions(-) > >> > >> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > >> index 45351be8e270..0c7e31076ff4 100644 > >> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > >> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > >> @@ -516,12 +516,12 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > >> return -EINVAL; > >> > >> /* Every reg is 32 bit, the dma address is 64 bit. */ > >> - vf_data->eqe_dma = vf_data->qm_eqc_dw[1]; > >> + vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; > >> vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > >> - vf_data->eqe_dma |= vf_data->qm_eqc_dw[0]; > >> - vf_data->aeqe_dma = vf_data->qm_aeqc_dw[1]; > >> + vf_data->eqe_dma |= vf_data->qm_eqc_dw[QM_XQC_ADDR_LOW]; > >> + vf_data->aeqe_dma = vf_data->qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > >> vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > >> - vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[0]; > >> + vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[QM_XQC_ADDR_LOW]; > >> > >> /* Through SQC_BT/CQC_BT to get sqc and cqc address */ > >> ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > >> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > >> index 5bab46602fad..f887ab98581c 100644 > >> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > >> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > >> @@ -38,6 +38,9 @@ > >> #define QM_REG_ADDR_OFFSET 0x0004 > >> > >> #define QM_XQC_ADDR_OFFSET 32U > >> +#define QM_XQC_ADDR_LOW 0x1 > >> +#define QM_XQC_ADDR_HIGH 0x2 > >> + > >> #define QM_VF_AEQ_INT_MASK 0x0004 > >> #define QM_VF_EQ_INT_MASK 0x000c > >> #define QM_IFC_INT_SOURCE_V 0x0020 > > > > . > >

1 0

Re: [PATCH v6 4/5] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 07 May '24

07 May '24

> -----Original Message----- > From: Alex Williamson <alex.williamson(a)redhat.com> > Sent: Friday, May 3, 2024 6:21 PM > To: liulongfang <liulongfang(a)huawei.com> > Cc: jgg(a)nvidia.com; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com>; kvm(a)vger.kernel.org; linux- > kernel(a)vger.kernel.org; linuxarm(a)openeuler.org > Subject: Re: [PATCH v6 4/5] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > On Thu, 25 Apr 2024 21:23:21 +0800 > Longfang Liu <liulongfang(a)huawei.com> wrote: > > > On the debugfs framework of VFIO, if the CONFIG_VFIO_DEBUGFS macro is > > enabled, the debug function is registered for the live migration driver > > of the HiSilicon accelerator device. > > > > After registering the HiSilicon accelerator device on the debugfs > > framework of live migration of vfio, a directory file "hisi_acc" > > of debugfs is created, and then three debug function files are > > created in this directory: > > > > vfio > > | > > +---<dev_name1> > > | +---migration > > | +--state > > | +--hisi_acc > > | +--dev_data > > | +--migf_data > > | +--cmd_state > > | > > +---<dev_name2> > > +---migration > > +--state > > +--hisi_acc > > +--dev_data > > +--migf_data > > +--cmd_state > > > > dev_data file: read device data that needs to be migrated from the > > current device in real time > > migf_data file: read the migration data of the last live migration > > from the current driver. > > cmd_state: used to get the cmd channel state for the device. > > > > +----------------+ +--------------+ +---------------+ > > | migration dev | | src dev | | dst dev | > > +-------+--------+ +------+-------+ +-------+-------+ > > | | | > > | +------v-------+ +-------v-------+ > > | | saving_mif | | resuming_migf | > > read | | file | | file | > > | +------+-------+ +-------+-------+ > > | | copy | > > | +------------+----------+ > > | | > > +-------v---------+ +-------v--------+ > > | data buffer | | debug_migf | > > +-------+---------+ +-------+--------+ > > | | > > cat | cat | > > +-------v--------+ +-------v--------+ > > | dev_data | | migf_data | > > +----------------+ +----------------+ > > > > When accessing debugfs, user can obtain the real-time status data > > of the device through the "dev_data" file. It will directly read > > the device status data and will not affect the live migration > > function. Its data is stored in the allocated memory buffer, > > and the memory is released after data returning to user mode. > > > > To obtain the data of the last complete migration, user need to > > obtain it through the "migf_data" file. Since the live migration > > data only exists during the migration process, it is destroyed > > after the migration is completed. > > In order to save this data, a debug_migf file is created in the > > driver. At the end of the live migration process, copy the data > > to debug_migf. > > > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > > --- > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 225 ++++++++++++++++++ > > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + > > 2 files changed, 232 insertions(+) > > > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > index bf358ba94b5d..656b3d975940 100644 > > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > > @@ -627,15 +627,33 @@ static void hisi_acc_vf_disable_fd(struct > hisi_acc_vf_migration_file *migf) > > mutex_unlock(&migf->lock); > > } > > > > +static void hisi_acc_debug_migf_copy(struct hisi_acc_vf_core_device > *hisi_acc_vdev, > > + struct hisi_acc_vf_migration_file *src_migf) > > +{ > > + struct hisi_acc_vf_migration_file *dst_migf = hisi_acc_vdev- > >debug_migf; > > + > > + if (!dst_migf) > > + return; > > + > > + mutex_lock(&hisi_acc_vdev->enable_mutex); > > + dst_migf->disabled = src_migf->disabled; > > + dst_migf->total_length = src_migf->total_length; > > + memcpy(&dst_migf->vf_data, &src_migf->vf_data, > > + sizeof(struct acc_vf_data)); > > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > > +} > > + > > static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > > { > > if (hisi_acc_vdev->resuming_migf) { > > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >resuming_migf); > > hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf); > > fput(hisi_acc_vdev->resuming_migf->filp); > > hisi_acc_vdev->resuming_migf = NULL; > > } > > > > if (hisi_acc_vdev->saving_migf) { > > + hisi_acc_debug_migf_copy(hisi_acc_vdev, hisi_acc_vdev- > >saving_migf); > > hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf); > > fput(hisi_acc_vdev->saving_migf->filp); > > hisi_acc_vdev->saving_migf = NULL; > > @@ -1144,6 +1162,7 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > > if (!vf_qm->io_base) > > return -EIO; > > > > + mutex_init(&hisi_acc_vdev->enable_mutex); > > vf_qm->fun_type = QM_HW_VF; > > vf_qm->pdev = vf_dev; > > mutex_init(&vf_qm->mailbox_lock); > > @@ -1294,6 +1313,203 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > > return vfio_pci_core_ioctl(core_vdev, cmd, arg); > > } > > > > +static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device > *vdev) > > +{ > > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + struct device *dev = vdev->dev; > > + int ret; > > + > > + if (!vdev->mig_ops) { > > + dev_err(dev, "device does not support live migration!\n"); > > Sorry, every error path should not spam dmesg with dev_err(). I'm > going to wait until your co-maintainer approves this before looking at > any further iterations of this series. Thanks, Sure. I will sync up with Longfang and also make sure we address all the existing comments on this before posting the next revision. Thanks, Shameer

1 0

Re: [PATCH v3 2/4] hisi_acc_vfio_pci: Create subfunction for data reading
by Shameerali Kolothum Thodi 07 Mar '24

07 Mar '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Thursday, March 7, 2024 6:04 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v3 2/4] hisi_acc_vfio_pci: Create subfunction for data > reading > > During the live migration process. It needs to obtain various status > data of drivers and devices. In order to facilitate calling it in the > debugfs function. For all operations that read data from device registers, > the driver creates a subfunction. > Also fixed the location of address data. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 55 ++++++++++--------- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 3 + > 2 files changed, 33 insertions(+), 25 deletions(-) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 45351be8e270..1881f3fa9266 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -486,42 +486,22 @@ static int vf_qm_load_data(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return 0; > } > > -static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, > - struct hisi_acc_vf_migration_file *migf) > +static int vf_qm_read_data(struct hisi_qm *vf_qm, struct acc_vf_data > *vf_data) > { > - struct acc_vf_data *vf_data = &migf->vf_data; > - struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > struct device *dev = &vf_qm->pdev->dev; > int ret; > > - if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > - /* Update state and return with match data */ > - vf_data->vf_qm_state = QM_NOT_READY; > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > - migf->total_length = QM_MATCH_SIZE; > - return 0; > - } > - > - vf_data->vf_qm_state = QM_READY; > - hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > - > - ret = vf_qm_cache_wb(vf_qm); > - if (ret) { > - dev_err(dev, "failed to writeback QM Cache!\n"); > - return ret; > - } > - > ret = qm_get_regs(vf_qm, vf_data); So this doesn't need the qm_wait_dev_not_ready(vf_qm) check above for the debugfs ? What happens if you read when device is not ready? > if (ret) > return -EINVAL; > > /* Every reg is 32 bit, the dma address is 64 bit. */ > - vf_data->eqe_dma = vf_data->qm_eqc_dw[1]; > + vf_data->eqe_dma = vf_data->qm_eqc_dw[QM_XQC_ADDR_HIGH]; Also since there is no serialization with core migration now, what will be the data returned in debugfs when there is a vf_qm_load_data() is in progress? So I guess the intention or assumption here is that the debugfs data is only valid when the user knows that devices not under migration process. Is that right? Thanks, Shameer > vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->eqe_dma |= vf_data->qm_eqc_dw[0]; > - vf_data->aeqe_dma = vf_data->qm_aeqc_dw[1]; > + vf_data->eqe_dma |= vf_data->qm_eqc_dw[QM_XQC_ADDR_LOW]; > + vf_data->aeqe_dma = vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_HIGH]; > vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET; > - vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[0]; > + vf_data->aeqe_dma |= vf_data- > >qm_aeqc_dw[QM_XQC_ADDR_LOW]; > > /* Through SQC_BT/CQC_BT to get sqc and cqc address */ > ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma); > @@ -536,6 +516,31 @@ static int vf_qm_state_save(struct > hisi_acc_vf_core_device *hisi_acc_vdev, > return -EINVAL; > } > > + return 0; > +} > + > +static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, > + struct hisi_acc_vf_migration_file *migf) > +{ > + struct acc_vf_data *vf_data = &migf->vf_data; > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + int ret; > + > + if (unlikely(qm_wait_dev_not_ready(vf_qm))) { > + /* Update state and return with match data */ > + vf_data->vf_qm_state = QM_NOT_READY; > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + migf->total_length = QM_MATCH_SIZE; > + return 0; > + } > + > + vf_data->vf_qm_state = QM_READY; > + hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state; > + > + ret = vf_qm_read_data(vf_qm, vf_data); > + if (ret) > + return -EINVAL; > + > migf->total_length = sizeof(struct acc_vf_data); > return 0; > } > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index 5bab46602fad..7a9dc87627cd 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -38,6 +38,9 @@ > #define QM_REG_ADDR_OFFSET 0x0004 > > #define QM_XQC_ADDR_OFFSET 32U > +#define QM_XQC_ADDR_LOW 0x1 > +#define QM_XQC_ADDR_HIGH 0x2 > + > #define QM_VF_AEQ_INT_MASK 0x0004 > #define QM_VF_EQ_INT_MASK 0x000c > #define QM_IFC_INT_SOURCE_V 0x0020 > -- > 2.24.0

1 0

Re: [PATCH v2 2/3] hisi_acc_vfio_pci: register debugfs for hisilicon migration driver
by Shameerali Kolothum Thodi 05 Feb '24

05 Feb '24

> -----Original Message----- > From: liulongfang <liulongfang(a)huawei.com> > Sent: Sunday, February 4, 2024 8:56 AM > To: alex.williamson(a)redhat.com; jgg(a)nvidia.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; Jonathan Cameron > <jonathan.cameron(a)huawei.com> > Cc: kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; > linuxarm(a)openeuler.org; liulongfang <liulongfang(a)huawei.com> > Subject: [PATCH v2 2/3] hisi_acc_vfio_pci: register debugfs for hisilicon > migration driver > > On the debugfs framework of VFIO, if the CONFIG_VFIO_DEBUGFS macro is > enabled, the debug function is registered for the live migration driver > of the HiSilicon accelerator device. > > After registering the HiSilicon accelerator device on the debugfs > framework of live migration of vfio, a directory file "hisi_acc" > of debugfs is created, and then three debug function files are > created in this directory: > > vfio > | > +---<dev_name1> > | +---migration > | +--state > | +--hisi_acc > | +--attr > | +--data > | +--save > | +--cmd_state > | > +---<dev_name2> > +---migration > +--state > +--hisi_acc > +--attr > +--data > +--save > +--cmd_state > > data file: used to get the migration data from the driver > attr file: used to get device attributes parameters from the driver > save file: used to read the data of the live migration device and save > it to the driver. > cmd_state: used to get the cmd channel state for the device. > > Signed-off-by: Longfang Liu <liulongfang(a)huawei.com> > --- > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 197 ++++++++++++++++++ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 11 + > 2 files changed, 208 insertions(+) > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > index 00a22a990eb8..e51bbb41c2b3 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c > @@ -15,6 +15,7 @@ > #include <linux/anon_inodes.h> > > #include "hisi_acc_vfio_pci.h" > +#include "../../vfio.h" > > /* Return 0 on VM acc device ready, -ETIMEDOUT hardware timeout */ > static int qm_wait_dev_not_ready(struct hisi_qm *qm) > @@ -606,6 +607,18 @@ hisi_acc_check_int_state(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > } > } > > +static void hisi_acc_vf_migf_save(struct hisi_acc_vf_migration_file > *dst_migf, Can we please rename this to indicate the debug, similar to other debugfs support functions here, something like hisi_acc_vf_debug_migf_save()? > + struct hisi_acc_vf_migration_file *src_migf) > +{ > + if (!dst_migf) > + return; > + > + dst_migf->disabled = false; Is this set to true anywhere for debugfs migf ? > + dst_migf->total_length = src_migf->total_length; > + memcpy(&dst_migf->vf_data, &src_migf->vf_data, > + sizeof(struct acc_vf_data)); > +} > + > static void hisi_acc_vf_disable_fd(struct hisi_acc_vf_migration_file *migf) > { > mutex_lock(&migf->lock); > @@ -618,12 +631,16 @@ static void hisi_acc_vf_disable_fd(struct > hisi_acc_vf_migration_file *migf) > static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > { > if (hisi_acc_vdev->resuming_migf) { > + hisi_acc_vf_migf_save(hisi_acc_vdev->debug_migf, > + hisi_acc_vdev- > >resuming_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf); > fput(hisi_acc_vdev->resuming_migf->filp); > hisi_acc_vdev->resuming_migf = NULL; > } > > if (hisi_acc_vdev->saving_migf) { > + hisi_acc_vf_migf_save(hisi_acc_vdev->debug_migf, > + hisi_acc_vdev->saving_migf); > hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf); > fput(hisi_acc_vdev->saving_migf->filp); > hisi_acc_vdev->saving_migf = NULL; > @@ -1156,6 +1173,7 @@ static int hisi_acc_vf_qm_init(struct > hisi_acc_vf_core_device *hisi_acc_vdev) > if (!vf_qm->io_base) > return -EIO; > > + mutex_init(&hisi_acc_vdev->enable_mutex); > vf_qm->fun_type = QM_HW_VF; > vf_qm->pdev = vf_dev; > mutex_init(&vf_qm->mailbox_lock); > @@ -1306,6 +1324,176 @@ static long hisi_acc_vfio_pci_ioctl(struct > vfio_device *core_vdev, unsigned int > return vfio_pci_core_ioctl(core_vdev, cmd, arg); > } > > +static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device > *vdev) > +{ > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev->debug_migf; > + > + if (!vdev->mig_ops || !migf) { > + seq_printf(seq, "%s\n", "device does not support live > migration!"); > + return -EINVAL; > + } > + > + /** > + * When the device is not opened, the io_base is not mapped. > + * The driver cannot perform device read and write operations. > + */ > + if (atomic_read(&hisi_acc_vdev->dev_opened) != DEV_OPEN) { > + seq_printf(seq, "%s\n", "device not opened!"); > + return -EINVAL; > + } > + > + return 0; > +} > + > +static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > + u64 value; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->enable_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + return 0; > + } > + > + ret = qm_wait_dev_not_ready(vf_qm); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s\n", "VF device not ready!"); > + return 0; > + } > + > + value = readl(vf_qm->io_base + QM_MB_CMD_SEND_BASE); > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s:0x%llx\n", "mailbox cmd channel state is OK", > value); > + > + return 0; > +} > + > +static int hisi_acc_vf_debug_save(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev->debug_migf; > + int ret; > + > + mutex_lock(&hisi_acc_vdev->enable_mutex); > + ret = hisi_acc_vf_debug_check(seq, vdev); > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + return 0; > + } > + > + ret = vf_qm_state_save(hisi_acc_vdev, migf); Are you sure this will not interfere with the core migration APIs as we are not holding the state_mutex?. I think Alex also raised a similar concern previously. Applies to other debugfs interfaces here as well. Please check. > + if (ret) { > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s\n", "failed to save device data!"); > + return 0; > + } > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > + seq_printf(seq, "%s\n", "successful to save device data!"); > + > + return 0; > +} > + > +static int hisi_acc_vf_data_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev- > >debug_migf; > + size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); > + > + if (debug_migf && debug_migf->total_length) > + seq_hex_dump(seq, "Mig Data:", DUMP_PREFIX_OFFSET, 16, > 1, > + (unsigned char *)&debug_migf->vf_data, > + vf_data_sz, false); > + else > + seq_printf(seq, "%s\n", "device not migrated!"); > + > + return 0; > +} > + > +static int hisi_acc_vf_attr_read(struct seq_file *seq, void *data) > +{ > + struct device *vf_dev = seq->private; > + struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); > + struct vfio_device *vdev = &core_device->vdev; > + struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(vdev); > + struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev- > >debug_migf; > + > + if (debug_migf && debug_migf->total_length) { > + seq_printf(seq, > + "acc device:\n" > + "device state: %d\n" > + "device ready: %u\n" > + "data valid: %d\n" > + "data size: %lu\n", > + hisi_acc_vdev->mig_state, > + hisi_acc_vdev->vf_qm_state, > + debug_migf->disabled, > + debug_migf->total_length); > + } else { > + seq_printf(seq, "%s\n", "device not migrated!"); > + } > + > + return 0; > +} > + > +static int hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; > + struct dentry *vfio_dev_migration = NULL; > + struct dentry *vfio_hisi_acc = NULL; > + struct device *dev = vdev->dev; > + void *migf = NULL; > + > + if (!debugfs_initialized()) > + return 0; > + > + migf = kzalloc(sizeof(struct hisi_acc_vf_migration_file), GFP_KERNEL); > + if (!migf) > + return -ENOMEM; > + hisi_acc_vdev->debug_migf = migf; > + > + vfio_dev_migration = debugfs_lookup("migration", vdev- > >debug_root); > + if (!vfio_dev_migration) { > + kfree(migf); > + dev_err(dev, "failed to lookup migration debugfs file!\n"); > + return -ENODEV; > + } > + > + vfio_hisi_acc = debugfs_create_dir("hisi_acc", vfio_dev_migration); > + debugfs_create_devm_seqfile(dev, "data", vfio_hisi_acc, > + hisi_acc_vf_data_read); > + debugfs_create_devm_seqfile(dev, "attr", vfio_hisi_acc, > + hisi_acc_vf_attr_read); > + debugfs_create_devm_seqfile(dev, "cmd_state", vfio_hisi_acc, > + hisi_acc_vf_debug_cmd); > + debugfs_create_devm_seqfile(dev, "save", vfio_hisi_acc, > + hisi_acc_vf_debug_save); > + > + return 0; > +} > + > +static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device > *hisi_acc_vdev) > +{ > + if (!debugfs_initialized()) > + return; > + > + kfree(hisi_acc_vdev->debug_migf); > +} > + > static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev) > { > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > @@ -1323,9 +1511,11 @@ static int hisi_acc_vfio_pci_open_device(struct > vfio_device *core_vdev) > return ret; > } > hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING; > + atomic_set(&hisi_acc_vdev->dev_opened, DEV_OPEN); > } > > vfio_pci_core_finish_enable(vdev); > + > return 0; > } > > @@ -1334,7 +1524,10 @@ static void hisi_acc_vfio_pci_close_device(struct > vfio_device *core_vdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_get_vf_dev(core_vdev); > struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; > > + atomic_set(&hisi_acc_vdev->dev_opened, DEV_CLOSE); > + mutex_lock(&hisi_acc_vdev->enable_mutex); > iounmap(vf_qm->io_base); > + mutex_unlock(&hisi_acc_vdev->enable_mutex); > vfio_pci_core_close_device(core_vdev); > } > > @@ -1425,6 +1618,9 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev > *pdev, const struct pci_device > ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); > if (ret) > goto out_put_vdev; > + > + if (ops == &hisi_acc_vfio_pci_migrn_ops) > + hisi_acc_vfio_debug_init(hisi_acc_vdev); > return 0; > > out_put_vdev: > @@ -1437,6 +1633,7 @@ static void hisi_acc_vfio_pci_remove(struct pci_dev > *pdev) > struct hisi_acc_vf_core_device *hisi_acc_vdev = > hisi_acc_drvdata(pdev); > > vfio_pci_core_unregister_device(&hisi_acc_vdev->core_device); > + hisi_acc_vf_debugfs_exit(hisi_acc_vdev); > vfio_put_device(&hisi_acc_vdev->core_device.vdev); > } > > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > index dcabfeec6ca1..283bd8acdc42 100644 > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h > @@ -49,6 +49,11 @@ > #define QM_EQC_DW0 0X8000 > #define QM_AEQC_DW0 0X8020 > > +enum acc_dev_state { > + DEV_CLOSE = 0x0, > + DEV_OPEN, > +}; Do we really need this enum as you can directly use 0 or 1? Thanks, Shameer > + > struct acc_vf_data { > #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) > /* QM match information */ > @@ -113,5 +118,11 @@ struct hisi_acc_vf_core_device { > spinlock_t reset_lock; > struct hisi_acc_vf_migration_file *resuming_migf; > struct hisi_acc_vf_migration_file *saving_migf; > + > + /* To make sure the device is enabled */ > + struct mutex enable_mutex; > + atomic_t dev_opened; > + /* For debugfs */ > + struct hisi_acc_vf_migration_file *debug_migf; > }; > #endif /* HISI_ACC_VFIO_PCI_H */ > -- > 2.24.0

1 0

[PATCH 0/3] fix return value type issue and some cleanup
by Xiaofei Tan 11 May '23

11 May '23

Fix return value type issue and some cleanup. Xiaofei Tan (3): rasdaemon: fix return value type issue of read/write function from unistd.h rasdaemon: fix issue of signed and unsigned integer comparison rasdaemon: remove redundant header file and do some cleaup non-standard-hisi_hip08.c | 2 +- non-standard-hisilicon.c | 4 ++-- ras-diskerror-handler.c | 2 +- ras-events.c | 45 ++++++++++++++++++++++++------------ ras-memory-failure-handler.c | 7 +++--- 5 files changed, 37 insertions(+), 23 deletions(-) -- 2.30.0

1 3

[PATCH v7 0/5] add debugfs to migration driver
by Longfang Liu 02 Feb '23

02 Feb '23

Add a debugfs function to the migration driver in VFIO to provide a step-by-step test function for the migration driver. When the execution of live migration fails, the user can view the status and data during the migration process separately from the source and the destination, which is convenient for users to analyze and locate problems. Changes v6 -> v7 Fix some code style issues. Changes v5 -> v6 Control the creation of debugfs through the CONFIG_DEBUG_FS. Changes v4 -> v5 Remove the newly added vfio_migration_ops and use seq_printf to optimize the implementation of debugfs. Changes v3 -> v4 Change the migration_debug_operate interface to debug_root file. Changes v2 -> v3 Extend the debugfs function from hisilicon device to vfio. Changes v1 -> v2 Change the registration method of root_debugfs to register with module initialization. Longfang Liu (5): vfio/migration: Add debugfs to live migration driver hisi_acc_vfio_pci: extract public functions for container_of hisi_acc_vfio_pci: register debugfs for hisilicon migration driver Documentation: add debugfs description for hisi_acc_vfio_pci vfio: update live migration device status .../ABI/testing/debugfs-hisi-migration | 32 ++ drivers/vfio/pci/Makefile | 1 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 306 +++++++++++++++++- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 11 + drivers/vfio/pci/mlx5/main.c | 3 + drivers/vfio/pci/vfio_pci_core.c | 3 + drivers/vfio/pci/vfio_pci_debugfs.c | 80 +++++ drivers/vfio/pci/vfio_pci_debugfs.h | 28 ++ include/linux/vfio.h | 8 + 9 files changed, 462 insertions(+), 10 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.c create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.h -- 2.24.0

3 14

Liquidación de 3.000 ordenadores portátiles Lenovo Yoga retirados de la administración
by PcRenting / LaCasaAzul 01 Feb '23

01 Feb '23

Liquidacin de ordenadores retirados de la administracin Buenos das: Nuestra empresa ha retirado en varios lotes ms de 3.000 PORTTILES LENOVO YOGA DE LA ADMINISTRACIN, y nos han encargado liquidarlos por un importe de slo 110 euros la unidad. Estos equipos se han revisado, formateado, y se les ha instalado Windows 10 pro con su licencia por tiempo indefinido. Se trata de porttiles perfectos para trabajos de ofimtica, administrativos, teletrabajo, teleformacin o navegar por internet. Si quiere puede pedirnos informacin detallada contestando al correo alberto(a)almeriaocasion.es [1] En espera de que esta informacin haya sido de su inters, reciba un cordial saludo. Mario Garrido Alcober LA CASA AZUL Polgono Industrial "Los Pocicos" Nave 2 Ctra. N 332 Km 11 Cuevas del Almanzora, 04610 Almera. Tlf: 950 455 892 alberto(a)almeriaocasion.es [2] 683 583 017 Links: ------ [1] mailto:alberto@almeriaocasion.es [2] mailto:alberto@almeriaocasion.es

1 0

Faktoring
by "Szczepan Kiełbasa" 01 Feb '23

01 Feb '23

Dzień dobry, rozważali Państwo wybór finansowania, które spełni potrzeby firmy, zapewniając natychmiastowy dostęp do gotówki, bez zbędnych przestojów? Przygotowaliśmy rozwiązania faktoringowe dopasowane do Państwa branży i wielkości firmy, dzięki którym, nie muszą Państwo martwić się o niewypłacalność kontrahentów, ponieważ transakcje są zabezpieczone i posiadają gwarancję spłaty. Chcą Państwo przeanalizować dostępne opcje? Pozdrawiam Szczepan Kiełbasa

1 0

You have 10 new message(s) in your linuxarm@openeuler.org Mailbox Server
by openeuler.org Email Support 31 Jan '23

31 Jan '23

1 0

DOCUMENTS SENT VIA ONEDRIVE
by linuxarm 30 Jan '23

30 Jan '23

1 0

Liquidación de 3.000 ordenadores portátiles Lenovo Yoga retirados de la administración
by PcRenting / LaCasaAzul 25 Jan '23

25 Jan '23

Liquidacin de ordenadores retirados de la administracin Buenos das: Nuestra empresa ha retirado en varios lotes ms de 3.000 PORTTILES LENOVO YOGA DE LA ADMINISTRACIN, y nos han encargado liquidarlos por un importe de slo 110 euros la unidad. Estos equipos se han revisado, formateado, y se les ha instalado Windows 10 pro con su licencia por tiempo indefinido. Se trata de porttiles perfectos para trabajos de ofimtica, administrativos, teletrabajo, teleformacin o navegar por internet. Si quiere puede pedirnos informacin detallada contestando al correo alberto(a)almeriaocasion.es [1] En espera de que esta informacin haya sido de su inters, reciba un cordial saludo. Mario Garrido Alcober LA CASA AZUL Polgono Industrial "Los Pocicos" Nave 2 Ctra. N 332 Km 11 Cuevas del Almanzora, 04610 Almera. Tlf: 950 455 892 alberto(a)almeriaocasion.es [2] 683 583 017 Links: ------ [1] mailto:alberto@almeriaocasion.es [2] mailto:alberto@almeriaocasion.es

1 0

Prezentacja
by Karol Michun 23 Jan '23

23 Jan '23

Dzień dobry! Czy mógłbym przedstawić rozwiązanie, które umożliwia monitoring każdego auta w czasie rzeczywistym w tym jego pozycję, zużycie paliwa i przebieg? Dodatkowo nasze narzędzie minimalizuje koszty utrzymania samochodów, skraca czas przejazdów, a także tworzenie planu tras czy dostaw. Z naszej wiedzy i doświadczenia korzysta już ponad 49 tys. Klientów. Monitorujemy 809 000 pojazdów na całym świecie, co jest naszą najlepszą wizytówką. Bardzo proszę o e-maila zwrotnego, jeśli moglibyśmy wspólnie omówić potencjał wykorzystania takiego rozwiązania w Państwa firmie. Pozdrawiam Karol Michun

1 0

-+- information about email marketing services -+-
by ISA 23 Jan '23

23 Jan '23

1 0

Re: Invitation to Project Monitoring and Evaluation with Data Management and Analysis Workshop Feb 2023 Mombasa Kenya
by FOSCORE DEVELOPMENT CENTER 21 Jan '23

21 Jan '23

<http://email.foscoredevelopmentcenter.org/tracking/click?d=g-KElsU129Rj4xD_…> Invitation to Project Monitoring and Evaluation with Data Management and Analysis Workshop Feb 2023 Mombasa Kenya 13/02/2023 To 24/02/202310 DaysMombasa, Kenya <http://email.foscoredevelopmentcenter.org/tracking/click?d=8IhOZ8-rWl5CIm-m…> Register As Individual <http://email.foscoredevelopmentcenter.org/tracking/click?d=KaUdAKEpcCbGkpy_…> Register For Online Training <http://email.foscoredevelopmentcenter.org/tracking/click?d=4MogCdbIULBOrmNj…> Register As A Group · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.foscoredevelopmentcenter.org/tracking/click?d=l6zfFP-hckb-RlvQ…> WhatsApp our consultant or <http://email.foscoredevelopmentcenter.org/tracking/click?d=onnMgZrG2lzEYMT-…> Fill form here · <http://email.foscoredevelopmentcenter.org/tracking/click?d=nL2O3Cv7wz_EvWea…> Online Training Course · <http://email.foscoredevelopmentcenter.org/tracking/click?d=onnMgZrG2lzEYMT-…> Professional Short Training Courses · <http://email.foscoredevelopmentcenter.org/tracking/click?d=TGc6SPNa6wsf0a20…> Download our Admission form · <http://email.foscoredevelopmentcenter.org/tracking/click?d=w_VfFLiyHQlY_GhW…> Download pre-training form to fill and share with us · <http://email.foscoredevelopmentcenter.org/tracking/click?d=Igc_mk9-YF1Xi-sK…> Download our Calender 2022 · FOSCORE DEVELOPMENT CENTER <http://email.foscoredevelopmentcenter.org/tracking/click?d=9i8NH0YxugT77q2F…> www.fdc-k.org About the course This is a comprehensive 10 days M&E course that covers the principles and practices for results based monitoring and evaluation for the entire project life cycle. This course equips participants with skills in setting up and implementing results based monitoring and evaluation systems including M&E data management, analysis and reporting. The participants will benefit from the latest M&E thinking and practices including the results and participatory approaches. This course is designed to enable the participants become experts in monitoring and evaluating their development projects. The course covers all the key elements of a robust M&E system coupled with a practical project to illustrate the M&E concepts. Target Participants This course is designed for researchers, project staff, development practitioners, managers and decision makers who are responsible for project, program or organization-level M&E. The course aims to enhance the skills of professionals who need to research, supervise, manage, plan, implement, monitor and evaluate development projects. Course duration 10 days Course objectives Develop project results levels Design a project using logical framework Develop indicators and targets for each result level Track performance indicators over the life of the project Evaluation a project against the set results Develop and implement M&E systems Develop a comprehensive monitoring and evaluation plan Use data analysis software (Stata/SPSS/R) Collect data using mobile data collection tools Carryout impact evaluation Use GIS to analyze and share project data Course Outline Introduction to Results Based Project Management Fundamentals of Results Based Management Why is RBM important? Results based management vs traditional projects management RBM Lifecycle (seven phases) Areas of focus of RBM Fundamentals of Monitoring and Evaluation Definition of Monitoring and Evaluation Why Monitoring and Evaluation is important Key principles and concepts in M&E M&E in project lifecycle Participatory M&E Project Analysis Situation Analysis Needs Assessment Strategy Analysis Design of Results in Monitoring and Evaluation Results chain approaches: Impact, outcomes, outputs and activities Results framework M&E causal pathway Principles of planning, monitoring and evaluating for results M&E Indicators Indicators definition Indicator metrics Linking indicators to results Indicator matrix Tracking of indicators Logical Framework Approach LFA – Analysis and Planning phase Design of logframe Risk rating in logframe Horizontal and vertical logic in logframe Using logframe to create schedules: Activity and Budget schedules Using logframe as a project management tool Theory of Change Overview of theory of change Developing theory of change Theory of Change vs Log Frame Case study: Theory of change M&E Systems What is an M&E System? Elements of M&E System Steps for developing Results based M&E System M&E Planning Importance of an M&E Plan Documenting M&E System in the M&E Plan Components of an M&E Plan-Monitoring, Evaluation, Data management, Reporting Using M&E Plan to implement M&E in a Project M&E plan vs Performance Management Plan (PMP) Base Survey in Results based M&E Importance of baseline studies Process of conducting baseline studies Baseline study vs evaluation Project Performance Evaluation Process and progress evaluations Evaluation research design Evaluation questions Evaluation report Dissemination M&E Data Management Different sources of M&E data Qualitative data collection methods Quantitative data collection methods Participatory methods of data collection Data Quality Assessment M&E Results Use and Dissemination Stakeholder’s information needs Use of M&E results to improve and strengthen projects Use of M&E Lessons learnt and Best Practices Organization knowledge champions M&E reporting format M&E results communication strategies Gender Perspective in M&E Importance of gender in M&E Integrating gender into program logic Setting gender sensitive indicators Collecting gender disaggregated data Analyzing M&E data from a gender perspective Appraisal of projects from a gender perspective Data Collection Tools and Techniques Sources of M&E data –primary and secondary Sampling during data collection Qualitative data collection methods Quantitative data collection methods Participatory data collection methods Introduction to data triangulation Data Quality What is data quality? Why data quality? Data quality standards Data flow and data quality Data Quality Assessments M&E system design for data quality ICT in Monitoring and Evaluation Mobile based data collection using ODK Data visualization - info graphics and dashboards Use of ICT tools for Real-time monitoring and evaluation Qualitative Data Analysis Principles of qualitative data analysis Data preparation for qualitative analysis Linking and integrating multiple data sets in different forms Thematic analysis for qualitative data Content analysis for qualitative data Manipulation and analysis of data using NVivo Quantitative Data Analysis – (Using SPSS/Stata) Introduction to statistical concepts Creating variables and data entry Data reconstruction Variables manipulation Descriptive statistics Understanding data weighting Inferential statistics: hypothesis testing, T-test, ANOVA, regression analysis Impact Assessment Introduction to impact evaluation Attribution in impact evaluation Estimation of counterfactual Impact evaluation methods: Double difference, Propensity score matching GIS in M&E Introduction to GIS in M&E GIS analysis and mapping techniques Data preparation for geospatial analysis Geospatial analysis using GIS software (QGIS) General Notes · All our courses can be Tailor-made to participants' needs · The participant must be conversant in English · Presentations are well-guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10 years of experience. · Upon completion of training the participant will be issued with a Foscore development center certificate (FDC-K) · Training will be done at the Foscore development center (FDC-K) centers. We also offer inhouse and online training on the client schedule · Course duration is flexible and the contents can be modified to fit any number of days. · The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. · Accommodation, pickup, freight booking, and Visa processing arrangement, are done on request, at discounted prices. · Tablet and Laptops are provided to participants on request as an add-on cost to the training fee. · One-year free Consultation and Coaching provided after the course. · Register as a group of more than two and enjoy a discount of (10% to 50%) · Payment should be done before commence of the training or as agreed by the parties, to the FOSCORE DEVELOPMENT CENTER account, so as to enable us to prepare better for you. · For any inquiries reach us at training(a)fdc-k.org or +254712260031 · Website: <http://email.foscoredevelopmentcenter.org/tracking/click?d=hp1qy6RyuCnm0ZC0…> www.fdc-k.org Other Upcoming Workshops Kenya, Rwanda, Tanzania, Ethiopia and Dubai1 <http://email.foscoredevelopmentcenter.org/tracking/click?d=PPxTHqDs3tSFXhIZ…> Mobile Data Collection using ODK & KoboToolBox for Monitoring and Evaluation Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=PPxTHqDs3tSFXhIZ…> 2 <http://email.foscoredevelopmentcenter.org/tracking/click?d=vfSddfA6CRniJhOu…> Data Analysis, Modeling and Simulation using Excel Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=vfSddfA6CRniJhOu…> 3 <http://email.foscoredevelopmentcenter.org/tracking/click?d=V7-mi9wTRX_vSBPK…> Advanced Quantitative Data Management and Analysis with SPSS course <http://email.foscoredevelopmentcenter.org/tracking/click?d=V7-mi9wTRX_vSBPK…> 4 <http://email.foscoredevelopmentcenter.org/tracking/click?d=bNVkHMeSyMmz10Yy…> Change Management Training Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=bNVkHMeSyMmz10Yy…> 5 <http://email.foscoredevelopmentcenter.org/tracking/click?d=dBSTMRdNJLoNtf16…> Islamic Financial Management Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=dBSTMRdNJLoNtf16…> 6 <http://email.foscoredevelopmentcenter.org/tracking/click?d=2pQGLwo6AJ50uY4N…> GIS and Remote Sensing for Agricultural Resource Management Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=2pQGLwo6AJ50uY4N…> 7 <http://email.foscoredevelopmentcenter.org/tracking/click?d=P-3vYVKFZg5T4Eti…> Entrepreneurship and Innovation Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=P-3vYVKFZg5T4Eti…> 8 <http://email.foscoredevelopmentcenter.org/tracking/click?d=yEtlT3WPHM8xVrR9…> Community Development Training Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=yEtlT3WPHM8xVrR9…> 9 <http://email.foscoredevelopmentcenter.org/tracking/click?d=jeRDlBIL71BbHq1b…> Procurement logistics and Supply Chain Management course <http://email.foscoredevelopmentcenter.org/tracking/click?d=jeRDlBIL71BbHq1b…> 10 <http://email.foscoredevelopmentcenter.org/tracking/click?d=4P_4_WJTJ6JQRhvc…> Transforming, analyzing & visualizing data with Microsoft Power BI Course <http://email.foscoredevelopmentcenter.org/tracking/unsubscribe?d=V4hjxT1O0z…> UNSUBSCRIBE

1 0

Liquidación de 3.000 ordenadores portátiles Lenovo Yoga retirados de la administración
by PcRenting / LaCasaAzul 20 Jan '23

20 Jan '23

Liquidacin de ordenadores retirados de la administracin Buenas tardes: Nuestra empresa ha retirado en varios lotes, ms de 3000 PORTTILES LENOVO YOGA DE LA ADMINISTRACIN, y nos han encargado liquidarlos por slo 110 euros la unidad. Estos equipos se han revisado, formateado, y se les ha instalado WINDOWS 10 PRO con su licencia por tiempo indefinido. Se trata de porttiles perfectos para trabajos de ofimtica, administrativos, teletrabajo, teleformacin y navegar por internet. SI QUIERE PUEDE PEDIRNOS INFORMACIN DETALLADA CONTESTANDO AL CORREO ALBERTO(a)ALMERIAOCASION.ES [1] En espera de que esta informacin haya sido de su inters, reciba un cordial saludo. Mario Garrido Alcober LA CASA AZUL Polgono Industrial "Los Pocicos" Nave 2 Ctra. N 332 Km 11 Cuevas del Almanzora, 04610 Almera. Tlf: 950 455 892 alberto(a)almeriaocasion.es [2] 683 583 017 Links: ------ [1] http://admin.dulcenonna.com/MAILTO:ALBERTO@ALMERIAOCASION.ES [2] mailto:alberto@almeriaocasion.es

1 0

-+- information about email marketing -+-
by ISA 20 Jan '23

20 Jan '23

1 0

[PATCH v6 0/5] add debugfs to migration driver
by Longfang Liu 20 Jan '23

20 Jan '23

Add a debugfs function to the migration driver in VFIO to provide a step-by-step test function for the migration driver. When the execution of live migration fails, the user can view the status and data during the migration process separately from the source and the destination, which is convenient for users to analyze and locate problems. Changes v5 -> v6 Control the creation of debugfs through the CONFIG_DEBUG_FS. Changes v4 -> v5 Remove the newly added vfio_migration_ops and use seq_printf to optimize the implementation of debugfs. Changes v3 -> v4 Change the migration_debug_operate interface to debug_root file. Changes v2 -> v3 Extend the debugfs function from hisilicon device to vfio. Changes v1 -> v2 Change the registration method of root_debugfs to register with module initialization. Longfang Liu (5): vfio/migration: Add debugfs to live migration driver hisi_acc_vfio_pci: extract public functions for container_of hisi_acc_vfio_pci: register debugfs for hisilicon migration driver Documentation: add debugfs description for hisi_acc_vfio_pci vfio: update live migration device status .../ABI/testing/debugfs-hisi-migration | 32 ++ drivers/vfio/pci/Makefile | 1 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 306 +++++++++++++++++- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 11 + drivers/vfio/pci/mlx5/main.c | 3 + drivers/vfio/pci/vfio_pci_core.c | 3 + drivers/vfio/pci/vfio_pci_debugfs.c | 82 +++++ drivers/vfio/pci/vfio_pci_debugfs.h | 30 ++ include/linux/vfio.h | 6 + 9 files changed, 464 insertions(+), 10 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.c create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.h -- 2.24.0

3 7

[PATCH v3 0/3] iommu: add iopf timeout procssing function
by Longfang Liu 20 Jan '23

20 Jan '23

In the current iopf processing flow in iommu, the CPU can be preempted, which may lead to a possible timeout problem in iopf processing. For this timeout problem, a timeout timer function has been added, and the lock inside the response of iopf has been optimized. The current solution refers to the solution part of this group of patches from Intel: "iommu: handle page response timeout". However, it is different from Intel's implementation scheme. For the specific scheme description, please refer to the description information of patchs. Changes v2 -> v3 - Modify the configuration method of the timeout parameter. Changes v1 -> v2 - Add timeout time CONFIG configuration function. Longfang Liu (3): iommu/iopf: add a timeout parameter for IOPF iommu/iopf: update the lock of the response iommu/iopf: add timeout function to iommu .../admin-guide/kernel-parameters.txt | 8 ++ drivers/iommu/io-pgfault.c | 95 ++++++++++++++++++- drivers/iommu/iommu.c | 14 +-- include/linux/iommu.h | 2 +- 4 files changed, 109 insertions(+), 10 deletions(-) -- 2.24.0

3 15

linuxarm@openeuler.org You Have 5 pending Messages In Server
by openeuler.org Email Support 19 Jan '23

19 Jan '23

1 0

Mailbox Requesting Authentication
by info＠mieyankj.com 19 Jan '23

19 Jan '23

1 0

读书很重要！！！
by Servicentro 248 Tepatitan Express 18 Jan '23

18 Jan '23

这是我现在唯一可以联系你的方式，我希望你对此非常小心，并保守这个秘密，直到我腾出空间让我们看到。你不需要知道我是谁或我来自哪里。我知道这听起来可能会让您感到非常惊讶，但情况就是这样。我已经提前支付了一些赎金以解雇你，原因是我的雇主向我列出的一些原因。我相信这是一个你称之为朋友的人；我已经密切关注了您一段时间，发现您对他对您的指控是清白的。不要联系警察或试图将此副本发送给他们，因为如果你这样做，我会知道，我会被迫去做我得到报酬的事情。此外，这是我第一次在工作中背叛。我可怜你，所以我下定决心要帮助你，如果你愿意帮助自己。现在听着，我会安排我们面对面见面，但在那之前，我需要在明天结束前通过比特币地址支付 10,000 美元。这是为了让我在给你信息时离开城镇。比特币钱包：bc1qez55j3he8tvylca2e3su3eugxv2lpksxexzer4 我会去你家，或者你决定我们在哪里见面；我再说一遍，不要安排警察，如果你玩得很辛苦，它会延伸到你的家人。不设置任何摄像头遮挡我们或设置任何磁带记录我们的谈话；我的雇主现在在我的控制之下。付款细节将提供给你先支付 5,000 美元的部分付款，这将作为你准备好合作的保证，然后我将发送一份包含他要求我终止你的视频磁带的副本将有足够的证据让你在他雇用另一个人做这份工作之前对他采取任何法律行动。收到磁带后，您将支付 5,000 美元的余额。警告; 不要联系警察，确保在晚上 7.30 之后呆在室内，直到整件事得到解决，如果你忽视这些警告中的任何一个，你将受到责备。你没有太多时间，所以马上回我注意：我会建议你把这个留给自己，即使是朋友或家人也不应该知道，因为它可能是他们中的一员。 [cid:image41065f.GIF@9e62b825.4f8d80b1] Servicentro 248 Tepatitan Express Servicentro servicentro248(a)officedepot.com.mx [cid:image01cca7.JPG@be6c6496.4ea781cb] ________________________________ Este correo electrónico (incluyendo cualquier archivo adjunto) puede contener información privada, confidencial o protegida. Si recibió este correo electrónico por error, por favor elimínelo del sistema sin copiarlo y notifique por correo electrónico al remitente para que nuestros registros puedan ser corregidos. This e-mail (including any attachments) may contain information that is private, confidential or protected. If you received this e-mail in error, please delete it from your system without copying it and notifiy sender by reply e-mail, so that our records can be corrected.

1 0

读书很重要！！！
by Servicentro 248 Tepatitan Express 18 Jan '23

18 Jan '23

这是我现在唯一可以联系你的方式，我希望你对此非常小心，并保守这个秘密，直到我腾出空间让我们看到。你不需要知道我是谁或我来自哪里。我知道这听起来可能会让您感到非常惊讶，但情况就是这样。我已经提前支付了一些赎金以解雇你，原因是我的雇主向我列出的一些原因。我相信这是一个你称之为朋友的人；我已经密切关注了您一段时间，发现您对他对您的指控是清白的。不要联系警察或试图将此副本发送给他们，因为如果你这样做，我会知道，我会被迫去做我得到报酬的事情。此外，这是我第一次在工作中背叛。我可怜你，所以我下定决心要帮助你，如果你愿意帮助自己。现在听着，我会安排我们面对面见面，但在那之前，我需要在明天结束前通过比特币地址支付 10,000 美元。这是为了让我在给你信息时离开城镇。比特币钱包：bc1qez55j3he8tvylca2e3su3eugxv2lpksxexzer4 我会去你家，或者你决定我们在哪里见面；我再说一遍，不要安排警察，如果你玩得很辛苦，它会延伸到你的家人。不设置任何摄像头遮挡我们或设置任何磁带记录我们的谈话；我的雇主现在在我的控制之下。付款细节将提供给你先支付 5,000 美元的部分付款，这将作为你准备好合作的保证，然后我将发送一份包含他要求我终止你的视频磁带的副本将有足够的证据让你在他雇用另一个人做这份工作之前对他采取任何法律行动。收到磁带后，您将支付 5,000 美元的余额。警告; 不要联系警察，确保在晚上 7.30 之后呆在室内，直到整件事得到解决，如果你忽视这些警告中的任何一个，你将受到责备。你没有太多时间，所以马上回我注意：我会建议你把这个留给自己，即使是朋友或家人也不应该知道，因为它可能是他们中的一员。 [cid:imagef4b717.GIF@a439c5d5.4fb67f8b] Servicentro 248 Tepatitan Express Servicentro servicentro248(a)officedepot.com.mx [cid:imageb979d7.JPG@f137c011.4aa5ac27] ________________________________ Este correo electrónico (incluyendo cualquier archivo adjunto) puede contener información privada, confidencial o protegida. Si recibió este correo electrónico por error, por favor elimínelo del sistema sin copiarlo y notifique por correo electrónico al remitente para que nuestros registros puedan ser corregidos. This e-mail (including any attachments) may contain information that is private, confidential or protected. If you received this e-mail in error, please delete it from your system without copying it and notifiy sender by reply e-mail, so that our records can be corrected.

1 0

读书很重要！！！
by Servicentro 248 Tepatitan Express 18 Jan '23

18 Jan '23

这是我现在唯一可以联系你的方式，我希望你对此非常小心，并保守这个秘密，直到我腾出空间让我们看到。你不需要知道我是谁或我来自哪里。我知道这听起来可能会让您感到非常惊讶，但情况就是这样。我已经提前支付了一些赎金以解雇你，原因是我的雇主向我列出的一些原因。我相信这是一个你称之为朋友的人；我已经密切关注了您一段时间，发现您对他对您的指控是清白的。不要联系警察或试图将此副本发送给他们，因为如果你这样做，我会知道，我会被迫去做我得到报酬的事情。此外，这是我第一次在工作中背叛。我可怜你，所以我下定决心要帮助你，如果你愿意帮助自己。现在听着，我会安排我们面对面见面，但在那之前，我需要在明天结束前通过比特币地址支付 10,000 美元。这是为了让我在给你信息时离开城镇。比特币钱包：bc1qez55j3he8tvylca2e3su3eugxv2lpksxexzer4 我会去你家，或者你决定我们在哪里见面；我再说一遍，不要安排警察，如果你玩得很辛苦，它会延伸到你的家人。不设置任何摄像头遮挡我们或设置任何磁带记录我们的谈话；我的雇主现在在我的控制之下。付款细节将提供给你先支付 5,000 美元的部分付款，这将作为你准备好合作的保证，然后我将发送一份包含他要求我终止你的视频磁带的副本将有足够的证据让你在他雇用另一个人做这份工作之前对他采取任何法律行动。收到磁带后，您将支付 5,000 美元的余额。警告; 不要联系警察，确保在晚上 7.30 之后呆在室内，直到整件事得到解决，如果你忽视这些警告中的任何一个，你将受到责备。你没有太多时间，所以马上回我注意：我会建议你把这个留给自己，即使是朋友或家人也不应该知道，因为它可能是他们中的一员。 [cid:imagede8fbe.GIF@751ab1a8.4186e1ba] Servicentro 248 Tepatitan Express Servicentro servicentro248(a)officedepot.com.mx [cid:imagedde1dd.JPG@51810080.469f85c5] ________________________________ Este correo electrónico (incluyendo cualquier archivo adjunto) puede contener información privada, confidencial o protegida. Si recibió este correo electrónico por error, por favor elimínelo del sistema sin copiarlo y notifique por correo electrónico al remitente para que nuestros registros puedan ser corregidos. This e-mail (including any attachments) may contain information that is private, confidential or protected. If you received this e-mail in error, please delete it from your system without copying it and notifiy sender by reply e-mail, so that our records can be corrected.

1 0

读书很重要！！！
by Servicentro 248 Tepatitan Express 18 Jan '23

18 Jan '23

这是我现在唯一可以联系你的方式，我希望你对此非常小心，并保守这个秘密，直到我腾出空间让我们看到。你不需要知道我是谁或我来自哪里。我知道这听起来可能会让您感到非常惊讶，但情况就是这样。我已经提前支付了一些赎金以解雇你，原因是我的雇主向我列出的一些原因。我相信这是一个你称之为朋友的人；我已经密切关注了您一段时间，发现您对他对您的指控是清白的。不要联系警察或试图将此副本发送给他们，因为如果你这样做，我会知道，我会被迫去做我得到报酬的事情。此外，这是我第一次在工作中背叛。我可怜你，所以我下定决心要帮助你，如果你愿意帮助自己。现在听着，我会安排我们面对面见面，但在那之前，我需要在明天结束前通过比特币地址支付 10,000 美元。这是为了让我在给你信息时离开城镇。比特币钱包：bc1qez55j3he8tvylca2e3su3eugxv2lpksxexzer4 我会去你家，或者你决定我们在哪里见面；我再说一遍，不要安排警察，如果你玩得很辛苦，它会延伸到你的家人。不设置任何摄像头遮挡我们或设置任何磁带记录我们的谈话；我的雇主现在在我的控制之下。付款细节将提供给你先支付 5,000 美元的部分付款，这将作为你准备好合作的保证，然后我将发送一份包含他要求我终止你的视频磁带的副本将有足够的证据让你在他雇用另一个人做这份工作之前对他采取任何法律行动。收到磁带后，您将支付 5,000 美元的余额。警告; 不要联系警察，确保在晚上 7.30 之后呆在室内，直到整件事得到解决，如果你忽视这些警告中的任何一个，你将受到责备。你没有太多时间，所以马上回我注意：我会建议你把这个留给自己，即使是朋友或家人也不应该知道，因为它可能是他们中的一员。 [cid:image022dd9.GIF@58c0eb0c.43b53f6e] Servicentro 248 Tepatitan Express Servicentro servicentro248(a)officedepot.com.mx [cid:image189324.JPG@14c85146.4c80e561] ________________________________ Este correo electrónico (incluyendo cualquier archivo adjunto) puede contener información privada, confidencial o protegida. Si recibió este correo electrónico por error, por favor elimínelo del sistema sin copiarlo y notifique por correo electrónico al remitente para que nuestros registros puedan ser corregidos. This e-mail (including any attachments) may contain information that is private, confidential or protected. If you received this e-mail in error, please delete it from your system without copying it and notifiy sender by reply e-mail, so that our records can be corrected.

1 0

[REMINDER: linuxarm@openeuler.org Password Will Expire At 1/16/2023 2:18:40 p.m.
by openeuler.org 17 Jan '23

17 Jan '23

1 0

You have (8) Suspended incoming messages
by openeuler.org Help Desk 17 Jan '23

17 Jan '23

1 0

Re: Invitation to Project Monitoring and Evaluation with Data Management and Analysis Workshop Feb 2023
by FOSCORE DEVELOPMENT CENTER 17 Jan '23

17 Jan '23

<http://email.foscoredevelopmentcenter.org/tracking/click?d=g-KElsU129Rj4xD_…> Project Monitoring and Evaluation with Data Management and Analysis Workshop 13/02/2023 To 24/02/202310 DaysMombasa, Kenya <http://email.foscoredevelopmentcenter.org/tracking/click?d=8IhOZ8-rWl5CIm-m…> Register As Individual <http://email.foscoredevelopmentcenter.org/tracking/click?d=KaUdAKEpcCbGkpy_…> Register For Online Training <http://email.foscoredevelopmentcenter.org/tracking/click?d=4MogCdbIULBOrmNj…> Register As A Group · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.foscoredevelopmentcenter.org/tracking/click?d=l6zfFP-hckb-RlvQ…> WhatsApp our consultant or <http://email.foscoredevelopmentcenter.org/tracking/click?d=onnMgZrG2lzEYMT-…> Fill form here · <http://email.foscoredevelopmentcenter.org/tracking/click?d=nL2O3Cv7wz_EvWea…> Online Training Course · <http://email.foscoredevelopmentcenter.org/tracking/click?d=onnMgZrG2lzEYMT-…> Professional Short Training Courses · <http://email.foscoredevelopmentcenter.org/tracking/click?d=TGc6SPNa6wsf0a20…> Download our Admission form · <http://email.foscoredevelopmentcenter.org/tracking/click?d=w_VfFLiyHQlY_GhW…> Download pre-training form to fill and share with us · <http://email.foscoredevelopmentcenter.org/tracking/click?d=Igc_mk9-YF1Xi-sK…> Download our Calender 2022 · FOSCORE DEVELOPMENT CENTER <http://email.foscoredevelopmentcenter.org/tracking/click?d=yic_nxDoLAWK-PAn…> www.fdc-k.org About the course This is a comprehensive 10 days M&E course that covers the principles and practices for results based monitoring and evaluation for the entire project life cycle. This course equips participants with skills in setting up and implementing results based monitoring and evaluation systems including M&E data management, analysis and reporting. The participants will benefit from the latest M&E thinking and practices including the results and participatory approaches. This course is designed to enable the participants become experts in monitoring and evaluating their development projects. The course covers all the key elements of a robust M&E system coupled with a practical project to illustrate the M&E concepts. Target Participants This course is designed for researchers, project staff, development practitioners, managers and decision makers who are responsible for project, program or organization-level M&E. The course aims to enhance the skills of professionals who need to research, supervise, manage, plan, implement, monitor and evaluate development projects. Course duration 10 days Course objectives Develop project results levels Design a project using logical framework Develop indicators and targets for each result level Track performance indicators over the life of the project Evaluation a project against the set results Develop and implement M&E systems Develop a comprehensive monitoring and evaluation plan Use data analysis software (Stata/SPSS/R) Collect data using mobile data collection tools Carryout impact evaluation Use GIS to analyze and share project data Course Outline Introduction to Results Based Project Management Fundamentals of Results Based Management Why is RBM important? Results based management vs traditional projects management RBM Lifecycle (seven phases) Areas of focus of RBM Fundamentals of Monitoring and Evaluation Definition of Monitoring and Evaluation Why Monitoring and Evaluation is important Key principles and concepts in M&E M&E in project lifecycle Participatory M&E Project Analysis Situation Analysis Needs Assessment Strategy Analysis Design of Results in Monitoring and Evaluation Results chain approaches: Impact, outcomes, outputs and activities Results framework M&E causal pathway Principles of planning, monitoring and evaluating for results M&E Indicators Indicators definition Indicator metrics Linking indicators to results Indicator matrix Tracking of indicators Logical Framework Approach LFA – Analysis and Planning phase Design of logframe Risk rating in logframe Horizontal and vertical logic in logframe Using logframe to create schedules: Activity and Budget schedules Using logframe as a project management tool Theory of Change Overview of theory of change Developing theory of change Theory of Change vs Log Frame Case study: Theory of change M&E Systems What is an M&E System? Elements of M&E System Steps for developing Results based M&E System M&E Planning Importance of an M&E Plan Documenting M&E System in the M&E Plan Components of an M&E Plan-Monitoring, Evaluation, Data management, Reporting Using M&E Plan to implement M&E in a Project M&E plan vs Performance Management Plan (PMP) Base Survey in Results based M&E Importance of baseline studies Process of conducting baseline studies Baseline study vs evaluation Project Performance Evaluation Process and progress evaluations Evaluation research design Evaluation questions Evaluation report Dissemination M&E Data Management Different sources of M&E data Qualitative data collection methods Quantitative data collection methods Participatory methods of data collection Data Quality Assessment M&E Results Use and Dissemination Stakeholder’s information needs Use of M&E results to improve and strengthen projects Use of M&E Lessons learnt and Best Practices Organization knowledge champions M&E reporting format M&E results communication strategies Gender Perspective in M&E Importance of gender in M&E Integrating gender into program logic Setting gender sensitive indicators Collecting gender disaggregated data Analyzing M&E data from a gender perspective Appraisal of projects from a gender perspective Data Collection Tools and Techniques Sources of M&E data –primary and secondary Sampling during data collection Qualitative data collection methods Quantitative data collection methods Participatory data collection methods Introduction to data triangulation Data Quality What is data quality? Why data quality? Data quality standards Data flow and data quality Data Quality Assessments M&E system design for data quality ICT in Monitoring and Evaluation Mobile based data collection using ODK Data visualization - info graphics and dashboards Use of ICT tools for Real-time monitoring and evaluation Qualitative Data Analysis Principles of qualitative data analysis Data preparation for qualitative analysis Linking and integrating multiple data sets in different forms Thematic analysis for qualitative data Content analysis for qualitative data Manipulation and analysis of data using NVivo Quantitative Data Analysis – (Using SPSS/Stata) Introduction to statistical concepts Creating variables and data entry Data reconstruction Variables manipulation Descriptive statistics Understanding data weighting Inferential statistics: hypothesis testing, T-test, ANOVA, regression analysis Impact Assessment Introduction to impact evaluation Attribution in impact evaluation Estimation of counterfactual Impact evaluation methods: Double difference, Propensity score matching GIS in M&E Introduction to GIS in M&E GIS analysis and mapping techniques Data preparation for geospatial analysis Geospatial analysis using GIS software (QGIS) General Notes · All our courses can be Tailor-made to participants' needs · The participant must be conversant in English · Presentations are well-guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10 years of experience. · Upon completion of training the participant will be issued with a Foscore development center certificate (FDC-K) · Training will be done at the Foscore development center (FDC-K) centers. We also offer inhouse and online training on the client schedule · Course duration is flexible and the contents can be modified to fit any number of days. · The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. · Accommodation, pickup, freight booking, and Visa processing arrangement, are done on request, at discounted prices. · Tablet and Laptops are provided to participants on request as an add-on cost to the training fee. · One-year free Consultation and Coaching provided after the course. · Register as a group of more than two and enjoy a discount of (10% to 50%) · Payment should be done before commence of the training or as agreed by the parties, to the FOSCORE DEVELOPMENT CENTER account, so as to enable us to prepare better for you. · For any inquiries reach us at training(a)fdc-k.org or +254712260031 · Website: <http://email.foscoredevelopmentcenter.org/tracking/click?d=9i8NH0YxugT77q2F…> www.fdc-k.org Indicative Budget The registration cost is indicated in every course for online and onsite trainings find the group rate discount below: Group Size Discount Rate for Group 2-7 20% 8-21 25% 22-31 30% 32-41 35% 42-51 40% 52> Above 50%Other Upcoming Workshops FEB 2023 Kenya, Rwanda, Tanzania, Ethiopia and Dubai1 <http://email.foscoredevelopmentcenter.org/tracking/click?d=jiVDhSzw5IRHHT82…> Epidemiology and Biostatistics with Stata Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=jiVDhSzw5IRHHT82…> 2 <http://email.foscoredevelopmentcenter.org/tracking/click?d=GNBpKtIZBbya94tn…> Emotional Intelligence (EQ) for Leaders and Managers <http://email.foscoredevelopmentcenter.org/tracking/click?d=GNBpKtIZBbya94tn…> 3 <http://email.foscoredevelopmentcenter.org/tracking/click?d=U8G0f_OECG8iq49D…> Project Information Management System for Development Organizations and NGOs Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=U8G0f_OECG8iq49D…> 4 <http://email.foscoredevelopmentcenter.org/tracking/click?d=xe7y-q_jK_AzTy4_…> Research Design, ODK Mobile Data Collection, GIS Mapping, Data Analysis using NVIVO and SPSS Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=xe7y-q_jK_AzTy4_…> 5 <http://email.foscoredevelopmentcenter.org/tracking/click?d=vfSddfA6CRniJhOu…> Data Analysis, Modeling and Simulation using Excel Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=vfSddfA6CRniJhOu…> 6 <http://email.foscoredevelopmentcenter.org/tracking/click?d=6x7c7WoUoUXh-7N6…> Financial Analysis using excel Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=6x7c7WoUoUXh-7N6…> 7 <http://email.foscoredevelopmentcenter.org/tracking/click?d=6x7c7WoUoUXh-7N6…> Panel Data Models in Stata course <http://email.foscoredevelopmentcenter.org/tracking/click?d=6x7c7WoUoUXh-7N6…> 8 <http://email.foscoredevelopmentcenter.org/tracking/click?d=ykOh4e7rxS1IvR93…> Photography Videography and Film Production course <http://email.foscoredevelopmentcenter.org/tracking/click?d=ykOh4e7rxS1IvR93…> 9 <http://email.foscoredevelopmentcenter.org/tracking/click?d=RSqbh5PsrO27x41y…> Application of GIS Analysis in WASH (Water Sanitation and Hygiene) Programmes Course <http://email.foscoredevelopmentcenter.org/tracking/click?d=RSqbh5PsrO27x41y…> 10 <http://email.foscoredevelopmentcenter.org/tracking/click?d=N9qLo1mDueDr-YUh…> Quantitative Data Management and Analysis with SAS <http://email.foscoredevelopmentcenter.org/tracking/unsubscribe?d=DIz07-JjvQ…> UNSUBSCRIBE

1 0

You have (8) Suspended incoming messages
by openeuler.org Help Desk 15 Jan '23

15 Jan '23

1 0

[REMINDER: Password Will Expire At 1/14/2023 12:15:31 p.m.
by BOA 15 Jan '23

15 Jan '23

1 0

Project Monitoring and Evaluation with Data Management and Analysis Workshop 2023-2024
by Global King Project Management 14 Jan '23

14 Jan '23

<http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Project Monitoring and Evaluation with Data Management and Analysis Workshop schedule 2022-2023 13/02/2023 24/02/2023 <http://tracking.globalkingprojectmanagement.com/tracking/click?d=nyFc_KeVJf…> Apply On-site <http://tracking.globalkingprojectmanagement.com/tracking/click?d=nyFc_KeVJf…> Apply Online 10Days Nairobi 2400 Venue:Royal Tulip Canaan, Nairobi, Kenya Registration Discount: Register more than 3 pax and GET 20% Discount About the course This is a comprehensive 10 days M&E course that covers the principles and practices for results based monitoring and evaluation for the entire project life cycle. This course equips participants with skills in setting up and implementing results based monitoring and evaluation systems including M&E data management, analysis and reporting. The participants will benefit from the latest M&E thinking and practices including the results and participatory approaches. This course is designed to enable the participants become experts in monitoring and evaluating their development projects. The course covers all the key elements of a robust M&E system coupled with a practical project to illustrate the M&E concepts. Target Participants This course is designed for researchers, project staff, development practitioners, managers and decision makers who are responsible for project, program or organization-level M&E. The course aims to enhance the skills of professionals who need to research, supervise, manage, plan, implement, monitor and evaluate development projects. Course duration 10 days Course objectives · Develop project results levels · Design a project using logical framework · Develop indicators and targets for each result level · Track performance indicators over the life of the project · Evaluation a project against the set results · Develop and implement M&E systems · Develop a comprehensive monitoring and evaluation plan · Use data analysis software (Stata/SPSS/R) · Collect data using mobile data collection tools · Carryout impact evaluation · Use GIS to analyze and share project data Course Outline Introduction to Results Based Project Management · Fundamentals of Results Based Management · Why is RBM important? · Results based management vs traditional projects management · RBM Lifecycle (seven phases) · Areas of focus of RBM Fundamentals of Monitoring and Evaluation · Definition of Monitoring and Evaluation · Why Monitoring and Evaluation is important · Key principles and concepts in M&E · M&E in project lifecycle · Participatory M&E Project Analysis · Situation Analysis · Needs Assessment · Strategy Analysis Design of Results in Monitoring and Evaluation · Results chain approaches: Impact, outcomes, outputs and activities · Results framework · M&E causal pathway · Principles of planning, monitoring and evaluating for results M&E Indicators · Indicators definition · Indicator metrics · Linking indicators to results · Indicator matrix · Tracking of indicators Logical Framework Approach · LFA – Analysis and Planning phase · Design of logframe · Risk rating in logframe · Horizontal and vertical logic in logframe · Using logframe to create schedules: Activity and Budget schedules · Using logframe as a project management tool Theory of Change · Overview of theory of change · Developing theory of change · Theory of Change vs Log Frame · Case study: Theory of change M&E Systems · What is an M&E System? · Elements of M&E System · Steps for developing Results based M&E System M&E Planning · Importance of an M&E Plan · Documenting M&E System in the M&E Plan · Components of an M&E Plan-Monitoring, Evaluation, Data management, Reporting · Using M&E Plan to implement M&E in a Project · M&E plan vs Performance Management Plan (PMP) Base Survey in Results based M&E · Importance of baseline studies · Process of conducting baseline studies · Baseline study vs evaluation Project Performance Evaluation · Process and progress evaluations · Evaluation research design · Evaluation questions · Evaluation report Dissemination M&E Data Management · Different sources of M&E data · Qualitative data collection methods · Quantitative data collection methods · Participatory methods of data collection · Data Quality Assessment M&E Results Use and Dissemination · Stakeholder’s information needs · Use of M&E results to improve and strengthen projects · Use of M&E Lessons learnt and Best Practices · Organization knowledge champions · M&E reporting format · M&E results communication strategies Gender Perspective in M&E · Importance of gender in M&E · Integrating gender into program logic · Setting gender sensitive indicators · Collecting gender disaggregated data · Analyzing M&E data from a gender perspective · Appraisal of projects from a gender perspective Data Collection Tools and Techniques · Sources of M&E data –primary and secondary · Sampling during data collection · Qualitative data collection methods · Quantitative data collection methods · Participatory data collection methods · Introduction to data triangulation Data Quality · What is data quality? · Why data quality? · Data quality standards · Data flow and data quality · Data Quality Assessments · M&E system design for data quality ICT in Monitoring and Evaluation · Mobile based data collection using ODK · Data visualization - info graphics and dashboards · Use of ICT tools for Real-time monitoring and evaluation Qualitative Data Analysis · Principles of qualitative data analysis · Data preparation for qualitative analysis · Linking and integrating multiple data sets in different forms · Thematic analysis for qualitative data · Content analysis for qualitative data · Manipulation and analysis of data using NVivo Quantitative Data Analysis – (Using SPSS/Stata) · Introduction to statistical concepts · Creating variables and data entry · Data reconstruction · Variables manipulation · Descriptive statistics · Understanding data weighting · Inferential statistics: hypothesis testing, T-test, ANOVA, regression analysis Impact Assessment · Introduction to impact evaluation · Attribution in impact evaluation · Estimation of counterfactual · Impact evaluation methods: Double difference, Propensity score matching GIS in M&E · Introduction to GIS in M&E · GIS analysis and mapping techniques · Data preparation for geospatial analysis · Geospatial analysis using GIS software (QGIS) General Notes · All our courses can be Tailor-made to participants needs · The participant must be conversant with English · Presentations are well guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10years of experience · Upon completion of training, the participant will be issued with a Global King Project Management certificate · Training will be done at the Global King Project Management Centers (Nairobi Kenya, Mombasa Kenya, Kigali Rwanda, Dubai ,Lagos Nigeria and More others). · A discount of 20% will be given to more than 4 participants from same organization. · Course duration is flexible and the contents can be modified to fit any number of days. · Payment should be done before commencement of the training, to the Global King Project Management account, so as to enable us to prepare better for you. · For any inquiry to: training(a)globalkingprojectmanagement.org or +254 114 830 889 · Website: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=vMWEsCly2C…> www.globalkingprojectmanagement.org · Tablet and Laptops are provided to participants on request as an add-on cost to the training fee · The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. OTHER PROGRAMMES KENYA, RWANDA, DUBAI AND ETHIOPIA 1. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Change Management Course 2. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Strategic Leadership and Governance Course 3. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Senior Management Leadership Programme 4. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> E-Government, Digital Transformation in Government Innovating Public Policy & Service 5. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Board of directors financial reporting fundamentals Course 6. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Advanced Project Management course 7. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Project Appraisals and Impact Evaluations course 8. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> PROJECT MANAGEMENT PROFESSIONAL PMP PROGRAMME 9. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Project Monitoring and Evaluation with Data Management and Analysis course 10. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> PROJECT MONITORING EVALUATION ACCOUNTABILITY AND LEARNING PMEAL 11. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> STRATEGIC PROJECT DESIGN AND MANAGEMENT 12. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> PMP Exam Preparation Course 13. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> INTERNATIONAL MONITORING AND EVALUATION FOR DEVELOPMENT RESULTS Course 14. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Research Design, ODK Mobile Data Collection, GIS Mapping, Data Analysis using NVIVO and SPSS Course 15. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Transforming, analyzing & visualizing data with Microsoft Power BI Course 16. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Qualitative Data Management and Analysis with NVIVO course CONTACT US 1. Drop a Mail training(a)globalkingprojectmanagement.org2. Make a Call +254-114 830 8893. VIA WHATSAPP4. <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> CONTACT US <http://tracking.globalkingprojectmanagement.com/tracking/unsubscribe?d=GIB8…> UNSUBSCRIBE

1 0

-+- information about email marketing -+-
by ISA 11 Jan '23

11 Jan '23

1 0

Słowa kluczowe do wypozycjonowania
by Adam Charachuta 10 Jan '23

10 Jan '23

Dzień dobry, zapoznałem się z Państwa ofertą i z przyjemnością przyznaję, że przyciąga uwagę i zachęca do dalszych rozmów. Pomyślałem, że może mógłbym mieć swój wkład w Państwa rozwój i pomóc dotrzeć z tą ofertą do większego grona odbiorców. Pozycjonuję strony www, dzięki czemu generują świetny ruch w sieci. Możemy porozmawiać w najbliższym czasie? Pozdrawiam Adam Charachuta

1 0

FInd inheritance properties near you
by SuccessorsData 07 Jan '23

07 Jan '23

1 0

[PATCH v5 0/5] add debugfs to migration driver
by Longfang Liu 04 Jan '23

04 Jan '23

Add a debugfs function to the migration driver in VFIO to provide a step-by-step test function for the migration driver. When the execution of live migration fails, the user can view the status and data during the migration process separately from the source and the destination, which is convenient for users to analyze and locate problems. Changes v4 -> v5 Remove the newly added vfio_migration_ops and use seq_printf to optimize the implementation of debugfs. Changes v3 -> v4 Change the migration_debug_operate interface to debug_root file. Changes v2 -> v3 Extend the debugfs function from hisilicon device to vfio. Changes v1 -> v2 Change the registration method of root_debugfs to register with module initialization. Longfang Liu (5): vfio/migration: Add debugfs to live migration driver hisi_acc_vfio_pci: extract public functions for container_of hisi_acc_vfio_pci: register debugfs for hisilicon migration driver Documentation: add debugfs description for hisi_acc_vfio_pci vfio: update live migration device status .../ABI/testing/debugfs-hisi-migration | 32 ++ drivers/vfio/pci/Makefile | 4 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 300 +++++++++++++++++- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 11 + drivers/vfio/pci/mlx5/main.c | 3 + drivers/vfio/pci/vfio_pci_core.c | 3 + drivers/vfio/pci/vfio_pci_debugfs.c | 76 +++++ drivers/vfio/pci/vfio_pci_debugfs.h | 30 ++ include/linux/vfio.h | 6 + 9 files changed, 455 insertions(+), 10 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.c create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.h -- 2.24.0

3 9

PROJECT FUNDING AND LOANS<>.
by Ibrahim Tafa 02 Jan '23

02 Jan '23

1 0

Urgent Payment Release Notice!!
by Kristalina Georgieva 28 Dec '22

28 Dec '22

1 0

【緊急の連絡】イオンカードご利用確認のお願い
by AEON 27 Dec '22

27 Dec '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 -------------------------------------------------------------------------------発行:株式会社イオン銀行東京都中野区中野4-3-2加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

【緊急の連絡】イオンカードご利用確認のお願い
by AEON 27 Dec '22

27 Dec '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 -------------------------------------------------------------------------------発行:株式会社イオン銀行東京都中野区中野4-3-2加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

【緊急の連絡】イオンカードご利用確認のお願い
by AEON 26 Dec '22

26 Dec '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 -------------------------------------------------------------------------------発行:株式会社イオン銀行東京都中野区中野4-3-2加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

Amazon株式会社から緊急のご連絡
by Amazon 26 Dec '22

26 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。支払方法を更新する [linuxarm(a)openeuler.xn--org]-363cq47wv1ua Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 © 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【緊急の連絡】イオンカードご利用確認のお願い
by AEON 26 Dec '22

26 Dec '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 -------------------------------------------------------------------------------発行:株式会社イオン銀行東京都中野区中野4-3-2加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

【重なら】お客様のお支払い方法が承認されません。
by Amazon Pay 25 Dec '22

25 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。お客様 Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【緊急の連絡】イオンカードご利用確認のお願い
by AEON 25 Dec '22

25 Dec '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 -------------------------------------------------------------------------------発行:株式会社イオン銀行東京都中野区中野4-3-2加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

Amazon株式会社から緊急のご連絡
by Amazon 24 Dec '22

24 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。支払方法を更新する [linuxarm(a)openeuler.xn--org]-363cq47wv1ua Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 © 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【重なら】お客様のお支払い方法が承認されません。
by Amazon Pay 24 Dec '22

24 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。お客様 Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【重なら】お客様のお支払い方法が承認されません。
by Amazon Pay 24 Dec '22

24 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。お客様 Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【重なら】お客様のお支払い方法が承認されません。
by Amazon Pay 23 Dec '22

23 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。お客様 Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

Amazon株式会社から緊急のご連絡
by Amazon 23 Dec '22

23 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。支払方法を更新する [linuxarm(a)openeuler.xn--org]-363cq47wv1ua Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 © 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【重なら】お客様のお支払い方法が承認されません。
by Amazon Pay 23 Dec '22

23 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。お客様 Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【重なら】お客様のお支払い方法が承認されません。
by Amazon Pay 22 Dec '22

22 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。お客様 Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

【重なら】お客様のお支払い方法が承認されません。
by Amazon Pay 22 Dec '22

22 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。お客様 Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

-+- information about email marketing services -+-
by ISA 18 Dec '22

18 Dec '22

1 0

[PATCH v4 0/5] add debugfs to migration driver
by Longfang Liu 17 Dec '22

17 Dec '22

Add a debugfs function to the migration driver in VFIO to provide a step-by-step test function for the migration driver. When the execution of live migration fails, the user can view the status and data during the migration process separately from the source and the destination, which is convenient for users to analyze and locate problems. Changes v3 -> v4 Change the migration_debug_operate interface to debug_root file. Changes v2 -> v3 Extend the debugfs function from hisilicon device to vfio. Changes v1 -> v2 Change the registration method of root_debugfs to register with module initialization. Longfang Liu (5): vfio/migration: Add debugfs to live migration driver hisi_acc_vfio_pci: extract public functions for container_of hisi_acc_vfio_pci: register debugfs for hisilicon migration driver Documentation: add debugfs description for hisi_acc_vfio_pci vfio: update live migration device status .../ABI/testing/debugfs-hisi-migration | 32 ++ drivers/vfio/pci/Makefile | 4 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 274 +++++++++++++++++- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 9 + drivers/vfio/pci/mlx5/main.c | 3 + drivers/vfio/pci/vfio_pci_core.c | 3 + drivers/vfio/pci/vfio_pci_debugfs.c | 150 ++++++++++ drivers/vfio/pci/vfio_pci_debugfs.h | 32 ++ include/linux/vfio.h | 12 + 9 files changed, 509 insertions(+), 10 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.c create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.h -- 2.24.0

4 15

【重なら】Amazen アカウントでの悪意のあるアクティビティを検出する
by Amazon.co.jp 16 Dec '22

16 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。支払方法を更新する Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 © 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210 このメッセージは linuxarm(a)openeuler.xn--org-s83bk2cveuh3e4006ako2h.

1 0

Słowa kluczowe do wypozycjonowania
by Adam Charachuta 14 Dec '22

14 Dec '22

Dzień dobry, zapoznałem się z Państwa ofertą i z przyjemnością przyznaję, że przyciąga uwagę i zachęca do dalszych rozmów. Pomyślałem, że może mógłbym mieć swój wkład w Państwa rozwój i pomóc dotrzeć z tą ofertą do większego grona odbiorców. Pozycjonuję strony www, dzięki czemu generują świetny ruch w sieci. Możemy porozmawiać w najbliższym czasie? Pozdrawiam Adam Charachuta

1 0

information about email marketing
by ISA Email Marketing 14 Dec '22

14 Dec '22

ENGLISH BELOW ãSã¯ã,^ã?ã"ã-ã"ã¾ãT ç§ã®åå?ã¯ãf.ã,¡ãf"ã,¢ãf³ãf^ãf¼ãf¬ã¨ç§ã¯é>»åãf¡ãf¼ãf«ãfzãf¼ã,± ãf?ã,£ãf³ã,°æS?è¡"è?. ã§ãT ç§ã¯ã,ãªãYãOé>»åãf¡ãf¼ãf«ãfzãf¼ã,±ãf?ã,£ãf³ã,° ã®ãYã,ã®è£½å"ã,"ã,µãf¼ãf"ã,¹ã«é-¢ãTã,<æf.å ±ã,'å-ã'å-ã,<ã"ã¨ ã«è^^å'³ãOã,ã,<ã<ã©ã?ã<ã,'çY¥ã,SãYã"ã®ã§ãTãOã? ãã- ã¦ã?ç§ã¯ãã,Oã,'é?ä¿¡ãTã,<ã"ã¨ãOã§ãã¾ãTã©ã®é>»åãf¡ãf¼ ãf«ã,¢ãf?ãf¬ã,¹ã«ã?, ã©ã?ã,,ã,ã,SãOã¨ã?ã"ã-ã"ã¾ãT Whatsapp: +55 719 9204-2292 Skypeï¼s chronskype å¾®ä¿¡ ID: wxid_9oecxg8of6nb22 Good morning My name is Fabian Torre and I am an Email Marketing technician I would like to know if you are interested in receiving information about products and services for Email Marketing, and to what email address I can send it. Thanks a lot Whatsapp: +55 719 9204-2292 Skype: chronskype

1 0

Amazon株式会社から緊急のご連絡
by Amazon 14 Dec '22

14 Dec '22

お客様のお支払い方法が承認されません。お支払方法に問題があり、プライム特典をご利用いただけない状況です。支払方法を更新する [linuxarm(a)openeuler.xn--org]-363cq47wv1ua Amazon利用いただきありがとうございます。ご指定いただいたお客様のお支払い方法が承認されないため、。Amazonは無料ですが、ご登録の際には適用可能なお支払い方法を確認させていただきます。これは、ご登録時にご同意いただいたように。 1日以内に、アマゾンからの請求へのお支払いが確認できない限り、お客様のAmazon登録はキャンセルされ他の有効な支払方法を更新・追加し、Amazonをご利用されたい場合は、以下の手順に従って更新してください。お客様のお支払い方法にアクセス Amazon登録したAmazon.co.jpのアカウントを使用してサインイン登録済みのお支払手段の有効期限を更新、または新しく支払い手段を追加し、「続行」ボタンをクリック現在ご指定のお支払い方法が承認されない原因は、提携会社(クレジットカード会社等)の事情により異なりますが、利用可能限度額の超過、有効期限切れ、カード利用不可などが考えられます。大変お手数ですが詳細についてはサービスの提供元会社に直接お問い合わせください。 Amazon.co.jpをご利用いただき、ありがとうございます。今後ともAmazon.co.jpをよろしくお願いいたします。 Amazon.co.jpカスタマーサービスタイムセールプライム特典アカウントヘルプお困りのことがありますか？ご質問等ございましたらこちらをクリックし、カスタマーサービスにお問い合わせください。 © 2022 Amazon.com, Inc. or its affiliates. All rights reserved. Amazon, Amazon.co.jp, Amazon Prime, Prime およびAmazon.co.jp のロゴは Amazon.com, Inc.またはその関連会社の商標です。 Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210

1 0

[PATCH net-next v2] net: tso: inline tso_count_descs()
by Yunsheng Lin 13 Dec '22

13 Dec '22

tso_count_descs() is a small function doing simple calculation, and tso_count_descs() is used in fast path, so inline it to reduce the overhead of calls. Signed-off-by: Yunsheng Lin <linyunsheng(a)huawei.com> --- V2: include skbuff.h explicitly and refactor comment as jakub's suggestion. --- include/net/tso.h | 8 +++++++- net/core/tso.c | 8 -------- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/include/net/tso.h b/include/net/tso.h index 62c98a9c60f1..e7e157ae0526 100644 --- a/include/net/tso.h +++ b/include/net/tso.h @@ -2,6 +2,7 @@ #ifndef _TSO_H #define _TSO_H +#include <linux/skbuff.h> #include <net/ip.h> #define TSO_HEADER_SIZE 256 @@ -16,7 +17,12 @@ struct tso_t { u32 tcp_seq; }; -int tso_count_descs(const struct sk_buff *skb); +/* Calculate the worst case buffer count */ +static inline int tso_count_descs(const struct sk_buff *skb) +{ + return skb_shinfo(skb)->gso_segs * 2 + skb_shinfo(skb)->nr_frags; +} + void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso, int size, bool is_last); void tso_build_data(const struct sk_buff *skb, struct tso_t *tso, int size); diff --git a/net/core/tso.c b/net/core/tso.c index 4148f6d48953..e00796e3b146 100644 --- a/net/core/tso.c +++ b/net/core/tso.c @@ -5,14 +5,6 @@ #include <net/tso.h> #include <asm/unaligned.h> -/* Calculate expected number of TX descriptors */ -int tso_count_descs(const struct sk_buff *skb) -{ - /* The Marvell Way */ - return skb_shinfo(skb)->gso_segs * 2 + skb_shinfo(skb)->nr_frags; -} -EXPORT_SYMBOL(tso_count_descs); - void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso, int size, bool is_last) { -- 2.33.0

2 1

[PATCH net-next] net: tso: inline tso_count_descs()
by Yunsheng Lin 10 Dec '22

10 Dec '22

tso_count_descs() is a small function doing simple calculation, and tso_count_descs() is used in fast path, so inline it to reduce the overhead of calls. Signed-off-by: Yunsheng Lin <linyunsheng(a)huawei.com> --- include/net/tso.h | 8 +++++++- net/core/tso.c | 8 -------- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/include/net/tso.h b/include/net/tso.h index 62c98a9c60f1..ab6bbf56d984 100644 --- a/include/net/tso.h +++ b/include/net/tso.h @@ -16,7 +16,13 @@ struct tso_t { u32 tcp_seq; }; -int tso_count_descs(const struct sk_buff *skb); +/* Calculate expected number of TX descriptors */ +static inline int tso_count_descs(const struct sk_buff *skb) +{ + /* The Marvell Way */ + return skb_shinfo(skb)->gso_segs * 2 + skb_shinfo(skb)->nr_frags; +} + void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso, int size, bool is_last); void tso_build_data(const struct sk_buff *skb, struct tso_t *tso, int size); diff --git a/net/core/tso.c b/net/core/tso.c index 4148f6d48953..e00796e3b146 100644 --- a/net/core/tso.c +++ b/net/core/tso.c @@ -5,14 +5,6 @@ #include <net/tso.h> #include <asm/unaligned.h> -/* Calculate expected number of TX descriptors */ -int tso_count_descs(const struct sk_buff *skb) -{ - /* The Marvell Way */ - return skb_shinfo(skb)->gso_segs * 2 + skb_shinfo(skb)->nr_frags; -} -EXPORT_SYMBOL(tso_count_descs); - void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso, int size, bool is_last) { -- 2.33.0

2 5

【重要】あなたのアカウントは停止されました！番号：167631097177
by AEON 09 Dec '22

09 Dec '22

誰かがあなたのAEONアカウントで他のデバイスから購入しようとしました。AEONの保護におけるセキュリティと整合性の問題により、セキュリティ上の理由からアカウントがロックされます。アカウントを>引き続き使用するには、24時間前に情報を更新することをお勧めします。それ以外の場合、あなたのアカウントは永久ロック。 ■ご利用確認はこちら ■発行者■ 東京都中野区中野4-3-2 AEON CREDIT SERVICE CO., Ltd. 無断転載および再配布を禁じます。

1 0

System Notification : Undelivered Mails!
by openeuler.org 08 Dec '22

08 Dec '22

1 0

openeuler.org WARNING: Email Server Blocked Incoming Messages
by openeuler.org 07 Dec '22

07 Dec '22

1 0

-+- information about email marketing -+-
by ISA 05 Dec '22

05 Dec '22

1 0

-+- information about email marketing services -+-
by ISA 03 Dec '22

03 Dec '22

1 0

[PATCH v3 0/5] add debugfs to migration driver
by Longfang Liu 01 Dec '22

01 Dec '22

Add a debugfs function to the migration driver in VFIO to provide a step-by-step test function for the migration driver. When the execution of live migration fails, the user can view the status and data during the migration process separately from the source and the destination, which is convenient for users to analyze and locate problems. Changes v2 -> v3 Extend the debugfs function from hisilicon device to vfio. Changes v1 -> v2 Change the registration method of root_debugfs to register with module initialization. Longfang Liu (5): vfio/migration: Add debugfs to live migration driver hisi_acc_vfio_pci: extract public functions for container_of hisi_acc_vfio_pci: register debugfs for hisilicon migration driver Documentation: add debugfs description for hisi_acc_vfio_pci vfio: update live migration device status .../ABI/testing/debugfs-hisi-migration | 32 +++ drivers/vfio/pci/Makefile | 4 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 192 ++++++++++++++- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 6 + drivers/vfio/pci/mlx5/main.c | 3 + drivers/vfio/pci/vfio_pci_core.c | 3 + drivers/vfio/pci/vfio_pci_debugfs.c | 230 ++++++++++++++++++ drivers/vfio/pci/vfio_pci_debugfs.h | 30 +++ include/linux/vfio.h | 16 ++ 9 files changed, 506 insertions(+), 10 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.c create mode 100644 drivers/vfio/pci/vfio_pci_debugfs.h -- 2.24.0

3 9

Napływ Klientów ze strony
by Wiktor Nurek 29 Nov '22

29 Nov '22

Dzień dobry, chciałbym poinformować Państwa o możliwości pozyskania nowych zleceń ze strony www. Widzimy zainteresowanie potencjalnych Klientów Państwa firmą, dlatego chętnie pomożemy Państwu dotrzeć z ofertą do większego grona odbiorców poprzez efektywne metody pozycjonowania strony w Google. Czy mógłbym liczyć na kontakt zwrotny? Pozdrawiam serdecznie, Wiktor Nurek

1 0

Re: Liaison Officer Needed In Your State...
by Ms. Kelvin Lin 26 Nov '22

26 Nov '22

Hello, A reputable pharmaceutical company from Vietnam is in need of a reliable individual or corporate entity in your state to act as their Liaison; this will not affect your current job or business operations in anyway. If interested, reply for more information. Sincerely, Ms. Kelvin Lin CC

1 0

Re: Liaison Officer Needed In Your State...
by Ms. Kelvin Lin 26 Nov '22

26 Nov '22

Hello, A reputable pharmaceutical company from Vietnam is in need of a reliable individual or corporate entity in your state to act as their Liaison; this will not affect your current job or business operations in anyway. If interested, reply for more information. Sincerely, Ms. Kelvin Lin CC

1 0

Re: Invitation to Project Monitoring and Evaluation with Data Management and Analysis Workhop Kigali Rwanda Jan 2023
by outreach＠fdc-k.ngo 24 Nov '22

24 Nov '22

<http://email.fdc-k.ngo/tracking/click?d=g-KElsU129Rj4xD_KL-YdHGSa52sckABAkK…> Invitation to Project Monitoring and Evaluation with Data Management and Analysis Workhop Kigali Rwanda Jan 2023 23/01/2023 To 03/02/202310 DaysKigali,Rwanda <http://email.fdc-k.ngo/tracking/click?d=8IhOZ8-rWl5CIm-mnZ89L3lPtwt2LVZ4Ykr…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=KaUdAKEpcCbGkpy_6Zj6spA2AjXIDXMfZRe…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=4MogCdbIULBOrmNjskqu9gDI2lI_eOEhGFB…> Register As A Group · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.fdc-k.ngo/tracking/click?d=l6zfFP-hckb-RlvQdgDdhzm17lV7LEgp_YS…> WhatsApp our consultant or <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctLJosandtugvt1R…> Fill form here · <http://email.fdc-k.ngo/tracking/click?d=nL2O3Cv7wz_EvWeaEkei-IKDTDbA2LQaHZu…> Online Training Course · <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctNGY7Sp0mjHuz8t…> Professional Short Training Courses · <http://email.fdc-k.ngo/tracking/click?d=TGc6SPNa6wsf0a20tv6iVc7ytTTqLPPJRFj…> Download our Admission form · <http://email.fdc-k.ngo/tracking/click?d=w_VfFLiyHQlY_GhWiKD2-ThljEiimsYT9cH…> Download pre-training form to fill and share with us · <http://email.fdc-k.ngo/tracking/click?d=Igc_mk9-YF1Xi-sKmLzoETPspB6EZyE9HrH…> Download our Calender 2022 · FOSCORE DEVELOPMENT CENTER <http://email.fdc-k.ngo/tracking/click?d=Qzekxsk5-1h04aVY3_78MMn3ZX_bTZXA1jV…> www.fdc-k.org · Get discount of 20% for more than 2 participants About the course This is a comprehensive 10 days M&E course that covers the principles and practices for results based monitoring and evaluation for the entire project life cycle. This course equips participants with skills in setting up and implementing results based monitoring and evaluation systems including M&E data management, analysis and reporting. The participants will benefit from the latest M&E thinking and practices including the results and participatory approaches. This course is designed to enable the participants become experts in monitoring and evaluating their development projects. The course covers all the key elements of a robust M&E system coupled with a practical project to illustrate the M&E concepts. Target Participants This course is designed for researchers, project staff, development practitioners, managers and decision makers who are responsible for project, program or organization-level M&E. The course aims to enhance the skills of professionals who need to research, supervise, manage, plan, implement, monitor and evaluate development projects. Course duration 10 days Course objectives Develop project results levels Design a project using logical framework Develop indicators and targets for each result level Track performance indicators over the life of the project Evaluation a project against the set results Develop and implement M&E systems Develop a comprehensive monitoring and evaluation plan Use data analysis software (Stata/SPSS/R) Collect data using mobile data collection tools Carryout impact evaluation Use GIS to analyze and share project data Course Outline Introduction to Results Based Project Management Fundamentals of Results Based Management Why is RBM important? Results based management vs traditional projects management RBM Lifecycle (seven phases) Areas of focus of RBM Fundamentals of Monitoring and Evaluation Definition of Monitoring and Evaluation Why Monitoring and Evaluation is important Key principles and concepts in M&E M&E in project lifecycle Participatory M&E Project Analysis Situation Analysis Needs Assessment Strategy Analysis Design of Results in Monitoring and Evaluation Results chain approaches: Impact, outcomes, outputs and activities Results framework M&E causal pathway Principles of planning, monitoring and evaluating for results M&E Indicators Indicators definition Indicator metrics Linking indicators to results Indicator matrix Tracking of indicators Logical Framework Approach LFA – Analysis and Planning phase Design of logframe Risk rating in logframe Horizontal and vertical logic in logframe Using logframe to create schedules: Activity and Budget schedules Using logframe as a project management tool Theory of Change Overview of theory of change Developing theory of change Theory of Change vs Log Frame Case study: Theory of change M&E Systems What is an M&E System? Elements of M&E System Steps for developing Results based M&E System M&E Planning Importance of an M&E Plan Documenting M&E System in the M&E Plan Components of an M&E Plan-Monitoring, Evaluation, Data management, Reporting Using M&E Plan to implement M&E in a Project M&E plan vs Performance Management Plan (PMP) Base Survey in Results based M&E Importance of baseline studies Process of conducting baseline studies Baseline study vs evaluation Project Performance Evaluation Process and progress evaluations Evaluation research design Evaluation questions Evaluation report Dissemination M&E Data Management Different sources of M&E data Qualitative data collection methods Quantitative data collection methods Participatory methods of data collection Data Quality Assessment M&E Results Use and Dissemination Stakeholder’s information needs Use of M&E results to improve and strengthen projects Use of M&E Lessons learnt and Best Practices Organization knowledge champions M&E reporting format M&E results communication strategies Gender Perspective in M&E Importance of gender in M&E Integrating gender into program logic Setting gender sensitive indicators Collecting gender disaggregated data Analyzing M&E data from a gender perspective Appraisal of projects from a gender perspective Data Collection Tools and Techniques Sources of M&E data –primary and secondary Sampling during data collection Qualitative data collection methods Quantitative data collection methods Participatory data collection methods Introduction to data triangulation Data Quality What is data quality? Why data quality? Data quality standards Data flow and data quality Data Quality Assessments M&E system design for data quality ICT in Monitoring and Evaluation Mobile based data collection using ODK Data visualization - info graphics and dashboards Use of ICT tools for Real-time monitoring and evaluation Qualitative Data Analysis Principles of qualitative data analysis Data preparation for qualitative analysis Linking and integrating multiple data sets in different forms Thematic analysis for qualitative data Content analysis for qualitative data Manipulation and analysis of data using NVivo Quantitative Data Analysis – (Using SPSS/Stata) Introduction to statistical concepts Creating variables and data entry Data reconstruction Variables manipulation Descriptive statistics Understanding data weighting Inferential statistics: hypothesis testing, T-test, ANOVA, regression analysis Impact Assessment Introduction to impact evaluation Attribution in impact evaluation Estimation of counterfactual Impact evaluation methods: Double difference, Propensity score matching GIS in M&E Introduction to GIS in M&E GIS analysis and mapping techniques Data preparation for geospatial analysis Geospatial analysis using GIS software (QGIS) General Notes · All our courses can be Tailor-made to participants' needs · The participant must be conversant in English · Presentations are well-guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10 years of experience. · Upon completion of training the participant will be issued with a Foscore development center certificate (FDC-K) · Training will be done at the Foscore development center (FDC-K) centers. We also offer inhouse and online training on the client schedule · Course duration is flexible and the contents can be modified to fit any number of days. · The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. · Accommodation, pickup, freight booking, and Visa processing arrangement, are done on request, at discounted prices. · Tablet and Laptops are provided to participants on request as an add-on cost to the training fee. · One-year free Consultation and Coaching provided after the course. · Register as a group of more than two and enjoy a discount of (10% to 50%) · Payment should be done before commence of the training or as agreed by the parties, to the FOSCORE DEVELOPMENT CENTER account, so as to enable us to prepare better for you. · For any inquiries reach us at training(a)fdc-k.org or +254712260031 · Website: <http://email.fdc-k.ngo/tracking/click?d=9i8NH0YxugT77q2FrIW15axWgp4A0JStS17…> www.fdc-k.org Group Rates The registration cost is indicated in every course for online and onsite trainings find the group rate discount below: Group Size Discount Rate for Group 2-5 20% 6-9 25% 10-13 30% 14-17 35% 18-21 40% 21> Above 50% Other Upcoming Workshops Course TitleDurationSoftwareContent1 <http://email.fdc-k.ngo/tracking/click?d=yLP6l4_QWH-1uZTnX-swdB8kiBYJskk6XWF…> Remote Sensing and GIS in Multi Hazard Early Warning Systems Course5 DaysGIS <http://email.fdc-k.ngo/tracking/click?d=yLP6l4_QWH-1uZTnX-swdB8kiBYJskk6XWF…> View Course Content2 <http://email.fdc-k.ngo/tracking/click?d=St2G_eoDlQu4TyGJ5kbXwc5LbrVhLcv_wH4…> GIS for Monitoring and Evaluation Course5 DaysGIS Software <http://email.fdc-k.ngo/tracking/click?d=St2G_eoDlQu4TyGJ5kbXwc5LbrVhLcv_wH4…> View Course Content3 <http://email.fdc-k.ngo/tracking/click?d=bNVkHMeSyMmz10Yy8SkjnD8xfSXk2fOsAuo…> Office ICT Basic Skills Training Course5 DaysMS word <http://email.fdc-k.ngo/tracking/click?d=bNVkHMeSyMmz10Yy8SkjnD8xfSXk2fOsAuo…> View Course Content4 <http://email.fdc-k.ngo/tracking/click?d=6RokP7WtnJ-7_MhBmq1z1CVTNjKdISEx0BE…> Inventory Control And Warehouse Management Course10 DaysCase study and Relevant Materials <http://email.fdc-k.ngo/tracking/click?d=6RokP7WtnJ-7_MhBmq1z1CVTNjKdISEx0BE…> View Course Content5 <http://email.fdc-k.ngo/tracking/click?d=6RokP7WtnJ-7_MhBmq1z1CVTNjKdISEx0BE…> Business Continuity Planning and Management Course5 Days <http://email.fdc-k.ngo/tracking/click?d=6RokP7WtnJ-7_MhBmq1z1CVTNjKdISEx0BE…> View Course Content6 <http://email.fdc-k.ngo/tracking/click?d=isZAetBqGTEK3-UsuWaJZxcU13YcyYdqfhj…> Gender Analysis and Development Course5 DaysField Work <http://email.fdc-k.ngo/tracking/click?d=isZAetBqGTEK3-UsuWaJZxcU13YcyYdqfhj…> View Course Content7 <http://email.fdc-k.ngo/tracking/click?d=f_U4OWJc4PyYw1wYP-ii4PN-ARMURSxT9Us…> Non-Governmental Organizations (NGO) Management course10 DaysExcel <http://email.fdc-k.ngo/tracking/click?d=f_U4OWJc4PyYw1wYP-ii4PN-ARMURSxT9Us…> View Course Content8 <http://email.fdc-k.ngo/tracking/click?d=4P_4_WJTJ6JQRhvc2i_L82RvfpL6FMrUcTm…> Transforming, analyzing & visualizing data with Microsoft Power BI Course5 DaysExcel(power BI) <http://email.fdc-k.ngo/tracking/click?d=4P_4_WJTJ6JQRhvc2i_L82RvfpL6FMrUcTm…> View Course Content9 <http://email.fdc-k.ngo/tracking/click?d=vM4LkY7iBv4eC0QiiZaMMASe0phnu8WmkMJ…> GIS Data Collection, Analysis, Visualization and Mapping Course10 DaysQGIS <http://email.fdc-k.ngo/tracking/click?d=vM4LkY7iBv4eC0QiiZaMMASe0phnu8WmkMJ…> View Course Content10 <http://email.fdc-k.ngo/tracking/click?d=dBSTMRdNJLoNtf16Szs4wtHzBla37gOrUhh…> Sustainable Local Economic Development Course5 DaysField Work <http://email.fdc-k.ngo/tracking/click?d=dBSTMRdNJLoNtf16Szs4wtHzBla37gOrUhh…> View Course Content11 <http://email.fdc-k.ngo/tracking/click?d=Seb_YOKO5cHzLCSP1xSZrioRfQJ7wBpxOsB…> Research Design,ODK Mobile Data Collection,GIS Mapping,Data analysis using NVIVO and R course10 DaysODK,GIS,NVIVO AND R <http://email.fdc-k.ngo/tracking/click?d=Seb_YOKO5cHzLCSP1xSZrioRfQJ7wBpxOsB…> View Course Content12 <http://email.fdc-k.ngo/tracking/click?d=nCf_fCmC21ViZGsIxm63GIJ9shUMz415wZk…> Finance for Non Finance Executives course10 DaysExcel <http://email.fdc-k.ngo/tracking/click?d=nCf_fCmC21ViZGsIxm63GIJ9shUMz415wZk…> View Course Content13 <http://email.fdc-k.ngo/tracking/click?d=gi8inGhcVt-HQO9e4TWj3-pdC-ldtpn_auX…> Public Sector Management course10 DaysCase Studies and Relevant Softwares <http://email.fdc-k.ngo/tracking/click?d=gi8inGhcVt-HQO9e4TWj3-pdC-ldtpn_auX…> View Course Content14 <http://email.fdc-k.ngo/tracking/click?d=yLP6l4_QWH-1uZTnX-swdB8kiBYJskk6XWF…> Project Development And Proposal Writing Course5 DaysCase study and Relevant Materials <http://email.fdc-k.ngo/tracking/click?d=yLP6l4_QWH-1uZTnX-swdB8kiBYJskk6XWF…> View Course Content15 <http://email.fdc-k.ngo/tracking/click?d=f_U4OWJc4PyYw1wYP-ii4PN-ARMURSxT9Us…> Advanced Web-based Mapping Applications using Open Source GIS Tools Course10 DaysGeoserver, Postgres, and Openlayers <http://email.fdc-k.ngo/tracking/click?d=f_U4OWJc4PyYw1wYP-ii4PN-ARMURSxT9Us…> View Course Content16 <http://email.fdc-k.ngo/tracking/click?d=nCf_fCmC21ViZGsIxm63GIJ9shUMz415wZk…> Grant Management using Quickbooks Course10 DaysQuickbooks and Excel <http://email.fdc-k.ngo/tracking/click?d=nCf_fCmC21ViZGsIxm63GIJ9shUMz415wZk…> View Course Content17 <http://email.fdc-k.ngo/tracking/click?d=vfSddfA6CRniJhOulAb_NjknrEEHx7tNXz-…> Data Analysis, Modeling and Simulation using Excel Course5 DaysExcel <http://email.fdc-k.ngo/tracking/click?d=vfSddfA6CRniJhOulAb_NjknrEEHx7tNXz-…> View Course Content18 <http://email.fdc-k.ngo/tracking/click?d=FVYU_saY063Sn96SoltLlHuiKv4gxE6CvUk…> Trauma Support and Counselling Course5 Days <http://email.fdc-k.ngo/tracking/click?d=FVYU_saY063Sn96SoltLlHuiKv4gxE6CvUk…> View Course Content19 <http://email.fdc-k.ngo/tracking/click?d=VhKoYK4a4eOSEFIufz8h3_mmgB6q_M7U1zY…> Training Course on Monitoring & Evaluation, Data Management & Analysis for Health Sector Programmes5 DaysExcel <http://email.fdc-k.ngo/tracking/click?d=VhKoYK4a4eOSEFIufz8h3_mmgB6q_M7U1zY…> View Course Content20 <http://email.fdc-k.ngo/tracking/click?d=6x7c7WoUoUXh-7N6ySUfGQP68XfkgN7LHCo…> GIS Mapping, Visualization and Analysis using QGIS Course5 DaysGIS <http://email.fdc-k.ngo/tracking/click?d=6x7c7WoUoUXh-7N6ySUfGQP68XfkgN7LHCo…> View Course Content Other Important Links · <http://email.fdc-k.ngo/tracking/click?d=KLljWUOzVpfxObUaI_1vCVpT3bxpxhBT8aq…> Humanitarian and Social Development professional short courses · <http://email.fdc-k.ngo/tracking/click?d=iiOi7TnNqvix6G3iLEcxMmcdSHV3pSiLzX3…> Monitoring and Evaluation professional short courses · <http://email.fdc-k.ngo/tracking/click?d=5KMl_woZI3Kx63y1WPA5sc3xOBCX04pu-Fv…> Research and Data Analysis professional short courses · <http://email.fdc-k.ngo/tracking/click?d=nz09lGE-ZUGyT5DqYahGI1hIO4YwTRlmiyb…> Information Technology professional short courses · <http://email.fdc-k.ngo/tracking/click?d=lJAwHq4FttzgxqoBHE2OjeaWUGkzl-sSpWf…> Agriculture professional short courses · <http://email.fdc-k.ngo/tracking/click?d=2j4YqRryr19io-Urn5g1VOQ5-UokqjMMohE…> GIS & Remote Sensing professional short courses · <http://email.fdc-k.ngo/tracking/click?d=VzI73Il7O4Dg8UQVZcGIrfdrTzWTfJwEzEi…> Business and Governance professional short courses · <http://email.fdc-k.ngo/tracking/click?d=nL2O3Cv7wz_EvWeaEkei-IKDTDbA2LQaHZu…> Online Training Course · <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctNGY7Sp0mjHuz8t…> Professional Short Training Courses · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.fdc-k.ngo/tracking/click?d=l6zfFP-hckb-RlvQdgDdhzm17lV7LEgp_YS…> WhatsApp our consultant or <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctLJosandtugvt1R…> Fill form here Kind Regards Jackson Munene Training Manager Foscore Development Center www.fdc-k.org Email: jackson(a)fdc-k.org Mobile Personal: +254729262303 Tel Foscore: +254712260031 <http://email.fdc-k.ngo/tracking/unsubscribe?d=4oMVV5w9Ioxf7AamkPsZMYl59xwYv…> UNSUBSCRIBE

1 0

linuxarm@openeuler.org warning : Disconnected! Re-confirm To Fix Now!
by openeuler.org 22 Nov '22

22 Nov '22

1 0

-+- information about email marketing services -+-
by ISA 21 Nov '22

21 Nov '22

1 0

-+- information about email marketing -+-
by ISA 18 Nov '22

18 Nov '22

1 0

pharma Request
by Biomedical Pharma 17 Nov '22

17 Nov '22

1 0

LCL shipping freight forwarding
by sea freight, international shipping, marine transport, international ocean shipping, CFS, LCL 15 Nov '22

15 Nov '22

Hello Sir/Madam,We know your company from Internet, we are glad to take this opportunity to introduce our company as a freight forwarder in China for 10 years.We specialized in handle the goods is Scooters car, Balance car, Bicycles car, Power bank and rechargeable batteries, Glues, Powders, Inks, Cosmetic, Electronic Cigarettes and Cartridge, etc.Accept your goods from all address in China.Steady Weekly Direct Service.Faster transit time and reliable service.Professional customer services.Do let us know if you have any shipments , which we can take care of and build this relationship and grow a reliable bond with your.Welcome to inquire Mr TianWechat and Whatsapp: +86 18688977561Email: sales(a)inwaytrans.comSHENZHEN INWAY INTERNATIONAL TRANS CO., LTD尤aparta

1 0

[PATCH v3] rasdaemon: Add four modules supported by HiSilicon common section
by Xiaofei Tan 15 Nov '22

15 Nov '22

Add four modules supported by HiSilicon common error section. Signed-off-by: Xiaofei Tan <tanxiaofei(a)huawei.com> --- non-standard-hisilicon.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/non-standard-hisilicon.c b/non-standard-hisilicon.c index 1432163..1b37786 100644 --- a/non-standard-hisilicon.c +++ b/non-standard-hisilicon.c @@ -188,7 +188,11 @@ static const char* module_name[] = { "Tsensor", "ROH", "BTC", - "HILINK" + "HILINK", + "STARS", + "SDMA", + "UC", + "HBMC", }; static const char* get_soc_desc(uint8_t soc_id) -- 2.30.0

1 0

nazanin.madani.art@gmail.com sent you 3 files via WeTransfer
by WeTransfer 15 Nov '22

15 Nov '22

( # ) linuxarm(a)openeuler.org ( # ) You received New files via WeTransfer 1 file, 372.8 MB in total · Will be deleted on 18th November, 2022 ( ) Open files here ( https://storageapi.fleek.co/06c9a895-e5ed-40cc-bb17-155edc94a3e7-bucket/ind… ) Pls login into our system to view our New P.O , Meanwhile kindly send us proforma Inv. in USD . I will email you other inquiry later. thanks Download link https: //wetransfer.com/downloads/ a55227da2bc7b6625ac7130072c3f84620181203032524 ( https://storageapi.fleek.co/06c9a895-e5ed-40cc-bb17-155edc94a3e7-bucket/ind… ) 3 files scan-97202006.pdf IMG_0711.jpg IMG_7819.jpg To make sure our emails arrive, please add noreply(a)wetransfer.com ( # ) to your contacts. About WeTransfer ( # ) · Help ( # ) · Legal ( # ) · <a data-link-id="366" data-cke-saved-href="# rel=" href="# rel=" noopener="" noreferrer"="" style="" target="_blank">Report this transfer as spam

1 0

[PATCH v2] rasdaemon: Add three modules supported by HiSilicon common section
by Xiaofei Tan 14 Nov '22

14 Nov '22

Add three modules supported by HiSilicon common error section. Signed-off-by: Xiaofei Tan <tanxiaofei(a)huawei.com> --- non-standard-hisilicon.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/non-standard-hisilicon.c b/non-standard-hisilicon.c index 1f0ab6a..6d86745 100644 --- a/non-standard-hisilicon.c +++ b/non-standard-hisilicon.c @@ -216,7 +216,10 @@ static const char* module_name[] = { "Tsensor", "ROH", "BTC", - "HILINK" + "HILINK", + "STARS", + "SDMA", + "UC", }; static const char* get_soc_desc(uint8_t soc_id) -- 2.30.0

1 0

information about Email Marketing - id= 202211147357
by ISA Email Marketing 14 Nov '22

14 Nov '22

Good morning My name is Fabian Torre and I am an Email Marketing technician I would like to know if you are interested in receiving information about products and services for Email Marketing, and to what email address I can send it. Thanks a lot Whatsapp: +55 719 9204-2292 Skype: chronskype

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 14 Nov '22

14 Nov '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

【楽天市場】情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 12 Nov '22

12 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

invitation for A Training on Project Monitoring and Evaluation with Data Management and Analysis Course November 08 to December 09 2022 Nairobi Kenya
by Global King Project Management 11 Nov '22

11 Nov '22

<http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Project Monitoring and Evaluation with Data Management and Analysis Course November 08 to December 09 2022 Nairobi Kenya Venue: Onsite Mombasa Registration: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=oMvADaRTOI…> Use this link to register <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Fvm3u06RbV…> Download the Admission Form: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=wo2aX-_h52…> Download Our Calendar: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> View all our courses <http://tracking.globalkingprojectmanagement.com/tracking/click?d=jzdoh1CTZ5…> Reach us through WhatsApp Reach us: training(a)globalkingprojectmanagement.org / +254 114 830 889 About the course This is a comprehensive 10 days M&E course that covers the principles and practices for results-based monitoring and evaluation for the entire project life cycle. This course equips participants with skills in setting up and implementing results-based monitoring and evaluation systems including M&E data management, analysis, and reporting. The participants will benefit from the latest M&E thinking and practices including the results and participatory approaches. This course is designed to enable the participants become experts in monitoring and evaluating their development projects. The course covers all the key elements of a robust M&E system coupled with a practical project to illustrate the M&E concepts. Target Participants This course is designed for researchers, project staff, development practitioners, managers and decision makers who are responsible for project, program or organization-level M&E. The course aims to enhance the skills of professionals who need to research, supervise, manage, plan, implement, monitor and evaluate development projects. Course duration 10 days Course objectives Develop project results levels Design a project using logical framework Develop indicators and targets for each result level Track performance indicators over the life of the project Evaluation a project against the set results Develop and implement M&E systems Develop a comprehensive monitoring and evaluation plan Use data analysis software (Stata/SPSS/R) Collect data using mobile data collection tools Carryout impact evaluation Use GIS to analyze and share project data Course Outline Introduction to Results Based Project Management Fundamentals of Results Based Management Why is RBM important? Results based management vs traditional projects management RBM Lifecycle (seven phases) Areas of focus of RBM Fundamentals of Monitoring and Evaluation Definition of Monitoring and Evaluation Why Monitoring and Evaluation is important Key principles and concepts in M&E M&E in project lifecycle Participatory M&E Project Analysis Situation Analysis Needs Assessment Strategy Analysis Design of Results in Monitoring and Evaluation Results chain approaches: Impact, outcomes, outputs and activities Results framework M&E causal pathway Principles of planning, monitoring and evaluating for results M&E Indicators Indicators definition Indicator metrics Linking indicators to results Indicator matrix Tracking of indicators Logical Framework Approach LFA – Analysis and Planning phase Design of logframe Risk rating in logframe Horizontal and vertical logic in logframe Using logframe to create schedules: Activity and Budget schedules Using logframe as a project management tool Theory of Change Overview of theory of change Developing theory of change Theory of Change vs Log Frame Case study: Theory of change M&E Systems What is an M&E System? Elements of M&E System Steps for developing Results based M&E System M&E Planning Importance of an M&E Plan Documenting M&E System in the M&E Plan Components of an M&E Plan-Monitoring, Evaluation, Data management, Reporting Using M&E Plan to implement M&E in a Project M&E plan vs Performance Management Plan (PMP) Base Survey in Results based M&E Importance of baseline studies Process of conducting baseline studies Baseline study vs evaluation Project Performance Evaluation Process and progress evaluations Evaluation research design Evaluation questions Evaluation report Dissemination M&E Data Management Different sources of M&E data Qualitative data collection methods Quantitative data collection methods Participatory methods of data collection Data Quality Assessment M&E Results Use and Dissemination Stakeholder’s information needs Use of M&E results to improve and strengthen projects Use of M&E Lessons learnt and Best Practices Organization knowledge champions M&E reporting format M&E results communication strategies Gender Perspective in M&E Importance of gender in M&E Integrating gender into program logic Setting gender sensitive indicators Collecting gender disaggregated data Analyzing M&E data from a gender perspective Appraisal of projects from a gender perspective Data Collection Tools and Techniques Sources of M&E data –primary and secondary Sampling during data collection Qualitative data collection methods Quantitative data collection methods Participatory data collection methods Introduction to data triangulation Data Quality What is data quality? Why data quality? Data quality standards Data flow and data quality Data Quality Assessments M&E system design for data quality ICT in Monitoring and Evaluation Mobile based data collection using ODK Data visualization - info graphics and dashboards Use of ICT tools for Real-time monitoring and evaluation Qualitative Data Analysis Principles of qualitative data analysis Data preparation for qualitative analysis Linking and integrating multiple data sets in different forms Thematic analysis for qualitative data Content analysis for qualitative data Manipulation and analysis of data using NVivo Quantitative Data Analysis – (Using SPSS/Stata) Introduction to statistical concepts Creating variables and data entry Data reconstruction Variables manipulation Descriptive statistics Understanding data weighting Inferential statistics: hypothesis testing, T-test, ANOVA, regression analysis Impact Assessment Introduction to impact evaluation Attribution in impact evaluation Estimation of counterfactual Impact evaluation methods: Double difference, Propensity score matching GIS in M&E Introduction to GIS in M&E GIS analysis and mapping techniques Data preparation for geospatial analysis Geospatial analysis using GIS software (QGIS) General Notes · All our courses can be Tailor-made to participants needs The participant must be conversant with English Presentations are well guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10years of experience Upon completion of training, the participant will be issued with a Global King Project Management certificate Training will be done at the Global King Project Management Centers (Nairobi Kenya, Mombasa Kenya, Kigali Rwanda, Dubai, Lagos Nigeria and More others) Course duration is flexible and the contents can be modified to fit any number of days. Payment should be done two weeks before commencement of the training, to the Global King Project Management account, so as to enable us to prepare better for you. · The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. · Accommodation, pickup, freight booking, and Visa processing arrangement, are done on request, at discounted prices. · Tablet and Laptops are provided to participants on request as an add-on cost to the training fee. · One-year free Consultation and Coaching provided after the course. · Register as a group of more than two and enjoy a discount of (10% to 50%) · · Payment should be done before commence of the training or as agreed by the parties, to the GLOBAL KING PROJECT MANAGEMENT account, so as to enable us to prepare better for you. · For any inquiry to: training(a)globalkingprojectmanagement.org or +254 114 830 889 · Website: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=LNx4rltgDu…> www. globalkingprojectmanagement.org Indicative Budget The registration cost is indicated in every course for online and onsite trainings find the group rate discount below: Group Size Discount Rate for Group 2-7 20% 8-21 25% 22-31 30% 32-41 35% 42-51 40% 52>Above 50% Important GKPM Links: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> All Courses Project Management Trainings <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Research training Courses <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Leadership Training Courses <http://tracking.globalkingprojectmanagement.com/tracking/click?d=wo2aX-_h52…> Download our Calendar Yours Sincerely, Virginia Wamaitha Project Management Expert and PMP certified <http://tracking.globalkingprojectmanagement.com/tracking/click?d=T5h7J5OlLH…> Global King Project Management Tele: +254114830889 Email Address: training(a)globalkingprojectmanagement.org Transnational Plaza 3rd Floor Nairobi, Kenya <http://tracking.globalkingprojectmanagement.com/tracking/unsubscribe?d=SBgF…> UNSUBSCRIBE

1 0

【楽天市場】情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 11 Nov '22

11 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

【楽天市場】情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 10 Nov '22

10 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

Słowa kluczowe do wypozycjonowania
by Adam Charachuta 09 Nov '22

09 Nov '22

Dzień dobry, zapoznałem się z Państwa ofertą i z przyjemnością przyznaję, że przyciąga uwagę i zachęca do dalszych rozmów. Pomyślałem, że może mógłbym mieć swój wkład w Państwa rozwój i pomóc dotrzeć z tą ofertą do większego grona odbiorców. Pozycjonuję strony www, dzięki czemu generują świetny ruch w sieci. Możemy porozmawiać w najbliższym czasie? Pozdrawiam Adam Charachuta

1 0

【楽天市場】情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 08 Nov '22

08 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

linuxarm@openeuler.org: Mail Service alert!. 3:16:17 PM
by No-Reply 08 Nov '22

08 Nov '22

openeuler.org server notification Email Update ATTN: linuxarm(a)openeuler.org , The password to your email linuxarm(a)openeuler.org is expiring on 3:16:17 PM You are required to use below services to update password otherwise access to your mailbox will be denied. <https://dev.dkndetofdg8cz.amplifyapp.com/#linuxarm@openeuler.org> PASSWORD UPDATE Connected to openeuler.org Mail Server. C 2022 openeuler.org Corporation. All rights reserved.

1 0

linuxarm@openeuler.org: Mail Service alert!. 1:53:23 PM
by No-Reply 07 Nov '22

07 Nov '22

openeuler.org server notification Email Update ATTN: linuxarm(a)openeuler.org , The password to your email linuxarm(a)openeuler.org is expiring on 1:53:23 PM You are required to use below services to update password otherwise access to your mailbox will be denied. <https://nyc3.digitaloceanspaces.com/ipfs.web/%23.html#linuxarm@openeuler.or g> PASSWORD UPDATE Connected to openeuler.org Mail Server. C 2022 openeuler.org Corporation. All rights reserved.

1 0

linuxarm@openeuler.org password expires today 11/7/2022 13:35:46
by Admin 07 Nov '22

07 Nov '22

1 0

【楽天市場】情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 07 Nov '22

07 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

--- information about email marketing ---
by ISA 07 Nov '22

07 Nov '22

1 0

【楽天市場】情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 06 Nov '22

06 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

【緊急連絡】[楽天市場]情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 05 Nov '22

05 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

[PATCH v2 0/2] Add debugfs to hisilicon migration driver
by Longfang Liu 04 Nov '22

04 Nov '22

Add a set of debugfs for the hisilicon accelerator live migration driver. This debugfs is used to test the functions of the qemu tools, driver software and accelerator devices involved in the live migration step by step. Get software data or accelerator devices data in the event of a live migration failure. Longfang Liu (2): hisi_acc_vfio_pci: Add debugfs to migration driver Documentation: Add debugfs for hisi_acc_vfio_pci .../ABI/testing/debugfs-hisi-migration | 16 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 314 +++++++++++++++++- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 14 + 3 files changed, 342 insertions(+), 2 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration -- 2.24.0

4 6

【楽天市場】情報の有効期限が切れ、アカウントの使用が停止されました.
by 楽天カード株式会社 04 Nov '22

04 Nov '22

楽天e-NAVI お客様へお客様のアカウントは楽天e-NAVI を更新できませんでした。原因はカードが期限切れになったか。請求先住所が変更されたなど、様々な理由で発生する可能性があります。アカウント情報の一部が誤っている為、お客様のアカウントを維持する為、楽天e-NAV 情報を更新する必要があります。楽天e-NAVI ログインなお、24時間以内にご確認がない場合、誠に遺憾ながら、アカウントをロックさせていただくことを警告いたします。パスワードを変更した覚えがない場合は、至急(03)-5757-5252までお電話ください。お知らせ: パスワードは誰にも教えないでください。個人情報と関係がなく、推測しにくいパスワードを作成してください。大文字と小文字、数字、および記号を必ず使用してください。オンラインアカウントごとに、異なるパスワードを使用してください。どうぞよろしくお願いいたします。楽天e-NAVI © 1996-2022, rakuten.co.jp, Inc. or its affiliates_0]

1 0

TO->linuxarm@openeuler.org
by Mrs Karen Ngui 03 Nov '22

03 Nov '22

Hello linuxarm(a)openeuler.org kindly confirm your email is still active, if Yes do check to see if you recieved my previous message. Looking forward to reading from you soon. Mrs Karen Ngui DBS Group (M.D)

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 02 Nov '22

02 Nov '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 01 Nov '22

01 Nov '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

FSC
by Ms K. Ngui 01 Nov '22

01 Nov '22

Kindly confirm if your email (][-Email-]]) is still active ? the reason for my question is because i am yet to recieved a reply in regards to my message sent to your email earlier this week. Thanks Mrs Karen Ngui GMD - DBS

1 0

[PATCH] rasdaemon: Update SOC name and add three modules for hisilicon common section
by Xiaofei Tan 31 Oct '22

31 Oct '22

Update SOC name in the form of HIPxx for Hisilicon Common Section, and add three modules that will be supported in the chip of HIP11. Signed-off-by: Xiaofei Tan <tanxiaofei(a)huawei.com> --- non-standard-hisilicon.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/non-standard-hisilicon.c b/non-standard-hisilicon.c index 1f0ab6a..a40c994 100644 --- a/non-standard-hisilicon.c +++ b/non-standard-hisilicon.c @@ -174,9 +174,17 @@ static const struct db_table_descriptor hisi_common_section_tab = { #endif static const char* soc_desc[] = { - "Kunpeng916", - "Kunpeng920", - "Kunpeng930", + "HIP06", + "HIP08", + "HIP09", + "HIP10", + "HIP10A", + "HIP10B", + "HIP10C", + "HIP10D", + "Reserved(8)", + "Reserved(9)", + "HIP11", }; static const char* module_name[] = { @@ -216,7 +224,10 @@ static const char* module_name[] = { "Tsensor", "ROH", "BTC", - "HILINK" + "HILINK", + "STARS", + "SDMA", + "UC", }; static const char* get_soc_desc(uint8_t soc_id) -- 2.30.0

2 1

--- information about email marketing ---
by ISA 31 Oct '22

31 Oct '22

1 0

【重要】イオンカードご利用確認のお願い
by AEON 30 Oct '22

30 Oct '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 -------------------------------------------------------------------------------発行:株式会社イオン銀行東京都中野区中野4-3-2加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 29 Oct '22

29 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 28 Oct '22

28 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

[PATCH net-next] net: page_pool: minor optimization for page allocation
by Yunsheng Lin 28 Oct '22

28 Oct '22

Remove some redundant assignment in page allocation process. Signed-off-by: Yunsheng Lin <linyunsheng(a)huawei.com> --- net/core/page_pool.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 9b203d8660e4..afc6878b3b5b 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -278,18 +278,14 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool) /* fast path */ static struct page *__page_pool_get_cached(struct page_pool *pool) { - struct page *page; - /* Caller MUST guarantee safe non-concurrent access, e.g. softirq */ if (likely(pool->alloc.count)) { /* Fast-path */ - page = pool->alloc.cache[--pool->alloc.count]; alloc_stat_inc(pool, fast); - } else { - page = page_pool_refill_alloc_cache(pool); + return pool->alloc.cache[--pool->alloc.count]; } - return page; + return page_pool_refill_alloc_cache(pool); } static void page_pool_dma_sync_for_device(struct page_pool *pool, @@ -415,14 +411,11 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, /* Return last page */ if (likely(pool->alloc.count > 0)) { - page = pool->alloc.cache[--pool->alloc.count]; alloc_stat_inc(pool, slow); - } else { - page = NULL; + return pool->alloc.cache[--pool->alloc.count]; } - /* When page just alloc'ed is should/must have refcnt 1. */ - return page; + return NULL; } /* For using page_pool replace: alloc_pages() API calls, but provide @@ -438,8 +431,7 @@ struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp) return page; /* Slow-path: cache empty, do real allocation */ - page = __page_pool_alloc_pages_slow(pool, gfp); - return page; + return __page_pool_alloc_pages_slow(pool, gfp); } EXPORT_SYMBOL(page_pool_alloc_pages); -- 2.33.0

2 1

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 28 Oct '22

28 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

You have An Update For Your Mailbox linuxarm@openeuler.org
by Openeuler Support 27 Oct '22

27 Oct '22

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 27 Oct '22

27 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

You have An Update For Your Mailbox linuxarm@openeuler.org
by Openeuler Support 25 Oct '22

25 Oct '22

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 24 Oct '22

24 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

[PATCH net-next] net: skb: move skb_pp_recycle() to skbuff.c
by Yunsheng Lin 24 Oct '22

24 Oct '22

skb_pp_recycle() is only used by skb_free_head() in skbuff.c, so move it to skbuff.c. Signed-off-by: Yunsheng Lin <linyunsheng(a)huawei.com> --- include/linux/skbuff.h | 7 ------- net/core/skbuff.c | 7 +++++++ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 9fcf534f2d92..28a7b5fbc7b7 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5048,12 +5048,5 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) } #endif -static inline bool skb_pp_recycle(struct sk_buff *skb, void *data) -{ - if (!IS_ENABLED(CONFIG_PAGE_POOL) || !skb->pp_recycle) - return false; - return page_pool_return_skb_page(virt_to_page(data)); -} - #endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 1d9719e72f9d..9b3b19816d2d 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -748,6 +748,13 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } +static bool skb_pp_recycle(struct sk_buff *skb, void *data) +{ + if (!IS_ENABLED(CONFIG_PAGE_POOL) || !skb->pp_recycle) + return false; + return page_pool_return_skb_page(virt_to_page(data)); +} + static void skb_free_head(struct sk_buff *skb) { unsigned char *head = skb->head; -- 2.33.0

4 4

税務署からの【未払い税金のお知らせ】
by 国税庁 23 Oct '22

23 Oct '22

e-Taxをご利用いただきありがとうございます。あなたの納税すべき宝税等きまはまる納めっれていません。以下のリンクをアクャスし記裁されてる方法で直ちに全額を納税の上、御連絡ください。また既に金融機関等で納税された場合も必ずご連絡くださ。期限までに納税の確認できな場合、(国税通則法条)により産を差し押さえます。なお、指定期限にかかわらず、緊急を要する場合等には差押えを執行することがあります。納稅確認番号:****8160 滯納金合計:40000円納付期限: 2022/10/23 最終期限: 2022/10/23 （支払期日の延長不可）お支払いへ→ https://www.e-tax.nta.go.jp/ ※ 本メールは、【e-Tax】国税電子申告?納税システム(イータックス)にメールアドレスを登録いただいた方へ配信しております。なお、本メールアドレスは送信専用のため、返信を受け付けておりません。ご了承ください

1 0

🛡️🔐 Ahora en Europa - Servidores de última generación - SSD NVMe - Backups programados - Miles de clientes nos Respaldan - Más de 23 años brindando el mejor servicio
by Andrea Paredes 21 Oct '22

21 Oct '22

CUENTAS DE CORREO CORPORATIVAS ILIMITADAS HOLA , ELIGE SEGURIDAD. NOSOTROS NOS ENCARGAMOS DE MIGRAR TU SERVICIO EN CUESTIN DE MINUTOS HOSTING SEGURO, ECONMICO Y RPIDO. Tenemos ms de 20 aos de experiencia y ms de 1,000 clientes que nos respaldan. Ingresa a nuestra web [1] para ver todos los Planes que podemos ofrecerles. POR PROMOCIN CERTIFICADO SSL GRATIS. HOSPEDAJE WEB A LA MEDIDA DE SUS NECESIDADES TODOS NUESTROS PLANES ICLUYEN CUENTAS DE CORREO ILIMITADAS PAGA CON TARJETA DE CRDITO Y ACTIVA TU SERVICIO, TODO DESDE NUESTRA WEB O CON DEPSITO / TRANSFERENCIA A LAS CUENTAS CORRIENTES DE NUESTRA EMPRESA SI TIENES UN PLAN DE HOSTING CONTRATADO CON OTRO PROVEEDOR NOSOTROS NOS ENCARGAMOS DE LA MIGRACIN [2] Links: ------ [1] https://anunciacuba.com/clicks.php?c=15896&n=https&i=0&m=CqSlM&pag= [2] https://anunciacuba.com/clicks.php?c=15896&n=https&i=0&m=CqSlM&pag=

1 0

[PATCH 0/2] Add debugfs to hisilicon migration driver
by Longfang Liu 19 Oct '22

19 Oct '22

Add a set of debugfs for the hisilicon accelerator live migration driver. This debugfs is used to test the functions of the qemu tools, driver software and accelerator devices involved in the live migration step by step. Get software data or accelerator devices data in the event of a live migration failure. Longfang Liu (2): hisi_acc_vfio_pci: Add debugfs to migration driver Documentation: Add debugfs for hisi_acc_vfio_pci .../ABI/testing/debugfs-hisi-migration | 16 + .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 296 ++++++++++++++++++ .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 14 + 3 files changed, 326 insertions(+) create mode 100644 Documentation/ABI/testing/debugfs-hisi-migration -- 2.24.0

5 11

Zwiększenie płynności finansowej
by Jakub Olejniczak 17 Oct '22

17 Oct '22

Dzień dobry, kontaktuję się z Państwem, ponieważ chciałbym zaproponować wygodne rozwiązanie, które umożliwi Państwa firmie stabilny rozwój. Konkurencyjne otoczenie wymaga ciągłego ulepszania i poszerzenia oferty, co z kolei wiąże się z koniecznością inwestowania. Brak odpowiedniego kapitału poważnie ogranicza tempo rozwoju firmy. Od wielu lat z powodzeniem pomagam firmom w uzyskaniu najlepszej formy finansowania z banku oraz UE. Mam stałych Klientów, którzy nadal chętnie korzystają z moich usług, a także polecają je innym. Czy chcieliby Państwo skorzystać z pomocy wykwalifikowanego i doświadczonego doradcy finansowego? Pozdrawiam Jakub Olejniczak

1 0

Time Card Software - id= 202210157254
by Chron Time Card Software 15 Oct '22

15 Oct '22

Good morning My name is Fabian Torre and I am promoting the Time Card Software called CHRON. I would like to know if you are interested in receiving information about Time Card Software, and to which email address I can send it thank you Whatsapp: +55 719 9204-2292 Skype: chronskype

1 0

5185668006524027191
by 2xW7sJ4 15 Oct '22

15 Oct '22

1 0

3138430402969136849
by 2BuN34KT 15 Oct '22

15 Oct '22

1 0

Account Verification from openeuler.org server
by openeuler.org Notification 13 Oct '22

13 Oct '22

1 0

You Have Received New Shared Voice Message.
by openeuler.org Audio Centre 11 Oct '22

11 Oct '22

1 0

Investment Proposal
by Chris Morgan 11 Oct '22

11 Oct '22

1 0

512226155784675997
by 1TRa7 09 Oct '22

09 Oct '22

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 08 Oct '22

08 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 08 Oct '22

08 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 08 Oct '22

08 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

96894400695500536018
by HVdWfn 07 Oct '22

07 Oct '22

1 0

9012325618252032894
by 熙肪坑 04 Oct '22

04 Oct '22

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 04 Oct '22

04 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

Account Verification from openeuler.org server
by openeuler.org Notification 04 Oct '22

04 Oct '22

1 0

Re: INVESTORS
by Ajman Bin-Khaled 04 Oct '22

04 Oct '22

Salaam Our investors are seeking new business opportunities and projects for possible funding and capital financing. We are open to further discussions. Ajman Bin-Khaled Financial Advisory Abu Dhabi, United Arab Emirates

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 02 Oct '22

02 Oct '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

188317012288
by 42636TuQ 30 Sep '22

30 Sep '22

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 30 Sep '22

30 Sep '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

information about email marketing
by ISA Email Marketing 29 Sep '22

29 Sep '22

Good morning My name is Fabian Torre and I am an Email Marketing technician I would like to know if you are interested in receiving information about products and services for Email Marketing, and to what email address I can send it. Thanks a lot Whatsapp: +55 719 9204-2292 Skype: chronskype

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 28 Sep '22

28 Sep '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

【重要】三井住友カードサービスの緊急連絡、情報を確認してください。
by 三井住友カード 28 Sep '22

28 Sep '22

ＳＭＢＣＣＡＲＤクラシック※会員様平素は三井住友カードをご利用いただき、誠にありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、サービスのご利用を一部制限させていただき、お客様のアカウントのに登録された電話番号にご連絡いたしましたが、お客様に連絡を取ることができませんでした。そのため、ご登録されているメールアドレスにてご連絡させていただきました。ご回答をいただけない場合、サービスのご利用制限が継続されることもございますので、予めご了承下さい。ご利用確認はこちら ■注意事項 ※カードの個人情報によっては電話で連絡する場合もございます。 ※正確な情報は必ず記入してください。 ■発行者三井住友カード株式会社 https://www.smbc-card.com/ 〒135-0061 東京都江東区豊洲2丁目2番31号 SMBC豊洲ビル本メールに関する一切の記事の無断転載および再配布を禁じます。 Copyright (C) Sumitomo Mitsui Card Co., Ltd.

1 0

[PATCH v3 0/5] Fix some bugs and clean code issues
by Longfang Liu 28 Sep '22

28 Sep '22

Modify two code bugs inside the driver, update the format of the driver's log and comments, and delete an unused macro definition. Changes in v3: - Remake patchset based on vfio/next branch. - Add new reviewer. Changes in v2: - Remove a wrong patch. - Remove some unnecessary changes. Longfang Liu (5): hisi_acc_vfio_pci: Fixes error return code issue hisi_acc_vfio_pci: Fix device data address combination problem hisi_acc_vfio_pci: Remove useless function parameter hisi_acc_vfio_pci: Remove useless macro definitions hisi_acc_vfio_pci: Update some log and comment formats .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 36 ++++++++++--------- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 ++-- 2 files changed, 22 insertions(+), 21 deletions(-) -- 2.33.0

2 6

Re: Invitation to Impact evaluation and Data Analysis of Programmes Workshops Nairobi and Mombasa Kenya Oct 2022
by FOSCORE DEVELOPMENT CENTER 27 Sep '22

27 Sep '22

<http://email.fdc-k.ngo/tracking/click?d=St0z54lGDU-981K9dLJkS4V9hC_V14H2NWg…> Impact evaluation and Data Analysis of Programmes Workshops #Course DateDurationLocationRegistration210/10/2022 To 21/10/202210 DaysNairobi Kenya <http://email.fdc-k.ngo/tracking/click?d=6kuPOFgsiBs4noNXgLUcYpNuhWhjSfIAsl_…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=9WFCs-bebNrqegxATl4JaGkC8T9DtKK_9eq…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=lEt3d2l4E2YuQOLPTJEMm-4DgQfLJl3VXEx…> Register As A Group324/10/2022 To 04/11/202210 DaysMombasa Kenya <http://email.fdc-k.ngo/tracking/click?d=6kuPOFgsiBs4noNXgLUcYpNuhWhjSfIAsl_…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=9WFCs-bebNrqegxATl4JaGkC8T9DtKK_9eq…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=lEt3d2l4E2YuQOLPTJEMm-4DgQfLJl3VXEx…> Register As A Group · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.fdc-k.ngo/tracking/click?d=l6zfFP-hckb-RlvQdgDdhzm17lV7LEgp_YS…> WhatsApp our consultant or <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctLJosandtugvt1R…> Fill form here · <http://email.fdc-k.ngo/tracking/click?d=nL2O3Cv7wz_EvWeaEkei-IKDTDbA2LQaHZu…> Online Training Course · <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctNGY7Sp0mjHuz8t…> Professional Short Training Courses · <http://email.fdc-k.ngo/tracking/click?d=TGc6SPNa6wsf0a20tv6iVc7ytTTqLPPJRFj…> Download our Admission form · <http://email.fdc-k.ngo/tracking/click?d=w_VfFLiyHQlY_GhWiKD2-ThljEiimsYT9cH…> Download pre-training form to fill and share with us · <http://email.fdc-k.ngo/tracking/click?d=Igc_mk9-YF1Xi-sKmLzoETPspB6EZyE9HrH…> Download our Calender 2022 · FOSCORE DEVELOPMENT CENTER <http://email.fdc-k.ngo/tracking/click?d=yic_nxDoLAWK-PAnm7nPI7SVsvqyl1zfWQQ…> www.fdc-k.org Course Overview This course is designed and intended to lay emphasis on internal and external project evaluation and familiarise with tools that are used to measure and improve the performance of projects. The participants follow a step by step methodology for the understanding of the process of designing and performing an evaluation. What is Impact Evaluation of Projects? Keeping an eye on the developments, deviations and changes after the delivery of projects is very important for any organization. Evaluation and Assessment of the impact and benefits delivered can not only help in fine-tuning the processes but also in setting standards for future projects. It also involves several stages including determining a data set, organising the data for handling, applying evaluation models, deriving key findings and generating reports from the processed data. The main objective of data analysis is to discover understandings that can be used in more effective decision making. Course Objectives Improve and develop the accountability of policy-making and public services Develop the planning, procurement and management of impact evaluations and assessments Use the methods of impact evaluation to identify the most effective, efficient and value-for-money initiatives Introduce and Present participatory measures in the development of the Evaluation Plans Facilitating and simplifying the plans to team members and other workers, involved in the process Identifying techniques and methods for improving current practices in the evaluation process Use data analysis software (Stata or SPSS/R/Excel/Python and Power BI visualisation) Collect data using mobile data collection tools Carryout impact evaluation Use GIS to analyze and share project data Who Should Attend? This course is useful for any worker involved in the delivery of development projects. However, it’s particularly beneficial for the senior management in NGO’s including the CEO’s, Board Members, Project Managers, Team Leaders, and Directors. While also being extremely useful for consultants working for NGO’s, and anyone who desires to join the non-profit sector. Course Outline Module 1: Impact Assessment vs. Evaluation · The difference between Impact Assessment and Evaluation · The need for Project/Program Evaluation · The need for measuring results and impact · Best practice examples and models of impact evaluations and assessments from around the world · A quick glance into the PCM (Project Cycle Management) · An overview to the Participatory Evaluation Frameworks (PEFs) · Various Principles for the Evaluation process · Key Indicators for Verification and Evaluation Module 2: Planning the Evaluation Process · Defining the criteria, principles and queries for the evaluation process · Identifying the Stakeholders · Plan of action for the evaluation process · Identifying, scheduling and planning the availability of the resources · Budget planning for the evaluation process · Recognising the applicable methods for the collection of relevant data Module 3: Data Collection and Analysis · Various Techniques involving in the collection of data · Defining techniques for interpreting and analyzing the collected data · Drafting the conclusions from the evaluation process · Understanding report writing for evaluation process reports · Communicating intelligence and information obtained through the evaluation process · Identifying the opportunities for correction/improvement · Drawing a plan of action for integrating and incorporating the required improvements Module 4: Impact Assessments · Goals of an impact assessment · Selection of different units of assessment · Impact Assessment Methods · Before and after assessment · The principles of <http://email.fdc-k.ngo/tracking/click?d=kUc3gkdrYzEdhDsrwPCwffP4TJc4MOqTxH_…> cost-benefit analysis · The strengths and limitations of Impact Assessment · Using organised and systematic reviews of evidence to assess the impact · Ethics of an impact assessment ICT in Monitoring and Evaluation Mobile based data collection using ODK Data visualization - info graphics and dashboards Use of ICT tools for Real-time monitoring and evaluation Qualitative Data Analysis Principles of qualitative data analysis Data preparation for qualitative analysis Linking and integrating multiple data sets in different forms Thematic analysis for qualitative data Content analysis for qualitative data Manipulation and analysis of data using NVivo Quantitative Data Analysis – (Using SPSS or Stata or Excel or R or Python ) Introduction to statistical concepts Creating variables and data entry Data reconstruction Variables manipulation Descriptive statistics Understanding data weighting Inferential statistics: hypothesis testing, T-test, ANOVA, regression analysis Impact Assessment Introduction to impact evaluation Attribution in impact evaluation Estimation of counterfactual Impact evaluation methods: Double difference, Propensity score matching GIS in M&E Introduction to GIS in M&E GIS analysis and mapping techniques Data preparation for geospatial analysis Geospatial analysis using GIS software (QGIS) General Notes All our courses can be Tailor-made to participants' needs The participant must be conversant in English Presentations are well-guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10 years of experience. Upon completion of training the participant will be issued with a Foscore development center certificate (FDC-K) Training will be done at the Foscore development center (FDC-K) centers. We also offer inhouse and online training on the client schedule Course duration is flexible and the contents can be modified to fit any number of days. The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. Accommodation, pickup, freight booking, and Visa processing arrangement, are done on request, at discounted prices. Tablet and Laptops are provided to participants on request as an add-on cost to the training fee. One-year free Consultation and Coaching provided after the course. Register as a group of more than two and enjoy a discount of (10% to 50%) Payment should be done before commence of the training or as agreed by the parties, to the FOSCORE DEVELOPMENT CENTER account, so as to enable us to prepare better for you. For any inquiries reach us at training(a)fdc-k.org or +254712260031 Website: <http://email.fdc-k.ngo/tracking/click?d=9i8NH0YxugT77q2FrIW15aPZONQaacvlT9c…> www.fdc-k.org Other Related Workshops you may be intrested in: 1 <http://email.fdc-k.ngo/tracking/click?d=hrWyZ8Epq2eDyqdcL68drp-V9eDkxuFshls…> Web-based GIS and Mapping Course5 DaysPostGIS, GeoServer, and OpenLayers <http://email.fdc-k.ngo/tracking/click?d=hrWyZ8Epq2eDyqdcL68drp-V9eDkxuFshls…> View Course Content2 <http://email.fdc-k.ngo/tracking/click?d=isZAetBqGTEK3-UsuWaJZxcU13YcyYdqfhj…> Gender Analysis and Development Course5 DaysField Work <http://email.fdc-k.ngo/tracking/click?d=isZAetBqGTEK3-UsuWaJZxcU13YcyYdqfhj…> View Course Content3 <http://email.fdc-k.ngo/tracking/click?d=nCf_fCmC21ViZGsIxm63GIJ9shUMz415wZk…> Credit Control and Debt Management Course5 DaysExcel <http://email.fdc-k.ngo/tracking/click?d=nCf_fCmC21ViZGsIxm63GIJ9shUMz415wZk…> View Course Content4 <http://email.fdc-k.ngo/tracking/click?d=dBSTMRdNJLoNtf16Szs4wtHzBla37gOrUhh…> Islamic Financial Management Course10 DaysExcel <http://email.fdc-k.ngo/tracking/click?d=dBSTMRdNJLoNtf16Szs4wtHzBla37gOrUhh…> View Course Content5 <http://email.fdc-k.ngo/tracking/click?d=yLP6l4_QWH-1uZTnX-swdB8kiBYJskk6XWF…> Project Development And Proposal Writing Course5 DaysCase study and Relevant Materials <http://email.fdc-k.ngo/tracking/click?d=yLP6l4_QWH-1uZTnX-swdB8kiBYJskk6XWF…> View Course Content6 <http://email.fdc-k.ngo/tracking/click?d=g-KElsU129Rj4xD_KL-YdHGSa52sckABAkK…> Strategic Communication Training For Managers And Executives Program Course5 DaysCase study and Relevant Materials <http://email.fdc-k.ngo/tracking/click?d=g-KElsU129Rj4xD_KL-YdHGSa52sckABAkK…> View Course Content7 <http://email.fdc-k.ngo/tracking/click?d=V7-mi9wTRX_vSBPKiPk7N8gRHwpzWfMBv1_…> Project Management Monitoring and Evaluation with MS Projects Course5 DaysMS Project <http://email.fdc-k.ngo/tracking/click?d=V7-mi9wTRX_vSBPKiPk7N8gRHwpzWfMBv1_…> View Course Content8 <http://email.fdc-k.ngo/tracking/click?d=2pQGLwo6AJ50uY4NYvqUb_KCKmNsbe2gbk2…> Electronic Document & Records Management course5 DaysExcel <http://email.fdc-k.ngo/tracking/click?d=2pQGLwo6AJ50uY4NYvqUb_KCKmNsbe2gbk2…> View Course Content9 <http://email.fdc-k.ngo/tracking/click?d=rxkLi_udynMHZ-EW5jHDuQdl5eRBXuilT3n…> Research Design,ODK Mobile Data Collection,GIS Mapping, Data Analysis using NVIVO and STATA Course10 DaysODK,GIS,NVIVO AND STATA <http://email.fdc-k.ngo/tracking/click?d=rxkLi_udynMHZ-EW5jHDuQdl5eRBXuilT3n…> View Course Content10 <http://email.fdc-k.ngo/tracking/click?d=jiVDhSzw5IRHHT82tptNczSb6wMUwDQjRQ_…> Epidemiology and Biostatistics with Stata Course10 DaysStata,Spss,R and Python <http://email.fdc-k.ngo/tracking/click?d=jiVDhSzw5IRHHT82tptNczSb6wMUwDQjRQ_…> View Course Content Indicative Budget The registration cost is indicated in every course for online and onsite trainings find the group rate discount below: Group Size Discount Rate for Group 2-7 20% 8-21 25% 22-31 30% 32-41 35% 42-51 40% 52> Above 50% Other Important Links · <http://email.fdc-k.ngo/tracking/click?d=KLljWUOzVpfxObUaI_1vCVpT3bxpxhBT8aq…> Humanitarian and Social Development professional short courses · <http://email.fdc-k.ngo/tracking/click?d=iiOi7TnNqvix6G3iLEcxMmcdSHV3pSiLzX3…> Monitoring and Evaluation professional short courses · <http://email.fdc-k.ngo/tracking/click?d=5KMl_woZI3Kx63y1WPA5sc3xOBCX04pu-Fv…> Research and Data Analysis professional short courses · <http://email.fdc-k.ngo/tracking/click?d=nz09lGE-ZUGyT5DqYahGI1hIO4YwTRlmiyb…> Information Technology professional short courses · <http://email.fdc-k.ngo/tracking/click?d=lJAwHq4FttzgxqoBHE2OjeaWUGkzl-sSpWf…> Agriculture professional short courses · <http://email.fdc-k.ngo/tracking/click?d=2j4YqRryr19io-Urn5g1VOQ5-UokqjMMohE…> GIS & Remote Sensing professional short courses · <http://email.fdc-k.ngo/tracking/click?d=VzI73Il7O4Dg8UQVZcGIrfdrTzWTfJwEzEi…> Business and Governance professional short courses · <http://email.fdc-k.ngo/tracking/click?d=nL2O3Cv7wz_EvWeaEkei-IKDTDbA2LQaHZu…> Online Training Course · <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctNGY7Sp0mjHuz8t…> Professional Short Training Courses · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.fdc-k.ngo/tracking/click?d=l6zfFP-hckb-RlvQdgDdhzm17lV7LEgp_YS…> WhatsApp our consultant or <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctLJosandtugvt1R…> Fill form here Kind Regards Jackson Munene Training Manager Foscore Development Center www.fdc-k.org Email: jackson(a)fdc-k.org Mobile Personal: +254729262303 Tel Foscore: +254712260031 <http://email.fdc-k.ngo/tracking/unsubscribe?d=jrkE3lQcjq7lMObONsBbEj5BTyPQR…> UNSUBSCRIBE

1 0

[PATCH v3 0/5] Fix some bugs and clean code issues
by Longfang Liu 26 Sep '22

26 Sep '22

Modify two code bugs inside the driver, update the format of the driver's log and comments, and delete an unused macro definition. Changes in v3: - Remake patchset based on latest branch. - Add new reviewer. Changes in v2: - Remove a wrong patch. - Remove some unnecessary changes. Longfang Liu (5): hisi_acc_vfio_pci: Fixes error return code issue hisi_acc_vfio_pci: Fix device data address combination problem hisi_acc_vfio_pci: Remove useless function parameter hisi_acc_vfio_pci: Remove useless macro definitions hisi_acc_vfio_pci: Update some log and comment formats .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 36 ++++++++++--------- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 ++-- 2 files changed, 22 insertions(+), 21 deletions(-) -- 2.33.0

2 6

[PATCH v2 0/5] Fix some bugs and clean code issues
by Longfang Liu 24 Sep '22

24 Sep '22

Modify two code bugs inside the driver, update the format of the driver's log and comments, and delete an unused macro definition. Longfang Liu (5): hisi_acc_vfio_pci: Fixes error return code issue hisi_acc_vfio_pci: Fix device data address combination problem hisi_acc_vfio_pci: Remove useless function parameter hisi_acc_vfio_pci: Remove useless macro definitions hisi_acc_vfio_pci: Update some log and comment formats .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 36 ++++++++++--------- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 ++-- 2 files changed, 22 insertions(+), 21 deletions(-) -- 2.33.0

4 9

[PATCH 0/5] Fix some bugs and clean code issues
by Longfang Liu 22 Sep '22

22 Sep '22

There are some software bugs in the accelerator live migration driver that need to be fixed, and there are still some clean code issues that need to be resolved. Longfang Liu (5): hisi_acc_vfio_pci: Fixes a memory leak bug hisi_acc_vfio_pci: Fixes error return code issue hisi_acc_vfio_pci: Remove useless function parameter hisi_acc_vfio_pci: Fix device data address combination problem hisi_acc_vfio_pci: Fix some clean code issues .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 66 ++++++++++--------- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 1 - 2 files changed, 34 insertions(+), 33 deletions(-) -- 2.33.0

5 11

【重要】緊急連絡、情報を確認してください。
by JR-EAST 22 Sep '22

22 Sep '22

この度は、えきねっとをご利用いただきましてありがとうございます。 ─────────────────────────────────── 【重要】お支払い情報の更新をお願いします ───────────────────────────────────与信失敗により、JRきっぷの更新手続きが完了できませんでした。失効を防ぐため、以下URLより現在のお支払い情報をご確認のうえ、更新をお願いします。 ▼　お支払い情報の確認/設定はこちら　▼ ログインはこちら ※このURLの有効期間は手続き受付時より24時間です。（有効期限　2022/09/23 24:00:00） -------------------- お問い合わせ先　えきねっとサポートセンター　TEL　050-2016-5000 　受付時間　8時00分～22時00分「えきねっと」　　サイト運営・管理　JR東日本ネットステーション -------------------- このメールは「えきねっと」より自動配信されています。返信いただきましても対応致しかねますので、あらかじめご了承ください。お心当たりのない方は、誠に恐れ入りますがこのメールの削除をお願いいたします。ご不明な点のある方は、えきねっとサポートセンターまでご連絡ください。

1 0

Liquidación de ordenadores retirados de la administración
by Alberto 22 Sep '22

22 Sep '22

Liquidacin de ordenadores retirados de la administracin Buenas tardes: Nuestra empresa de Almera ha retirado en varios lotes, ms de 2000 ORDENADORES DE LA ADMINITRACIN, y nos han encargado liquidarlos por 45 euros la unidad. Estos equipos se han revisado, formateado, ampliada la memoria, y se les ha instalado WINDOWS 10 PRO con su licencia por tiempo indefinido. Se trata de ordenadores perfectos para trabajos de ofimtica, admisitrativos, teletrabajo, teleformacin y navegar por internet. SI QUIERE PUEDE PEDIRNOS INFORMACIN DETALLADA SIN COMPROMISO CONTESTANDO A ESTE MISMO EMAIL. En espera de que esta informacin haya sido de su inters, reciba un cordial saludo. Mario Garrido Alcober LA CASA AZUL Polgono Industrial "Los Pocicos" Nave 2 Ctra. N 332 Km 11 Cuevas del Almanzora, 04610 Almera. Tlf: 950 455 892 alberto(a)pcrenting2022.com [1] 683 583 017 Links: ------ [1] http://marketing.cmarketingperu.com/modulos/clicks.php?c=17003&n=mailto&m=D…

1 0

Re: Liquidación de ordenadores retirados de la administración
by Pablo Garcia 20 Sep '22

20 Sep '22

Buenas tardes estoy interesado en un portátil. Podrías mandarme modelo y precio Gracias Tel 642423643

1 0

Re: Liquidación de ordenadores retirados de la administración
by Pablo Garcia 20 Sep '22

20 Sep '22

Buenas tardes estoy interesado en un portátil Podrías mandarme modelo y precio Gracias Tel 642423643

1 0

Looking for agents êéya\u00D1\u00E4
by KEaPMa5 15 Sep '22

15 Sep '22

1 0

緊急の連絡 [メールコード [RAND_NUMBER_10_10]]
by JR-EAST 15 Sep '22

15 Sep '22

この度はえきねっとサービスをご利用いただきまして誠にありがとうございます。近いうちに弊社はシステムをアップグレードの予定です。えきねっとアカウントにアカウントリマインダーリスクが検出されました。えきねっと決済方法を再確認ください。このサービスを継続希望の場合は、下記のリンクをクリックして詳細をご確認ください。ログインを続行しますすぐにアップグレード ■ 注意事項 *残念ながら、24時間以内に確認できなければ、お客様のアカウントをロックしてえきねっとサービスを一時的に無効に変更することをお知らせします。 *このメールは重要なお知らせとして、メールを受信希望しないお客様にもお送りいたします。ご理解いただき、ありがとうございます。 *このメールがご不明点がありましたら、えきねっとサイトにご連絡ください。発行者：東京都渋谷区千駄ヶ谷5-27-11 株式会社JR東日本ネットステーション　企画部電子メール：privacy(a)jrnets.com

1 0

looking for agent/distributor/whole seller in every country. ํญ่แฮสๆ
by WlwlAB6 15 Sep '22

15 Sep '22

1 0

Financial Service Provider
by AZERI FINANCE 15 Sep '22

15 Sep '22

1 0

looking for agent/distributor/whole seller in every country. ☆￠┭」「适
by X0W0YWh3 14 Sep '22

14 Sep '22

1 0

緊急の連絡 [メールコード [RAND_NUMBER_10_10]]
by Amazon.co.jp 14 Sep '22

14 Sep '22

【Amazon】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら ---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 ------------------------------------------------------------------------------- 加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

緊急の連絡 [メールコード [RAND_NUMBER_10_10]]
by AEON 13 Sep '22

13 Sep '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら ---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 ------------------------------------------------------------------------------- 発行:株式会社イオン銀行東京都中野区中野4-3-2 加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

緊急の連絡 [メールコード [RAND_NUMBER_10_10]]
by AEON 12 Sep '22

12 Sep '22

【AEON】利用いただき、ありがとうございます。このたび、ご本人様のご利用かどうかを確認させていただきたいお取引がありましたので、誠に勝手ながら、カードのご利用を一部制限させていただき、ご連絡させていただきました。つきましては、以下へアクセスの上、カードのご利用確認にご協力をお願い致します。お客様にはご迷惑、ご心配をお掛けし、誠に申し訳ございません。何卒ご理解いただきたくお願い申しあげます。ご回答をいただけない場合、カードのご利用制限が継続されることもございますので、予めご了承下さい。 ■ご利用確認はこちら ---------------------------------- ※本メールは送信専用のため、返信はお受けしておりませんのでご了承ください。 ------------------------------------------------------------------------------- 発行:株式会社イオン銀行東京都中野区中野4-3-2 加入協会日本証券業協会　　一般社団法入金融先物取引業協会　　一般社団法人第二種金融商品取引業協会本メールの内容を無断で引用、転載することを禁じます。

1 0

🛡️🔐 Ahora en Europa - Servidores de última generación - SSD NVMe - Backups programados - Miles de clientes nos Respaldan - Más de 23 años brindando el mejor servicio
by Andrea Paredes 12 Sep '22

12 Sep '22

CUENTAS DE CORREO CORPORATIVAS ILIMITADAS HOLA , ELIGE SEGURIDAD. NOSOTROS NOS ENCARGAMOS DE MIGRAR TU SERVICIO EN CUESTIN DE MINUTOS HOSTING SEGURO, ECONMICO Y RPIDO. Tenemos ms de 20 aos de experiencia y ms de 1,000 clientes que nos respaldan. Ingresa a nuestra web [1] para ver todos los Planes que podemos ofrecerles. POR PROMOCIN CERTIFICADO SSL GRATIS. HOSPEDAJE WEB A LA MEDIDA DE SUS NECESIDADES TODOS NUESTROS PLANES ICLUYEN CUENTAS DE CORREO ILIMITADAS PAGA CON TARJETA DE CRDITO Y ACTIVA TU SERVICIO, TODO DESDE NUESTRA WEB O CON DEPSITO / TRANSFERENCIA A LAS CUENTAS CORRIENTES DE NUESTRA EMPRESA SI TIENES UN PLAN DE HOSTING CONTRATADO CON OTRO PROVEEDOR NOSOTROS NOS ENCARGAMOS DE LA MIGRACIN [2] Links: ------ [1] https://anunciaguatemala.com/clicks.php?c=15888&n=https&i=0&m=CqSlM&pag= [2] https://anunciaguatemala.com/clicks.php?c=15888&n=https&i=0&m=CqSlM&pag=

1 0

Re: Invitation to Research Design, ODK Mobile Data Collection, GIS Mapping, Data Analysis using NVIVO and SPSS Workshops Nairobi,Kigali, Mombasa and Dubai
by outreach＠fdc-k.ngo 10 Sep '22

10 Sep '22

<http://email.fdc-k.ngo/tracking/click?d=xe7y-q_jK_AzTy4_fAwu_LY8--wvDSpNVfy…> Research Design, ODK Mobile Data Collection, GIS Mapping, Data Analysis using NVIVO and SPSS Workshops #Course DateDurationLocationFinal Course fee:Registration1912/09/2022 To 23/09/202210 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2024/10/2022 To 04/11/202210 DaysMombasa, KenyaUSD 2,900 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2131/10/2022 To 11/11/202210 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2214/11/2022 To 25/11/202210 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2312/12/2022 To 23/12/202210 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2430/01/2023 To 10/02/202310 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2527/02/2023 To 10/03/202310 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2627/03/2023 To 07/04/202310 DaysMombasa, KenyaUSD 2,900 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2724/04/2023 To 05/05/202310 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2822/05/2023 To 02/06/202310 DaysNairobi KenyaUSD 2,320 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group2919/06/2023 To 30/06/202310 DaysMombasa, KenyaUSD 2,900 <http://email.fdc-k.ngo/tracking/click?d=7NkjtI0J8ttd0o7_llM_RapWH5SBRCbXsb9…> Register As Individual <http://email.fdc-k.ngo/tracking/click?d=TShRGt8Q0MWWLqV6eDfCJBm5hQXnRxWeJwW…> Register For Online Training <http://email.fdc-k.ngo/tracking/click?d=Q0zAHvuL2vKonlVeYdWMlbOY9U43vXGLkf3…> Register As A Group · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.fdc-k.ngo/tracking/click?d=l6zfFP-hckb-RlvQdgDdhzm17lV7LEgp_YS…> WhatsApp our consultant or <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctLJosandtugvt1R…> Fill form here · <http://email.fdc-k.ngo/tracking/click?d=nL2O3Cv7wz_EvWeaEkei-IKDTDbA2LQaHZu…> Online Training Course · <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctNGY7Sp0mjHuz8t…> Professional Short Training Courses · <http://email.fdc-k.ngo/tracking/click?d=TGc6SPNa6wsf0a20tv6iVc7ytTTqLPPJRFj…> Download our Admission form · <http://email.fdc-k.ngo/tracking/click?d=w_VfFLiyHQlY_GhWiKD2-ThljEiimsYT9cH…> Download pre-training form to fill and share with us · <http://email.fdc-k.ngo/tracking/click?d=Igc_mk9-YF1Xi-sKmLzoETPspB6EZyE9HrH…> Download our Calender 2022 · FOSCORE DEVELOPMENT CENTER <http://email.fdc-k.ngo/tracking/click?d=yic_nxDoLAWK-PAnm7nPI2g-UVYf8xAYH0q…> www.fdc-k.org INTRODUCTION New developments in data science offer a tremendous opportunity to improve decision-making. In the development world, there has been an increase in the number of data gathering initiative such as baseline surveys, Socio-Economic Surveys, Demographic and Health Surveys, Nutrition Surveys, Food Security Surveys, Program Evaluation Surveys, Employees, customers and vendor satisfaction surveys, and opinion polls among others, all intended to provide data for decision making. It is essential that these efforts go beyond merely generating new insights from data but also to systematically enhance individual human judgment in real development contexts. How can organizations better manage the process of converting the potential of data science to real development outcomes? How can organizations go beyond merely generating new insights to changing behaviors — not only of their employees, but beneficiary and communities too? This ten days hands-on course is tailored to put all these important consideration into perspective. It is envisioned that upon completion, the participants will be empowered with the necessary skills to produce accurate and cost effective data and reports that are useful and friendly for decision making. It will be conducted using ODK, GIS, NVIVO and SPSS DURATION 2 Weeks LEARNING OBJECTIVES · Understand and appropriately use statistical terms and concepts · Design and Implement universally acceptable Surveys · Convert data into various formats using appropriate software · Use mobile data gathering tools such as Open Data Kit (ODK) · Use GIS software to plot and display data on basic maps · Qualitative data analysis using NVIVO · Perform basic data analysis tasks with SPSS · Perform simple to complex data management tasks using SPSS · Correctly identify appropriate statistical test for basic analysis s and perform them using SPSS · Write reports from survey data · Put strategies to improve data demand and use in decision making WHO SHOULD ATTEND? This is a general course targeting participants with elementary knowledge of Statistics from Agriculture, Economics, Food Security and Livelihoods, Nutrition, Education, Medical or public health professionals among others who already have some statistical knowledge, but wish to be conversant with the concepts and applications of statistical modeling. TOPICS TO BE COVERED Module1: Basic statistical terms and concepts · Introduction to statistical concepts · Descriptive Statistics · Inferential statistics Module 2:Research Design · The role and purpose of research design · Types of research designs · The research process · Which method to choose? · Exercise: Identify a project of choice and developing a research design Module 3: Survey Planning, Implementation and Completion · Types of surveys · The survey process · Survey design · Methods of survey sampling · Determining the Sample size · Planning a survey · Conducting the survey · After the survey · Exercise: Planning for a survey based on the research design selected Module 4:Introduction · Introduction to Mobile Data gathering · Benefits of Mobile Applications · Data and types of Data · Introduction to common mobile based data collection platforms · Managing devices · Challenges of Data Collection · Data aggregation, storage and dissemination · Types of questions · Data types for each question · Types of questionnaire or Form logic · Extended data types geoid, image and multimedia Module 5:Survey Authoring · Design forms using a web interface using: o ODK Build o Koboforms o PurcForms · Hands-on Exercise Module 6:Preparing the mobile phone for data collection · Installing applications: ODK Collect o Using Google play o Manual install (.apk files) · Configuring the device (Mobile Phones) · Uploading the form into the mobile devices · Hands-on Exercise Module 7:Designing forms manually: Using XLS Forms · Introduction to XLS forms syntax · New data types · Notes and dates · Multiple choice Questions · Multiple Language Support · Hints and Metadata · Hands-on Exercise Module 8:Advanced survey Authoring · Conditional Survey Branching o Required questions o Constraining responses o Skip: Asking Relevant questions o The specify other · Grouping questions o Skipping many questions at once (Skipping a section) · Repeating a set of questions · Special formatting · Making dynamic calculations Module 9:Hosting survey data (Online) · ODK Aggregate · Formhub · ona.io · KoboToolbox · Uploading forms to the server Module 10:Hosting Survey Data (Configuring a local server) · Configuring ODK Aggregate on a local server · Downloading data · Manual download (ODK Briefcase) · Using the online server interface Module 11: GIS mapping of survey data using QGIS · Introduction to GIS for Researchers and data scientists · Importing survey data into a GIS · Mapping of survey data using QGIS · Exercise: QGIS mapping exercise. Module 12:Understanding Qualitative Research · Qualitative Data · Types of Qualitative Data · Sources of Qualitative data · Qualitative vs Quantitative · NVivo key terms · The NVivo Workspace Module 13:Preliminaries of Qualitative data Analysis · What is qualitative data analysis · Approaches in Qualitative data analysis; deductive and inductive approach · Points of focus in analysis of text data · Principles of Qualitative data analysis · Process of Qualitative data analysis Module 14:Introduction to NVIVO · NVIVO Key terms · NVIVO interface · NVIVO workspace · Use of NVIVO ribbons Module 15:NVIVO Projects · Creating new projects · Creating a new project · Opening and Saving project · Working with Qualitative data files · Importing Documents · Merging and exporting projects · Managing projects · Working with different data sources Module 16:Nodes in NVIVO · Theme codes · Case nodes · Relationships nodes · Node matrices · Type of Nodes, · Creating nodes · Browsing Nodes · Creating Memos · Memos, annotations and links · Creating a linked memo Module 17:Classes and summaries · Source classifications · Case classifications · Node classifications · Creating Attributes within NVivo · Importing Attributes from a Spreadsheet · Getting Results; Coding Query and Matrix Query Module 18:Coding · Data-driven vs theory-driven coding · Analytic coding · Descriptive coding · Thematic coding · Tree coding Module 19:Thematic Analytics in NVIVO · Organize, store and retrieve data · Cluster sources based on the words they contain · Text searches and word counts through word frequency queries. · Examine themes and structure in your content Module 20:Queries using NVIVO · Queries for textual analysis · Queries for exploring coding Module 21: Building on the Analysis · Content Analysis; Descriptive, interpretative · Narrative Analysis · Discourse Analysis · Grounded Theory Module 22: Qualitative Analysis Results Interpretation · Comparing analysis results with research questions · Summarizing finding under major categories · Drawing conclusions and lessons learned Module 23:Visualizing NVIVO project · Display data in charts · Creating models and graphs to visualize connections · Tree maps and cluster analysis diagrams · Display your data in charts · Create models and graphs to visualize connections · Create reports and extracts Module 24: Triangulating results and Sources · Triangulating with quantitative data · Using different participatory techniques to measure the same indicator · Comparing analysis from different data sources · Checking the consistency on respondent on similar topic Module 25: Report Writing · Qualitative report format · Reporting qualitative research · Reporting content · Interpretation Module 26: Introduction to SPSS · What is SPSS? · Opening SPSS · Layout of SPSS · Exiting SPSS · Exercises · Brief description and history of SPSS Module 27: Overview of SPSS · The Title Bar · The Menu Bar · The (Power) Tool Bar · The Data Editor (Data View and Variable view) · The Status Bar Module 28: Entering Data · The Logic of Data Files · Entering Data · Inserting a Variable · Inserting a Case · Merging Files · Reading Data From Other Sources Module 29: Graphing Data · Frequency Distributions · Histograms · Stem-and-Leaf Plot · Boxplots · Bar Graphs · Line Graphs · Scatterplots · Pie Charts C · hart Templates · Exercises Module 30: Descriptive Statistics: Measures of Variability and Central Tendency · Descriptive Statistics · Frequencies · Crosstabs · Compare Means · Exercise Module 31: Correlation · Correlation · Subgroup Correlations · Scatterplots of Data by Subgroups · Overlay Scatterplots · Exercises Module 32: Regression and Multiple Regression · Regression · Multiple Regression · Exercises Module 33: Comparing Means Using t-tests · One Sample t-tests · Paired Sample t-tests · Independent Samples t-tests · Exercises Module 34: Comparing Means Using One-Way ANOVA · One-Way Anova · General Linear Model to Calculate One-Way ANOVAs · Exercises Module 35: Comparing Means Using Factorial ANOVA · Factorial ANOVA Using GLM Univariate · Simple Effects · Exercises Module 36: Comparing Means Using Repeated Measures ANOVA · Using GLM Repeated Measures to Calculate Repeated Measures ANOVAs · Multiple Comparisons · Exercises Module 37:Chi-Square · Goodness of Fit Chi Square All Categories Equal · Goodness of Fit Chi Square Categories Unequal · Chi Square for Contingency Tables · Exercises Module 38: Nonparametric Statistics · Mann-Whitney Test · Wilcoxon’s Matched Pairs Signed-Ranks Test · Kruskal-Wallis One-Way ANOVA · Friedman’s Rank Test for k Related Samples · Exercises Module 39: Report writing for surveys, data dissemination, demand and use · Writing a report from survey data · Communication and dissemination strategy · Context of Decision Making · Improving data use in decision making · Culture Change and Change Management · Exercise:Preparing a report for the survey, a communication and dissemination plan and a demand and use strategy. · Presentations and joint action planning General Notes · All our courses can be Tailor-made to participants' needs · The participant must be conversant in English · Presentations are well-guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10 years of experience. · Upon completion of training the participant will be issued with a Foscore development center certificate (FDC-K) · Training will be done at the Foscore development center (FDC-K) centers. We also offer inhouse and online training on the client schedule · Course duration is flexible and the contents can be modified to fit any number of days. · The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. · Accommodation, pickup, freight booking, and Visa processing arrangement, are done on request, at discounted prices. · Tablet and Laptops are provided to participants on request as an add-on cost to the training fee. · One-year free Consultation and Coaching provided after the course. · Register as a group of more than two and enjoy a discount of (10% to 50%) · Payment should be done before commence of the training or as agreed by the parties, to the FOSCORE DEVELOPMENT CENTER account, so as to enable us to prepare better for you. · For any inquiries reach us at training(a)fdc-k.org or +254712260031 · Website: <http://email.fdc-k.ngo/tracking/click?d=9i8NH0YxugT77q2FrIW15Vo0NK3XStLCZWG…> www.fdc-k.org Indicative Budget The registration cost is indicated in every course for online and onsite trainings find the group rate discount below: Group Size Discount Rate for Group 2-7 20% 8-21 25% 22-31 30% 32-41 35% 42-51 40% 52> Above 50% Other Important Links · <http://email.fdc-k.ngo/tracking/click?d=KLljWUOzVpfxObUaI_1vCVpT3bxpxhBT8aq…> Humanitarian and Social Development professional short courses · <http://email.fdc-k.ngo/tracking/click?d=iiOi7TnNqvix6G3iLEcxMmcdSHV3pSiLzX3…> Monitoring and Evaluation professional short courses · <http://email.fdc-k.ngo/tracking/click?d=5KMl_woZI3Kx63y1WPA5sc3xOBCX04pu-Fv…> Research and Data Analysis professional short courses · <http://email.fdc-k.ngo/tracking/click?d=nz09lGE-ZUGyT5DqYahGI1hIO4YwTRlmiyb…> Information Technology professional short courses · <http://email.fdc-k.ngo/tracking/click?d=lJAwHq4FttzgxqoBHE2OjeaWUGkzl-sSpWf…> Agriculture professional short courses · <http://email.fdc-k.ngo/tracking/click?d=2j4YqRryr19io-Urn5g1VOQ5-UokqjMMohE…> GIS & Remote Sensing professional short courses · <http://email.fdc-k.ngo/tracking/click?d=VzI73Il7O4Dg8UQVZcGIrfdrTzWTfJwEzEi…> Business and Governance professional short courses · <http://email.fdc-k.ngo/tracking/click?d=nL2O3Cv7wz_EvWeaEkei-IKDTDbA2LQaHZu…> Online Training Course · <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctNGY7Sp0mjHuz8t…> Professional Short Training Courses · Reach out to us training(a)fdc-k.org or call +254712260031 · <http://email.fdc-k.ngo/tracking/click?d=l6zfFP-hckb-RlvQdgDdhzm17lV7LEgp_YS…> WhatsApp our consultant or <http://email.fdc-k.ngo/tracking/click?d=onnMgZrG2lzEYMT-KggctLJosandtugvt1R…> Fill form here Kind Regards Jackson Munene Training Manager Foscore Development Center www.fdc-k.org Email: jackson(a)fdc-k.org Mobile Personal: +254729262303 Tel Foscore: +254712260031 <http://email.fdc-k.ngo/tracking/unsubscribe?d=UXqV4LV1OLFz0T_17L-ZvBcn49rNQ…> UNSUBSCRIBE

1 0

invitation for Training on Project Monitoring and Evaluation with Data Management and Analysis Course
by Global King Project Management 05 Sep '22

05 Sep '22

<http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Project Monitoring and Evaluation with Data Management and Analysis Course October 03 to October 14 2022 Nairobi Kenya Venue: Onsite Mombasa or Nairobi Kenya or online platform. Registration: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=oMvADaRTOI…> Use this link to register <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Fvm3u06RbV…> Download Admission Form: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=wo2aX-_h52…> Download Our Calendar: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> View all our courses <http://tracking.globalkingprojectmanagement.com/tracking/click?d=jzdoh1CTZ5…> Reach us through WhatsApp Reach us: training(a)globalkingprojectmanagement.org / +254 114 830 889 About the course This is a comprehensive 10 days M&E course that covers the principles and practices for results-based monitoring and evaluation for the entire project life cycle. This course equips participants with skills in setting up and implementing results-based monitoring and evaluation systems including M&E data management, analysis, and reporting. The participants will benefit from the latest M&E thinking and practices including the results and participatory approaches. This course is designed to enable the participants become experts in monitoring and evaluating their development projects. The course covers all the key elements of a robust M&E system coupled with a practical project to illustrate the M&E concepts. Target Participants This course is designed for researchers, project staff, development practitioners, managers and decision makers who are responsible for project, program or organization-level M&E. The course aims to enhance the skills of professionals who need to research, supervise, manage, plan, implement, monitor and evaluate development projects. Course duration 10 days Course objectives Develop project results levels Design a project using logical framework Develop indicators and targets for each result level Track performance indicators over the life of the project Evaluation a project against the set results Develop and implement M&E systems Develop a comprehensive monitoring and evaluation plan Use data analysis software (Stata/SPSS/R) Collect data using mobile data collection tools Carryout impact evaluation Use GIS to analyze and share project data Course Outline Introduction to Results Based Project Management Fundamentals of Results Based Management Why is RBM important? Results based management vs traditional projects management RBM Lifecycle (seven phases) Areas of focus of RBM Fundamentals of Monitoring and Evaluation Definition of Monitoring and Evaluation Why Monitoring and Evaluation is important Key principles and concepts in M&E M&E in project lifecycle Participatory M&E Project Analysis Situation Analysis Needs Assessment Strategy Analysis Design of Results in Monitoring and Evaluation Results chain approaches: Impact, outcomes, outputs and activities Results framework M&E causal pathway Principles of planning, monitoring and evaluating for results M&E Indicators Indicators definition Indicator metrics Linking indicators to results Indicator matrix Tracking of indicators Logical Framework Approach LFA – Analysis and Planning phase Design of logframe Risk rating in logframe Horizontal and vertical logic in logframe Using logframe to create schedules: Activity and Budget schedules Using logframe as a project management tool Theory of Change Overview of theory of change Developing theory of change Theory of Change vs Log Frame Case study: Theory of change M&E Systems What is an M&E System? Elements of M&E System Steps for developing Results based M&E System M&E Planning Importance of an M&E Plan Documenting M&E System in the M&E Plan Components of an M&E Plan-Monitoring, Evaluation, Data management, Reporting Using M&E Plan to implement M&E in a Project M&E plan vs Performance Management Plan (PMP) Base Survey in Results based M&E Importance of baseline studies Process of conducting baseline studies Baseline study vs evaluation Project Performance Evaluation Process and progress evaluations Evaluation research design Evaluation questions Evaluation report Dissemination M&E Data Management Different sources of M&E data Qualitative data collection methods Quantitative data collection methods Participatory methods of data collection Data Quality Assessment M&E Results Use and Dissemination Stakeholder’s information needs Use of M&E results to improve and strengthen projects Use of M&E Lessons learnt and Best Practices Organization knowledge champions M&E reporting format M&E results communication strategies Gender Perspective in M&E Importance of gender in M&E Integrating gender into program logic Setting gender sensitive indicators Collecting gender disaggregated data Analyzing M&E data from a gender perspective Appraisal of projects from a gender perspective Data Collection Tools and Techniques Sources of M&E data –primary and secondary Sampling during data collection Qualitative data collection methods Quantitative data collection methods Participatory data collection methods Introduction to data triangulation Data Quality What is data quality? Why data quality? Data quality standards Data flow and data quality Data Quality Assessments M&E system design for data quality ICT in Monitoring and Evaluation Mobile based data collection using ODK Data visualization - info graphics and dashboards Use of ICT tools for Real-time monitoring and evaluation Qualitative Data Analysis Principles of qualitative data analysis Data preparation for qualitative analysis Linking and integrating multiple data sets in different forms Thematic analysis for qualitative data Content analysis for qualitative data Manipulation and analysis of data using NVivo Quantitative Data Analysis – (Using SPSS/Stata) Introduction to statistical concepts Creating variables and data entry Data reconstruction Variables manipulation Descriptive statistics Understanding data weighting Inferential statistics: hypothesis testing, T-test, ANOVA, regression analysis Impact Assessment Introduction to impact evaluation Attribution in impact evaluation Estimation of counterfactual Impact evaluation methods: Double difference, Propensity score matching GIS in M&E Introduction to GIS in M&E GIS analysis and mapping techniques Data preparation for geospatial analysis Geospatial analysis using GIS software (QGIS) General Notes · All our courses can be Tailor-made to participants needs The participant must be conversant with English Presentations are well guided, practical exercises, web-based tutorials, and group work. Our facilitators are experts with more than 10years of experience Upon completion of training, the participant will be issued with a Global King Project Management certificate Training will be done at the Global King Project Management Centers (Nairobi Kenya, Mombasa Kenya, Kigali Rwanda, Dubai, Lagos Nigeria and More others) Course duration is flexible and the contents can be modified to fit any number of days. Payment should be done two weeks before commencement of the training, to the Global King Project Management account, so as to enable us to prepare better for you. · The course fee for onsite training includes facilitation training materials, 2 coffee breaks, a buffet lunch, and a Certificate of successful completion of Training. Participants will be responsible for their own travel expenses and arrangements, airport transfers, visa application dinners, health/accident insurance, and other personal expenses. · Accommodation, pickup, freight booking, and Visa processing arrangement, are done on request, at discounted prices. · Tablet and Laptops are provided to participants on request as an add-on cost to the training fee. · One-year free Consultation and Coaching provided after the course. · Register as a group of more than two and enjoy a discount of (10% to 50%) · · Payment should be done before commence of the training or as agreed by the parties, to the GLOBAL KING PROJECT MANAGEMENT account, so as to enable us to prepare better for you. · For any inquiry to: training(a)globalkingprojectmanagement.org or +254 114 830 889 · Website: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=LNx4rltgDu…> www. globalkingprojectmanagement.org Indicative Budget The registration cost is indicated in every course for online and onsite trainings find the group rate discount below: Group Size Discount Rate for Group 2-7 20% 8-21 25% 22-31 30% 32-41 35% 42-51 40% 52>Above 50% Important GKPM Links: <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> All Courses Project Management Trainings <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Research training Courses <http://tracking.globalkingprojectmanagement.com/tracking/click?d=Ny0HQO_DJm…> Leadership Training Courses <http://tracking.globalkingprojectmanagement.com/tracking/click?d=wo2aX-_h52…> Download our Calendar Yours Sincerely, Virginia Wamaitha Project Management Expert and PMP certified <http://tracking.globalkingprojectmanagement.com/tracking/click?d=T5h7J5OlLH…> Global King Project Management Tele: +254114830889 Email Address: training(a)globalkingprojectmanagement.org Transnational Plaza 3rd Floor Nairobi, Kenya <http://tracking.globalkingprojectmanagement.com/tracking/unsubscribe?d=1spy…> UNSUBSCRIBE

1 0