May 2021 - Kernel - mailweb.openeuler.org

perf spe-c2c record失败的问题
by haoxin 15 Dec '21

15 Dec '21

您好，我采用最细的openEuler-1.0测试perf spe-c2c的功能，但是发现/sys/devices下没有arm_spe_0这个目录执行perf spe-c2c record时显示： event syntax error: 'arm_spe_0/ts_enable=1,pct_enable=1,pa_enable=1,load_filter=1,jitter=1,store_filter=1,min_latency=0/' \___ Cannot find PMU `arm_spe_0'. Missing kernel support? 想问下您这是为什么？需要哪里开启spe功能吗？

3 2

[PATCH] 4.19: PTP_KVM support for arm/arm64
by lisiheng 14 Sep '21

14 Sep '21

From: lisiheng <lisiheng(a)bingosoft.net> PTP_KVM is used for high precision time sync between host and guests. It relies on transferring the wall clock and counter value from the host to the guest using a KVM-specific hypercall. mainline inclusion from mainline-v4.19 category: feature bugzilla:https://gitee.com/openeuler/kernel/issues/I3R900 CVE: NA Reference: https://www.kernel.org/doc/html/latest/_sources/virt/kvm/arm/ptp_kvm.rst.txt https://github.com/kata-containers/packaging/blob/master/kernel/patches/4.1… Signed-off-by: lisiheng <lisiheng(a)bingosoft.net> --- drivers/clocksource/arm_arch_timer.c | 24 ++++++ drivers/ptp/Kconfig | 2 +- drivers/ptp/Makefile | 1 + drivers/ptp/ptp_kvm_arm64.c | 52 ++++++++++++ drivers/ptp/{ptp_kvm.c => ptp_kvm_common.c} | 89 +++++---------------- drivers/ptp/ptp_kvm_x86.c | 87 ++++++++++++++++++++ fs/jbd2/transaction.c | 17 ---- include/asm-generic/ptp_kvm.h | 12 +++ include/linux/arm-smccc.h | 5 ++ virt/kvm/arm/hypercalls.c | 14 ++++ 10 files changed, 216 insertions(+), 87 deletions(-) create mode 100644 drivers/ptp/ptp_kvm_arm64.c rename drivers/ptp/{ptp_kvm.c => ptp_kvm_common.c} (55%) create mode 100644 drivers/ptp/ptp_kvm_x86.c create mode 100644 include/asm-generic/ptp_kvm.h diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c index 6847a5fe13fd..f84bed34e9ea 100644 --- a/drivers/clocksource/arm_arch_timer.c +++ b/drivers/clocksource/arm_arch_timer.c @@ -1653,3 +1653,27 @@ static int __init arch_timer_acpi_init(struct acpi_table_header *table) } TIMER_ACPI_DECLARE(arch_timer, ACPI_SIG_GTDT, arch_timer_acpi_init); #endif + +#if IS_ENABLED(CONFIG_PTP_1588_CLOCK_KVM) +#include <linux/arm-smccc.h> +int kvm_arch_ptp_get_clock_fn(long *cycle, struct timespec64 *ts, + struct clocksource **cs) +{ + struct arm_smccc_res hvc_res; + + arm_smccc_1_1_invoke(ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID, + &hvc_res); + + if ((long)(hvc_res.a0) < 0) + return -EOPNOTSUPP; + + ts->tv_sec = hvc_res.a0; + ts->tv_nsec = hvc_res.a1; + *cycle = hvc_res.a2 << 32 | hvc_res.a3; + *cs = &clocksource_counter; + + return 0; +} +EXPORT_SYMBOL_GPL(kvm_arch_ptp_get_clock_fn); +#endif + diff --git a/drivers/ptp/Kconfig b/drivers/ptp/Kconfig index d137c480db46..318b3f5df1ea 100644 --- a/drivers/ptp/Kconfig +++ b/drivers/ptp/Kconfig @@ -109,7 +109,7 @@ config PTP_1588_CLOCK_PCH config PTP_1588_CLOCK_KVM tristate "KVM virtual PTP clock" depends on PTP_1588_CLOCK - depends on KVM_GUEST && X86 + depends on KVM_GUEST && X86 || ARM64 default y help This driver adds support for using kvm infrastructure as a PTP diff --git a/drivers/ptp/Makefile b/drivers/ptp/Makefile index 19efa9cfa950..1bf4940a88a6 100644 --- a/drivers/ptp/Makefile +++ b/drivers/ptp/Makefile @@ -4,6 +4,7 @@ # ptp-y := ptp_clock.o ptp_chardev.o ptp_sysfs.o +ptp_kvm-y := ptp_kvm_common.o ptp_kvm_$(ARCH).o obj-$(CONFIG_PTP_1588_CLOCK) += ptp.o obj-$(CONFIG_PTP_1588_CLOCK_DTE) += ptp_dte.o obj-$(CONFIG_PTP_1588_CLOCK_IXP46X) += ptp_ixp46x.o diff --git a/drivers/ptp/ptp_kvm_arm64.c b/drivers/ptp/ptp_kvm_arm64.c new file mode 100644 index 000000000000..07ee2fc2c7b1 --- /dev/null +++ b/drivers/ptp/ptp_kvm_arm64.c @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Virtual PTP 1588 clock for use with KVM guests + * Copyright (C) 2019 ARM Ltd. + * All Rights Reserved + */ + +#include <linux/kernel.h> +#include <linux/err.h> +#include <asm/hypervisor.h> +#include <linux/module.h> +#include <linux/psci.h> +#include <linux/arm-smccc.h> +#include <linux/timecounter.h> +#include <linux/sched/clock.h> +#include <asm/arch_timer.h> + + +int kvm_arch_ptp_init(void) +{ + struct arm_smccc_res hvc_res; + + arm_smccc_1_1_invoke(ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID, + &hvc_res); + if ((long)(hvc_res.a0) < 0) + return -EOPNOTSUPP; + + return 0; +} + +int kvm_arch_ptp_get_clock_generic(struct timespec64 *ts, + struct arm_smccc_res *hvc_res) +{ + arm_smccc_1_1_invoke(ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID, + hvc_res); + if ((long)(hvc_res->a0) < 0) + return -EOPNOTSUPP; + + ts->tv_sec = hvc_res->a0; + ts->tv_nsec = hvc_res->a1; + + return 0; +} + +int kvm_arch_ptp_get_clock(struct timespec64 *ts) +{ + struct arm_smccc_res hvc_res; + + kvm_arch_ptp_get_clock_generic(ts, &hvc_res); + + return 0; +} diff --git a/drivers/ptp/ptp_kvm.c b/drivers/ptp/ptp_kvm_common.c similarity index 55% rename from drivers/ptp/ptp_kvm.c rename to drivers/ptp/ptp_kvm_common.c index c67dd11e08b1..69a7cd05d188 100644 --- a/drivers/ptp/ptp_kvm.c +++ b/drivers/ptp/ptp_kvm_common.c @@ -1,29 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0-or-later /* * Virtual PTP 1588 clock for use with KVM guests * * Copyright (C) 2017 Red Hat Inc. * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * */ #include <linux/device.h> #include <linux/err.h> #include <linux/init.h> #include <linux/kernel.h> +#include <linux/slab.h> #include <linux/module.h> #include <uapi/linux/kvm_para.h> #include <asm/kvm_para.h> -#include <asm/pvclock.h> -#include <asm/kvmclock.h> -#include <uapi/asm/kvm_para.h> +#include <asm-generic/ptp_kvm.h> #include <linux/ptp_clock_kernel.h> @@ -34,56 +24,31 @@ struct kvm_ptp_clock { DEFINE_SPINLOCK(kvm_ptp_lock); -static struct pvclock_vsyscall_time_info *hv_clock; - -static struct kvm_clock_pairing clock_pair; -static phys_addr_t clock_pair_gpa; static int ptp_kvm_get_time_fn(ktime_t *device_time, struct system_counterval_t *system_counter, void *ctx) { - unsigned long ret; + unsigned long ret, cycle; struct timespec64 tspec; - unsigned version; - int cpu; - struct pvclock_vcpu_time_info *src; + struct clocksource *cs; spin_lock(&kvm_ptp_lock); preempt_disable_notrace(); - cpu = smp_processor_id(); - src = &hv_clock[cpu].pvti; - - do { - /* - * We are using a TSC value read in the hosts - * kvm_hc_clock_pairing handling. - * So any changes to tsc_to_system_mul - * and tsc_shift or any other pvclock - * data invalidate that measurement. - */ - version = pvclock_read_begin(src); - - ret = kvm_hypercall2(KVM_HC_CLOCK_PAIRING, - clock_pair_gpa, - KVM_CLOCK_PAIRING_WALLCLOCK); - if (ret != 0) { - pr_err_ratelimited("clock pairing hypercall ret %lu\n", ret); - spin_unlock(&kvm_ptp_lock); - preempt_enable_notrace(); - return -EOPNOTSUPP; - } - - tspec.tv_sec = clock_pair.sec; - tspec.tv_nsec = clock_pair.nsec; - ret = __pvclock_read_cycles(src, clock_pair.tsc); - } while (pvclock_read_retry(src, version)); + + ret = kvm_arch_ptp_get_clock_fn(&cycle, &tspec, &cs); + if (ret != 0) { + pr_err_ratelimited("clock pairing hypercall ret %lu\n", ret); + spin_unlock(&kvm_ptp_lock); + preempt_enable_notrace(); + return -EOPNOTSUPP; + } preempt_enable_notrace(); - system_counter->cycles = ret; - system_counter->cs = &kvm_clock; + system_counter->cycles = cycle; + system_counter->cs = cs; *device_time = timespec64_to_ktime(tspec); @@ -126,17 +91,13 @@ static int ptp_kvm_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts) spin_lock(&kvm_ptp_lock); - ret = kvm_hypercall2(KVM_HC_CLOCK_PAIRING, - clock_pair_gpa, - KVM_CLOCK_PAIRING_WALLCLOCK); + ret = kvm_arch_ptp_get_clock(&tspec); if (ret != 0) { pr_err_ratelimited("clock offset hypercall ret %lu\n", ret); spin_unlock(&kvm_ptp_lock); return -EOPNOTSUPP; } - tspec.tv_sec = clock_pair.sec; - tspec.tv_nsec = clock_pair.nsec; spin_unlock(&kvm_ptp_lock); memcpy(ts, &tspec, sizeof(struct timespec64)); @@ -176,22 +137,12 @@ static void __exit ptp_kvm_exit(void) static int __init ptp_kvm_init(void) { - long ret; - - if (!kvm_para_available()) - return -ENODEV; - clock_pair_gpa = slow_virt_to_phys(&clock_pair); - hv_clock = pvclock_get_pvti_cpu0_va(); - - if (!hv_clock) - return -ENODEV; - - ret = kvm_hypercall2(KVM_HC_CLOCK_PAIRING, clock_pair_gpa, - KVM_CLOCK_PAIRING_WALLCLOCK); - if (ret == -KVM_ENOSYS || ret == -KVM_EOPNOTSUPP) - return -ENODEV; + int ret; + ret = kvm_arch_ptp_init(); + if (ret) + return -EOPNOTSUPP; kvm_ptp_clock.caps = ptp_kvm_caps; kvm_ptp_clock.ptp_clock = ptp_clock_register(&kvm_ptp_clock.caps, NULL); diff --git a/drivers/ptp/ptp_kvm_x86.c b/drivers/ptp/ptp_kvm_x86.c new file mode 100644 index 000000000000..a52cf1c2990c --- /dev/null +++ b/drivers/ptp/ptp_kvm_x86.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Virtual PTP 1588 clock for use with KVM guests + * + * Copyright (C) 2017 Red Hat Inc. + */ + +#include <asm/pvclock.h> +#include <asm/kvmclock.h> +#include <linux/module.h> +#include <uapi/asm/kvm_para.h> +#include <uapi/linux/kvm_para.h> +#include <linux/ptp_clock_kernel.h> + +phys_addr_t clock_pair_gpa; +struct kvm_clock_pairing clock_pair; +struct pvclock_vsyscall_time_info *hv_clock; + +int kvm_arch_ptp_init(void) +{ + int ret; + + if (!kvm_para_available()) + return -ENODEV; + + clock_pair_gpa = slow_virt_to_phys(&clock_pair); + hv_clock = pvclock_get_pvti_cpu0_va(); + if (!hv_clock) + return -ENODEV; + + ret = kvm_hypercall2(KVM_HC_CLOCK_PAIRING, clock_pair_gpa, + KVM_CLOCK_PAIRING_WALLCLOCK); + if (ret == -KVM_ENOSYS || ret == -KVM_EOPNOTSUPP) + return -ENODEV; + + return 0; +} + +int kvm_arch_ptp_get_clock(struct timespec64 *ts) +{ + long ret; + + ret = kvm_hypercall2(KVM_HC_CLOCK_PAIRING, + clock_pair_gpa, + KVM_CLOCK_PAIRING_WALLCLOCK); + if (ret != 0) + return -EOPNOTSUPP; + + ts->tv_sec = clock_pair.sec; + ts->tv_nsec = clock_pair.nsec; + + return 0; +} + +int kvm_arch_ptp_get_clock_fn(unsigned long *cycle, struct timespec64 *tspec, + struct clocksource **cs) +{ + unsigned long ret; + unsigned int version; + int cpu; + struct pvclock_vcpu_time_info *src; + + cpu = smp_processor_id(); + src = &hv_clock[cpu].pvti; + + do { + /* + * We are using a TSC value read in the hosts + * kvm_hc_clock_pairing handling. + * So any changes to tsc_to_system_mul + * and tsc_shift or any other pvclock + * data invalidate that measurement. + */ + version = pvclock_read_begin(src); + + ret = kvm_hypercall2(KVM_HC_CLOCK_PAIRING, + clock_pair_gpa, + KVM_CLOCK_PAIRING_WALLCLOCK); + tspec->tv_sec = clock_pair.sec; + tspec->tv_nsec = clock_pair.nsec; + *cycle = __pvclock_read_cycles(src, clock_pair.tsc); + } while (pvclock_read_retry(src, version)); + + *cs = &kvm_clock; + + return 0; +} diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c index ffa6d3530f4b..4055929a043c 100644 --- a/fs/jbd2/transaction.c +++ b/fs/jbd2/transaction.c @@ -2085,7 +2085,6 @@ int jbd2_journal_try_to_free_buffers(journal_t *journal, { struct buffer_head *head; struct buffer_head *bh; - bool has_write_io_error = false; int ret = 0; J_ASSERT(PageLocked(page)); @@ -2110,26 +2109,10 @@ int jbd2_journal_try_to_free_buffers(journal_t *journal, jbd_unlock_bh_state(bh); if (buffer_jbd(bh)) goto busy; - - /* - * If we free a metadata buffer which has been failed to - * write out, the jbd2 checkpoint procedure will not detect - * this failure and may lead to filesystem inconsistency - * after cleanup journal tail. - */ - if (buffer_write_io_error(bh)) { - pr_err("JBD2: Error while async write back metadata bh %llu.", - (unsigned long long)bh->b_blocknr); - has_write_io_error = true; - } } while ((bh = bh->b_this_page) != head); ret = try_to_free_buffers(page); - busy: - if (has_write_io_error) - jbd2_journal_abort(journal, -EIO); - return ret; } diff --git a/include/asm-generic/ptp_kvm.h b/include/asm-generic/ptp_kvm.h new file mode 100644 index 000000000000..d87a08495710 --- /dev/null +++ b/include/asm-generic/ptp_kvm.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * linux/drivers/clocksource/arm_arch_timer.c + * + * Copyright (C) 2019 ARM Ltd. + * All Rights Reserved + */ + +int kvm_arch_ptp_init(void); +int kvm_arch_ptp_get_clock(struct timespec64 *ts); +int kvm_arch_ptp_get_clock_fn(unsigned long *cycle, + struct timespec64 *tspec, void *cs); diff --git a/include/linux/arm-smccc.h b/include/linux/arm-smccc.h index 5b5f52ca6ac8..5a46bb6df894 100644 --- a/include/linux/arm-smccc.h +++ b/include/linux/arm-smccc.h @@ -76,6 +76,11 @@ ARM_SMCCC_SMC_32, \ 0, 1) +#define ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + 0, 2) + #define ARM_SMCCC_ARCH_WORKAROUND_1 \ ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ ARM_SMCCC_SMC_32, \ diff --git a/virt/kvm/arm/hypercalls.c b/virt/kvm/arm/hypercalls.c index 7a6c5f18dff2..495b25b9ed13 100644 --- a/virt/kvm/arm/hypercalls.c +++ b/virt/kvm/arm/hypercalls.c @@ -15,6 +15,9 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) u32 val = SMCCC_RET_NOT_SUPPORTED; u32 feature; gpa_t gpa; + struct timespec64 ts; + u64 cycles, cycle_high, cycle_low; + struct system_time_snapshot systime_snapshot; switch (func_id) { case ARM_SMCCC_VERSION_FUNC_ID: @@ -63,6 +66,17 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) case ARM_SMCCC_HV_PV_SCHED_KICK_CPU: val = kvm_pvsched_kick_vcpu(vcpu); break; + + case ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID: + ktime_get_real_ts64(&ts); + ktime_get_snapshot(&systime_snapshot); + cycles = systime_snapshot.cycles - vcpu_vtimer(vcpu)->cntvoff; + cycle_high = cycles >> 32; + cycle_low = cycles << 32 >> 32; + + smccc_set_retval(vcpu, ts.tv_sec, ts.tv_nsec, cycle_high, cycle_low); + return 1; + default: return kvm_psci_call(vcpu); } -- 2.23.0

2 4

[PATCH kernel-4.19] ata: ahci: Disable SXS for Hisilicon Kunpeng920
by Yang Yingliang 31 May '21

31 May '21

From: Xingui Yang <yangxingui(a)huawei.com> mainline inclusion from mainline-v5.12-rc6 commit 234e6d2c18f5 category: bugfix bugzilla: NA CVE: NA On Hisilicon Kunpeng920, ESP is set to 1 by default for all ports of SATA controller. In some scenarios, some ports are not external SATA ports, and it cause disks connected to these ports to be identified as removable disks. So disable the SXS capability on the software side to prevent users from mistakenly considering non-removable disks as removable disks and performing related operations. Signed-off-by: Xingui Yang <yangxingui(a)huawei.com> Signed-off-by: Luo Jiaxing <luojiaxing(a)huawei.com> Reviewed-by: John Garry <john.garry(a)huawei.com> Link: https://lore.kernel.org/r/1615544676-61926-1-git-send-email-luojiaxing@huaw… Signed-off-by: Jens Axboe <axboe(a)kernel.dk> Signed-off-by: Yang Xingui <yangxingui(a)huawei.com> Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> --- drivers/ata/ahci.c | 5 +++++ drivers/ata/ahci.h | 1 + drivers/ata/libahci.c | 5 +++++ 3 files changed, 11 insertions(+) diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c index 8df0ec85cc7b9..bfb4c7a677717 100644 --- a/drivers/ata/ahci.c +++ b/drivers/ata/ahci.c @@ -1761,6 +1761,11 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) hpriv->flags |= AHCI_HFLAG_NO_DEVSLP; #ifdef CONFIG_ARM64 + if (pdev->vendor == PCI_VENDOR_ID_HUAWEI && + pdev->device == 0xa235 && + pdev->revision < 0x30) + hpriv->flags |= AHCI_HFLAG_NO_SXS; + if (pdev->vendor == 0x177d && pdev->device == 0xa01c) hpriv->irq_handler = ahci_thunderx_irq_handler; #endif diff --git a/drivers/ata/ahci.h b/drivers/ata/ahci.h index 9290e787abdc4..52d0851653e5f 100644 --- a/drivers/ata/ahci.h +++ b/drivers/ata/ahci.h @@ -254,6 +254,7 @@ enum { AHCI_HFLAG_IS_MOBILE = (1 << 25), /* mobile chipset, use SATA_MOBILE_LPM_POLICY as default lpm_policy */ + AHCI_HFLAG_NO_SXS = (1 << 28), /* SXS not supported */ /* ap->flags bits */ diff --git a/drivers/ata/libahci.c b/drivers/ata/libahci.c index 4577f76e736bf..faead87546106 100644 --- a/drivers/ata/libahci.c +++ b/drivers/ata/libahci.c @@ -509,6 +509,11 @@ void ahci_save_initial_config(struct device *dev, struct ahci_host_priv *hpriv) cap |= HOST_CAP_ALPM; } + if ((cap & HOST_CAP_SXS) && (hpriv->flags & AHCI_HFLAG_NO_SXS)) { + dev_info(dev, "controller does not support SXS, disabling CAP_SXS\n"); + cap &= ~HOST_CAP_SXS; + } + if (hpriv->force_port_map && port_map != hpriv->force_port_map) { dev_info(dev, "forcing port_map 0x%x -> 0x%x\n", port_map, hpriv->force_port_map); -- 2.25.1

1 0

[PATCH kernel-4.19 1/6] NFS: Refactor nfs_instantiate() for dentry referencing callers
by Yang Yingliang 31 May '21

31 May '21

From: Benjamin Coddington <bcodding(a)redhat.com> mainline inclusion from mainline-v5.4-rc1 commit 406cd91533dcc5e82ef2373c39e6a531d944131e category: bugfix bugzilla: NA CVE: NA -------------------------------- Since commit b0c6108ecf64 ("nfs_instantiate(): prevent multiple aliases for directory inode"), nfs_instantiate() may succeed without actually instantiating the dentry that was passed in. That can be problematic for some callers in NFSv3, so this patch breaks things up so we can get the actual dentry obtained. Signed-off-by: Benjamin Coddington <bcodding(a)redhat.com> Signed-off-by: Anna Schumaker <Anna.Schumaker(a)Netapp.com> Signed-off-by: Zhang Xiaoxu <zhangxiaoxu5(a)huawei.com> Reviewed-by: Zhang Yi <yi.zhang(a)huawei.com> Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> --- fs/nfs/dir.c | 41 +++++++++++++++++++++++++++-------------- include/linux/nfs_fs.h | 3 +++ 2 files changed, 30 insertions(+), 14 deletions(-) diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c index 733fd9e4f0a15..0252d9af03e81 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -1692,24 +1692,23 @@ static int nfs4_lookup_revalidate(struct dentry *dentry, unsigned int flags) #endif /* CONFIG_NFSV4 */ -/* - * Code common to create, mkdir, and mknod. - */ -int nfs_instantiate(struct dentry *dentry, struct nfs_fh *fhandle, +struct dentry * +nfs_add_or_obtain(struct dentry *dentry, struct nfs_fh *fhandle, struct nfs_fattr *fattr, struct nfs4_label *label) { struct dentry *parent = dget_parent(dentry); struct inode *dir = d_inode(parent); struct inode *inode; - struct dentry *d; - int error = -EACCES; + struct dentry *d = NULL; + int error; d_drop(dentry); /* We may have been initialized further down */ if (d_really_is_positive(dentry)) goto out; + if (fhandle->size == 0) { error = NFS_PROTO(dir)->lookup(dir, &dentry->d_name, fhandle, fattr, NULL); if (error) @@ -1725,18 +1724,32 @@ int nfs_instantiate(struct dentry *dentry, struct nfs_fh *fhandle, } inode = nfs_fhget(dentry->d_sb, fhandle, fattr, label); d = d_splice_alias(inode, dentry); - if (IS_ERR(d)) { - error = PTR_ERR(d); - goto out_error; - } - dput(d); out: dput(parent); - return 0; + return d; out_error: nfs_mark_for_revalidate(dir); - dput(parent); - return error; + d = ERR_PTR(error); + goto out; +} +EXPORT_SYMBOL_GPL(nfs_add_or_obtain); + +/* + * Code common to create, mkdir, and mknod. + */ +int nfs_instantiate(struct dentry *dentry, struct nfs_fh *fhandle, + struct nfs_fattr *fattr, + struct nfs4_label *label) +{ + struct dentry *d; + + d = nfs_add_or_obtain(dentry, fhandle, fattr, label); + if (IS_ERR(d)) + return PTR_ERR(d); + + /* Callers don't care */ + dput(d); + return 0; } EXPORT_SYMBOL_GPL(nfs_instantiate); diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h index 620306b1604d7..dcf01bdcc3e3b 100644 --- a/include/linux/nfs_fs.h +++ b/include/linux/nfs_fs.h @@ -487,6 +487,9 @@ extern const struct file_operations nfs_dir_operations; extern const struct dentry_operations nfs_dentry_operations; extern void nfs_force_lookup_revalidate(struct inode *dir); +extern struct dentry *nfs_add_or_obtain(struct dentry *dentry, + struct nfs_fh *fh, struct nfs_fattr *fattr, + struct nfs4_label *label); extern int nfs_instantiate(struct dentry *dentry, struct nfs_fh *fh, struct nfs_fattr *fattr, struct nfs4_label *label); extern int nfs_may_open(struct inode *inode, struct rpc_cred *cred, int openflags); -- 2.25.1

1 5

[PATCH kernel-4.19 00/50] backport linux-4.19.192
by Yang Yingliang 29 May '21

29 May '21

Anirudh Rayabharam (3): rapidio: handle create_workqueue() failure net: stmicro: handle clk_prepare() failure during init video: hgafb: correctly handle card detect failure during probe Atul Gopinathan (1): cdrom: gdrom: deallocate struct gdrom_unit fields in remove_gdrom Christophe JAILLET (1): openrisc: Fix a memory leak Dan Carpenter (1): firmware: arm_scpi: Prevent the ternary sign expansion bug Daniel Wagner (1): nvmet: seset ns->file when open fails Du Cheng (1): ethernet: sun: niu: fix missing checks of niu_pci_eeprom_read() Greg Kroah-Hartman (18): Revert "ALSA: sb8: add a check for request_region" Revert "rapidio: fix a NULL pointer dereference when create_workqueue() fails" Revert "serial: mvebu-uart: Fix to avoid a potential NULL pointer dereference" Revert "video: hgafb: fix potential NULL pointer dereference" Revert "net: stmicro: fix a missing check of clk_prepare" Revert "leds: lp5523: fix a missing check of return value of lp55xx_read" Revert "hwmon: (lm80) fix a missing check of bus read in lm80 probe" Revert "video: imsttfb: fix potential NULL pointer dereferences" Revert "ecryptfs: replace BUG_ON with error handling code" Revert "scsi: ufs: fix a missing check of devm_reset_control_get" Revert "gdrom: fix a memory leak bug" cdrom: gdrom: initialize global variable at init time Revert "media: rcar_drif: fix a memory disclosure" Revert "rtlwifi: fix a potential NULL pointer dereference" Revert "qlcnic: Avoid potential NULL pointer dereference" Revert "niu: fix missing checks of niu_pci_eeprom_read" net: rtlwifi: properly check for alloc_workqueue() failure Linux 4.19.192 Hans de Goede (1): platform/x86: dell-smbios-wmi: Fix oops on rmmod dell_smbios Hui Wang (1): ALSA: hda/realtek: reset eapd coeff to default value for alc287 Igor Matheus Andrade Torrente (1): video: hgafb: fix potential NULL pointer dereference Jan Beulich (1): xen-pciback: reconfigure also from backend watch handler Leon Romanovsky (1): RDMA/rxe: Clear all QP fields if creation failed Luiz Augusto von Dentz (1): Bluetooth: SMP: Fail if remote and local public keys are identical Maciej W. Rozycki (2): vgacon: Record video mode changes with VT_RESIZEX vt: Fix character height handling with VT_RESIZEX Maor Gottlieb (1): RDMA/mlx5: Recover from fatal event in dual port mode Mikulas Patocka (1): dm snapshot: fix crash with transient storage and zero chunk size Oleg Nesterov (1): ptrace: make ptrace() fail if the tracee changed its pid unexpectedly PeiSen Hou (1): ALSA: hda/realtek: Add some CLOVE SSIDs of ALC293 Phillip Potter (2): scsi: ufs: handle cleanup correctly on devm_reset_control_get error leds: lp5523: check return value of lp5xx_read and jump to cleanup code Ronnie Sahlberg (1): cifs: fix memory leak in smb2_copychunk_range Takashi Iwai (2): ALSA: line6: Fix racy initialization of LINE6 MIDI ALSA: usb-audio: Validate MS endpoint descriptors Takashi Sakamoto (3): ALSA: dice: fix stream format for TC Electronic Konnekt Live at high sampling transfer frequency ALSA: dice: fix stream format at middle sampling rate for Alesis iO 26 ALSA: bebob/oxfw: fix Kconfig entry for Mackie d.2 Pro Tetsuo Handa (1): tty: vt: always invoke vc->vc_sw->con_resize callback Tom Seewald (1): qlcnic: Add null check after calling netdev_alloc_skb Zhen Lei (1): scsi: qla2xxx: Fix error return code in qla82xx_write_flash_dword() Zqiang (1): locking/mutex: clear MUTEX_FLAGS if wait_list is empty due to signal Makefile | 2 +- arch/openrisc/kernel/setup.c | 2 + drivers/cdrom/gdrom.c | 13 ++++- drivers/firmware/arm_scpi.c | 4 +- drivers/hwmon/lm80.c | 11 +--- drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/sw/rxe/rxe_qp.c | 7 +++ drivers/leds/leds-lp5523.c | 2 +- drivers/md/dm-snap.c | 1 + drivers/media/platform/rcar_drif.c | 1 - .../ethernet/qlogic/qlcnic/qlcnic_ethtool.c | 3 +- .../net/ethernet/stmicro/stmmac/dwmac-sunxi.c | 8 +-- drivers/net/ethernet/sun/niu.c | 32 +++++++---- drivers/net/wireless/realtek/rtlwifi/base.c | 19 +++---- drivers/nvme/target/io-cmd-file.c | 8 ++- drivers/platform/x86/dell-smbios-wmi.c | 3 +- drivers/rapidio/rio_cm.c | 17 +++--- drivers/scsi/qla2xxx/qla_nx.c | 3 +- drivers/scsi/ufs/ufs-hisi.c | 15 +++-- drivers/tty/serial/mvebu-uart.c | 3 - drivers/tty/vt/vt.c | 2 +- drivers/tty/vt/vt_ioctl.c | 6 +- drivers/video/console/vgacon.c | 56 +++++++++++-------- drivers/video/fbdev/core/fbcon.c | 2 +- drivers/video/fbdev/hgafb.c | 21 ++++--- drivers/video/fbdev/imsttfb.c | 5 -- drivers/xen/xen-pciback/xenbus.c | 22 ++++++-- fs/cifs/smb2ops.c | 2 + fs/ecryptfs/crypto.c | 6 +- include/linux/console_struct.h | 1 + kernel/locking/mutex-debug.c | 4 +- kernel/locking/mutex-debug.h | 2 +- kernel/locking/mutex.c | 18 ++++-- kernel/locking/mutex.h | 4 +- kernel/ptrace.c | 18 +++++- net/bluetooth/smp.c | 9 +++ sound/firewire/Kconfig | 4 +- sound/firewire/bebob/bebob.c | 2 +- sound/firewire/dice/dice-alesis.c | 2 +- sound/firewire/dice/dice-tcelectronic.c | 4 +- sound/firewire/oxfw/oxfw.c | 1 - sound/isa/sb/sb8.c | 4 -- sound/pci/hda/patch_realtek.c | 20 ++++++- sound/usb/line6/driver.c | 4 ++ sound/usb/line6/pod.c | 5 -- sound/usb/line6/variax.c | 6 -- sound/usb/midi.c | 4 ++ 47 files changed, 236 insertions(+), 153 deletions(-) -- 2.25.1

1 50

[PATCH kernel-4.19 1/5] Revert "mm: memcontrol: fix slub memory accounting"
by Yang Yingliang 29 May '21

29 May '21

hulk inclusion category: bugfix bugzilla: 51349 CVE: NA ------------------------------------------------- This patchset https://patchwork.kernel.org/project/linux-block/cover/20190826111627.7505-… will cause perfmance regression, so revert it and use another way to fix the warning introduced by fix CVE-2021-27365. Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> Reviewed-by: Kefeng Wang <wangkefeng.wang(a)huawei.com> Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> --- mm/slab_common.c | 4 ++-- mm/slub.c | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index b8b0df81bece3..321a9abed5d9d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1259,8 +1259,8 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) page = alloc_pages(flags, order); if (likely(page)) { ret = page_address(page); - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, - PAGE_SIZE << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, + 1 << order); } kmemleak_alloc(ret, size, 1, flags); kasan_kmalloc_large(ret, size, flags); diff --git a/mm/slub.c b/mm/slub.c index 12f23ceab1177..0d69d5b3ceefe 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3799,8 +3799,8 @@ static void *kmalloc_large_node(size_t size, gfp_t flags, int node) page = alloc_pages_node(node, flags, order); if (page) { ptr = page_address(page); - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, - PAGE_SIZE << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, + 1 << order); } kmalloc_large_node_hook(ptr, size, flags); @@ -3940,8 +3940,8 @@ void kfree(const void *x) BUG_ON(!PageCompound(page)); kfree_hook(object); - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, - -(PAGE_SIZE << order)); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, + -(1 << order)); __free_pages(page, order); return; } -- 2.25.1

1 4

完成任务，获得荣誉，openEuler 社区的这项比赛有点逆天！
by Marketing openEuler 27 May '21

27 May '21

先说比赛规则独立开发者参赛 *做题得分* *每周结算* 这是任务打榜赛不满足？来看一下*特性命题赛*？独立开发者组队参赛区域赛、半决赛、总决赛过五关斩六将谁可以独占鳌头？你将是*社区贡献者* *你的 ID* 会被铭刻在社区还有*“一”点奖金* [image: WechatIMG16.jpeg]

1 0

[PATCH kernel-4.19] arm/ras: Report ARM processor information to userspace
by Yang Yingliang 27 May '21

27 May '21

From: Shenwei Luo <luoshengwei(a)huawei.com> kunpeng inclusion category: feature bugzilla: https://bugzilla.openeuler.org/show_bug.cgi?id=45 CVE: NA The ARM processor error section includes several ARM processor error information, several ARM processor context information and several vendor specific error information structures. Report all of these information to userspace via perf i/f. Shengwei Luo: backport for openEuler 20.xx kernel. V2: report severity info to userspace V1: fix the error in the original patch. Ensure all info to be parsed correctly. Original-Author: Jason Tian <jason(a)os.amperecomputing.com> Signed-off-by: Gong Chen <chengong15(a)huawei.com> Signed-off-by: Shengwei Luo <luoshengwei(a)huawei.com> Cc: Chen Wei <chenwei68(a)huawei.com> Reviewed-by: Xiongfeng Wang <wangxiongfeng2(a)huawei.com> Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> --- arch/arm64/kernel/ras.c | 5 +++-- drivers/acpi/apei/ghes.c | 6 +++--- drivers/ras/ras.c | 40 ++++++++++++++++++++++++++++++++++-- include/acpi/ghes.h | 2 +- include/linux/ras.h | 4 ++-- include/ras/ras_event.h | 44 +++++++++++++++++++++++++++++++++++----- 6 files changed, 86 insertions(+), 15 deletions(-) diff --git a/arch/arm64/kernel/ras.c b/arch/arm64/kernel/ras.c index 0414289b0707c..181e609e1cd4c 100644 --- a/arch/arm64/kernel/ras.c +++ b/arch/arm64/kernel/ras.c @@ -118,13 +118,14 @@ void sea_notify_process(void) } } -void ghes_arm_process_error(struct ghes *ghes, struct cper_sec_proc_arm *err) +void ghes_arm_process_error(struct ghes *ghes, + struct cper_sec_proc_arm *err, int sec_sev) { int i; bool info_saved = false; struct cper_arm_err_info *err_info; - log_arm_hw_error(err); + log_arm_hw_error(err, sec_sev); if ((ghes->generic->notify.type != ACPI_HEST_NOTIFY_SEA) || (ghes->estatus->error_severity != CPER_SEV_RECOVERABLE)) diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c index e2704619e7015..e807e8f74d1e0 100644 --- a/drivers/acpi/apei/ghes.c +++ b/drivers/acpi/apei/ghes.c @@ -465,9 +465,9 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata) } void __weak ghes_arm_process_error(struct ghes *ghes, - struct cper_sec_proc_arm *err) + struct cper_sec_proc_arm *err, int sec_sev) { - log_arm_hw_error(err); + log_arm_hw_error(err, sec_sev); } static void ghes_do_proc(struct ghes *ghes, @@ -510,7 +510,7 @@ static void ghes_do_proc(struct ghes *ghes, else if (guid_equal(sec_type, &CPER_SEC_PROC_ARM)) { struct cper_sec_proc_arm *err = acpi_hest_get_payload(gdata); - ghes_arm_process_error(ghes, err); + ghes_arm_process_error(ghes, err, sec_sev); } else if (guid_equal(sec_type, &CPER_SEC_TS_CORE)) { blocking_notifier_call_chain(&ghes_ts_err_chain, 0, acpi_hest_get_payload(gdata)); diff --git a/drivers/ras/ras.c b/drivers/ras/ras.c index 3f38907320dcc..9302ed7f42588 100644 --- a/drivers/ras/ras.c +++ b/drivers/ras/ras.c @@ -21,9 +21,45 @@ void log_non_standard_event(const uuid_le *sec_type, const uuid_le *fru_id, trace_non_standard_event(sec_type, fru_id, fru_text, sev, err, len); } -void log_arm_hw_error(struct cper_sec_proc_arm *err) +void log_arm_hw_error(struct cper_sec_proc_arm *err, const u8 sev) { - trace_arm_event(err); + u32 pei_len; + u32 ctx_len = 0; + u32 vsei_len; + u8 *pei_err; + u8 *ctx_err; + u8 *ven_err_data; + struct cper_arm_err_info *err_info; + struct cper_arm_ctx_info *ctx_info; + int n, sz; + + pei_len = sizeof(struct cper_arm_err_info) * err->err_info_num; + pei_err = (u8 *)err + sizeof(struct cper_sec_proc_arm); + + err_info = (struct cper_arm_err_info *)(err + 1); + ctx_info = (struct cper_arm_ctx_info *)(err_info + err->err_info_num); + ctx_err = (u8 *)ctx_info; + for (n = 0; n < err->context_info_num; n++) { + sz = sizeof(struct cper_arm_ctx_info) + ctx_info->size; + ctx_info = (struct cper_arm_ctx_info *)((long)ctx_info + sz); + ctx_len += sz; + } + + vsei_len = err->section_length - (sizeof(struct cper_sec_proc_arm) + + pei_len + ctx_len); + if (vsei_len < 0) { + printk(FW_BUG + "section length: %d\n", err->section_length); + printk(FW_BUG + "section length is too small\n"); + pr_warn(FW_BUG + "firmware-generated error record is incorrect\n"); + vsei_len = 0; + } + ven_err_data = (u8 *)ctx_info; + + trace_arm_event(err, pei_err, pei_len, ctx_err, ctx_len, + ven_err_data, vsei_len, sev); } static int __init ras_init(void) diff --git a/include/acpi/ghes.h b/include/acpi/ghes.h index 4c536a2638df7..9aaeaaa3d1a7f 100644 --- a/include/acpi/ghes.h +++ b/include/acpi/ghes.h @@ -124,7 +124,7 @@ int ghes_notify_sea(void); static inline int ghes_notify_sea(void) { return -ENOENT; } #endif extern void ghes_arm_process_error(struct ghes *ghes, - struct cper_sec_proc_arm *err); + struct cper_sec_proc_arm *err, int sec_sev); struct ghes_mem_err { int notify_type; diff --git a/include/linux/ras.h b/include/linux/ras.h index 7c3debb47c87a..3431b4a5fa42d 100644 --- a/include/linux/ras.h +++ b/include/linux/ras.h @@ -29,7 +29,7 @@ static inline int cec_add_elem(u64 pfn) { return -ENODEV; } void log_non_standard_event(const guid_t *sec_type, const guid_t *fru_id, const char *fru_text, const u8 sev, const u8 *err, const u32 len); -void log_arm_hw_error(struct cper_sec_proc_arm *err); +void log_arm_hw_error(struct cper_sec_proc_arm *err, const u8 sev); #else static inline void log_non_standard_event(const guid_t *sec_type, @@ -37,7 +37,7 @@ log_non_standard_event(const guid_t *sec_type, const u8 sev, const u8 *err, const u32 len) { return; } static inline void -log_arm_hw_error(struct cper_sec_proc_arm *err) { return; } +log_arm_hw_error(struct cper_sec_proc_arm *err, const u8 sev) { return; } #endif #endif /* __RAS_H__ */ diff --git a/include/ras/ras_event.h b/include/ras/ras_event.h index a0794632fd01a..7c8cb123ba32d 100644 --- a/include/ras/ras_event.h +++ b/include/ras/ras_event.h @@ -168,11 +168,23 @@ TRACE_EVENT(mc_event, * This event is generated when hardware detects an ARM processor error * has occurred. UEFI 2.6 spec section N.2.4.4. */ +#define APEIL "ARM Processor Err Info data len" +#define APEID "ARM Processor Err Info raw data" +#define APECIL "ARM Processor Err Context Info data len" +#define APECID "ARM Processor Err Context Info raw data" +#define VSEIL "Vendor Specific Err Info data len" +#define VSEID "Vendor Specific Err Info raw data" TRACE_EVENT(arm_event, - TP_PROTO(const struct cper_sec_proc_arm *proc), + TP_PROTO(const struct cper_sec_proc_arm *proc, const u8 *pei_err, + const u32 pei_len, + const u8 *ctx_err, + const u32 ctx_len, + const u8 *oem, + const u32 oem_len, + u8 sev), - TP_ARGS(proc), + TP_ARGS(proc, pei_err, pei_len, ctx_err, ctx_len, oem, oem_len, sev), TP_STRUCT__entry( __field(u64, mpidr) @@ -180,6 +192,13 @@ TRACE_EVENT(arm_event, __field(u32, running_state) __field(u32, psci_state) __field(u8, affinity) + __field(u32, pei_len) + __dynamic_array(u8, buf, pei_len) + __field(u32, ctx_len) + __dynamic_array(u8, buf1, ctx_len) + __field(u32, oem_len) + __dynamic_array(u8, buf2, oem_len) + __field(u8, sev) ), TP_fast_assign( @@ -199,12 +218,27 @@ TRACE_EVENT(arm_event, __entry->running_state = ~0; __entry->psci_state = ~0; } + __entry->pei_len = pei_len; + memcpy(__get_dynamic_array(buf), pei_err, pei_len); + __entry->ctx_len = ctx_len; + memcpy(__get_dynamic_array(buf1), ctx_err, ctx_len); + __entry->oem_len = oem_len; + memcpy(__get_dynamic_array(buf2), oem, oem_len); + __entry->sev = sev; ), - TP_printk("affinity level: %d; MPIDR: %016llx; MIDR: %016llx; " - "running state: %d; PSCI state: %d", + TP_printk("error: %d; affinity level: %d; MPIDR: %016llx; MIDR: %016llx; " + "running state: %d; PSCI state: %d; " + "%s: %d; %s: %s; %s: %d; %s: %s; %s: %d; %s: %s", + __entry->sev, __entry->affinity, __entry->mpidr, __entry->midr, - __entry->running_state, __entry->psci_state) + __entry->running_state, __entry->psci_state, + APEIL, __entry->pei_len, APEID, + __print_hex(__get_dynamic_array(buf), __entry->pei_len), + APECIL, __entry->ctx_len, APECID, + __print_hex(__get_dynamic_array(buf1), __entry->ctx_len), + VSEIL, __entry->oem_len, VSEID, + __print_hex(__get_dynamic_array(buf2), __entry->oem_len)) ); /* -- 2.25.1

1 0

[PATCH openEuler-1.0-LTS] io_uring: truncate lengths larger than MAX_RW_COUNT on provide buffers
by Yang Yingliang 27 May '21

27 May '21

From: Thadeu Lima de Souza Cascardo <cascardo(a)canonical.com> mainline inclusion from mainline-v5.13-rc1 commit d1f82808877bb10d3deee7cf3374a4eb3fb582db category: bugfix bugzilla: NA CVE: CVE-2021-3491 -------------------------------- Read and write operations are capped to MAX_RW_COUNT. Some read ops rely on that limit, and that is not guaranteed by the IORING_OP_PROVIDE_BUFFERS. Truncate those lengths when doing io_add_buffers, so buffer addresses still use the uncapped length. Also, take the chance and change struct io_buffer len member to __u32, so it matches struct io_provide_buffer len member. This fixes CVE-2021-3491, also reported as ZDI-CAN-13546. Fixes: ddf0322db79c ("io_uring: add IORING_OP_PROVIDE_BUFFERS") Reported-by: Billy Jheng Bing-Jhong (@st424204) Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo(a)canonical.com> Signed-off-by: Jens Axboe <axboe(a)kernel.dk> Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> Reviewed-by: Xiu Jianfeng <xiujianfeng(a)huawei.com> Reviewed-by: Zhang Yi <yi.zhang(a)huawei.com> Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> --- fs/io_uring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 02ebfc46228a9..f661231cdfad6 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -216,7 +216,7 @@ struct fixed_file_data { struct io_buffer { struct list_head list; __u64 addr; - __s32 len; + __u32 len; __u16 bid; }; @@ -3523,7 +3523,7 @@ static int io_add_buffers(struct io_provide_buf *pbuf, struct io_buffer **head) break; buf->addr = addr; - buf->len = pbuf->len; + buf->len = min_t(__u32, pbuf->len, MAX_RW_COUNT); buf->bid = bid; addr += pbuf->len; bid++; -- 2.25.1

1 0

【Meeting Notice】openEuler kernel 技术分享第六期 & 双周例会 Time: 2021-05-28 14:00-18:00
by Meeting Book 27 May '21

27 May '21

1 0