- Linuxarm - mailweb.openeuler.org

Re: [RFC PATCH v5 4/4] scheduler: Add cluster scheduler level for x86
by Song Bao Hua (Barry Song) 21 Apr '21

21 Apr '21

> -----Original Message----- > From: Tim Chen [mailto:tim.c.chen@linux.intel.com] > Sent: Wednesday, April 21, 2021 6:32 AM > To: Song Bao Hua (Barry Song) <song.bao.hua(a)hisilicon.com>; > catalin.marinas(a)arm.com; will(a)kernel.org; rjw(a)rjwysocki.net; > vincent.guittot(a)linaro.org; bp(a)alien8.de; tglx(a)linutronix.de; > mingo(a)redhat.com; lenb(a)kernel.org; peterz(a)infradead.org; > dietmar.eggemann(a)arm.com; rostedt(a)goodmis.org; bsegall(a)google.com; > mgorman(a)suse.de > Cc: msys.mizuma(a)gmail.com; valentin.schneider(a)arm.com; > gregkh(a)linuxfoundation.org; Jonathan Cameron <jonathan.cameron(a)huawei.com>; > juri.lelli(a)redhat.com; mark.rutland(a)arm.com; sudeep.holla(a)arm.com; > aubrey.li(a)linux.intel.com; linux-arm-kernel(a)lists.infradead.org; > linux-kernel(a)vger.kernel.org; linux-acpi(a)vger.kernel.org; x86(a)kernel.org; > xuwei (O) <xuwei5(a)huawei.com>; Zengtao (B) <prime.zeng(a)hisilicon.com>; > guodong.xu(a)linaro.org; yangyicong <yangyicong(a)huawei.com>; Liguozhu (Kenneth) > <liguozhu(a)hisilicon.com>; linuxarm(a)openeuler.org; hpa(a)zytor.com > Subject: Re: [RFC PATCH v5 4/4] scheduler: Add cluster scheduler level for x86 > > > > On 3/23/21 4:21 PM, Song Bao Hua (Barry Song) wrote: > > >> > >> On 3/18/21 9:16 PM, Barry Song wrote: > >>> From: Tim Chen <tim.c.chen(a)linux.intel.com> > >>> > >>> There are x86 CPU architectures (e.g. Jacobsville) where L2 cahce > >>> is shared among a cluster of cores instead of being exclusive > >>> to one single core. > >>> > >>> To prevent oversubscription of L2 cache, load should be > >>> balanced between such L2 clusters, especially for tasks with > >>> no shared data. > >>> > >>> Also with cluster scheduling policy where tasks are woken up > >>> in the same L2 cluster, we will benefit from keeping tasks > >>> related to each other and likely sharing data in the same L2 > >>> cluster. > >>> > >>> Add CPU masks of CPUs sharing the L2 cache so we can build such > >>> L2 cluster scheduler domain. > >>> > >>> Signed-off-by: Tim Chen <tim.c.chen(a)linux.intel.com> > >>> Signed-off-by: Barry Song <song.bao.hua(a)hisilicon.com> > >> > >> > >> Barry, > >> > >> Can you also add this chunk to the patch. > >> Thanks. > > > > Sure, Tim, Thanks. I'll put that into patch 4/4 in v6. > > > > Barry, > > This chunk will also need to be added to return cluster id for x86. > Please add it in your next rev. Yes. Thanks. I'll put this in either RFC v7 or Patch v1. For spreading path, things are much easier, though packing path is quite tricky. But It seems RFC v6 has been quite close to what we want to achieve to pack related tasks by scanning cluster for tasks within same NUMA: https://lore.kernel.org/lkml/20210420001844.9116-1-song.bao.hua@hisilicon.c… If couples have been already in same LLC(numa), scanning clusters will gather them further. If they are running in different NUMA nodes, the original scanning LLC will move them to the same node, after that, scanning cluster might put them closer to each other. it seems it is kind of the two-level packing Dietmar has suggested. So perhaps we won't have RFC v7, I will probably send patch v1 afterwards. > > Thanks. > > Tim > > --- > > diff --git a/arch/x86/include/asm/topology.h > b/arch/x86/include/asm/topology.h > index 800fa48c9fcd..2548d824f103 100644 > --- a/arch/x86/include/asm/topology.h > +++ b/arch/x86/include/asm/topology.h > @@ -109,6 +109,7 @@ extern const struct cpumask *cpu_clustergroup_mask(int cpu); > #define topology_physical_package_id(cpu) (cpu_data(cpu).phys_proc_id) > #define topology_logical_die_id(cpu) (cpu_data(cpu).logical_die_id) > #define topology_die_id(cpu) (cpu_data(cpu).cpu_die_id) > +#define topology_cluster_id(cpu) (per_cpu(cpu_l2c_id, cpu)) > #define topology_core_id(cpu) (cpu_data(cpu).cpu_core_id) > > extern unsigned int __max_die_per_package; Thanks Barry

1 0

[PATCH net v4 0/2] fix packet stuck problem for lockless qdisc
by Yunsheng Lin 21 Apr '21

21 Apr '21

This patchset fixes the packet stuck problem mentioned in [1]. Patch 1: Add STATE_MISSED flag to fix packet stuck problem. Patch 2: Fix a tx_action rescheduling problem after STATE_MISSED flag is added in patch 1. V4: Change STATE_NEED_RESCHEDULE to STATE_MISSED and add patch 2. [1]. https://lkml.org/lkml/2019/10/9/42 Yunsheng Lin (2): net: sched: fix packet stuck problem for lockless qdisc net: sched: fix endless tx action reschedule during deactivation include/net/pkt_sched.h | 7 +------ include/net/sch_generic.h | 37 ++++++++++++++++++++++++++++++++++++- net/core/dev.c | 26 ++++++++++++++++++++++---- net/sched/sch_generic.c | 16 +++++++++++++++- 4 files changed, 74 insertions(+), 12 deletions(-) -- 2.7.4

2 6

Re: [RFC PATCH 2/3] vfio/hisilicon: register the driver to vfio
by Jason Gunthorpe 20 Apr '21

20 Apr '21

On Tue, Apr 20, 2021 at 09:28:46PM +0800, liulongfang wrote: > >> So, I still don't understand what the security risk you are talking about is, > >> and what do you think the security design should look like? > >> Can you elaborate on it? > > > > Each security domain must have its own PCI BDF. > > > The basic unit to perform the live migration function is the VF, and the basic > function of the VF is the business function of the device. If the live migration > function and the business function are completely separated, and they are bound > to their respective VFs, it will result in the ability to execute the business > in the guest A functional VF cannot perform live migration, and a VF with a live > migration function cannot perform business functions in the guest. > > In fact, the scenario that requires live migration is to migrate its business > functions to the VFs of other hosts when the VF's business functions are heavily > loaded. > This usage scenario itself requires that the VF needs to have these two functions > at the same time. Other vendors are managing, it is not an unsolvable problem. I think these patches can't advance in any form without somehow addressing the isolation issue. Jason

1 0

[PATCH net v3] net: sched: fix packet stuck problem for lockless qdisc
by Yunsheng Lin 20 Apr '21

20 Apr '21

Lockless qdisc has below concurrent problem: cpu0 cpu1 . . q->enqueue . . . qdisc_run_begin() . . . dequeue_skb() . . . sch_direct_xmit() . . . . q->enqueue . qdisc_run_begin() . return and do nothing . . qdisc_run_end() . cpu1 enqueue a skb without calling __qdisc_run() because cpu0 has not released the lock yet and spin_trylock() return false for cpu1 in qdisc_run_begin(), and cpu0 do not see the skb enqueued by cpu1 when calling dequeue_skb() because cpu1 may enqueue the skb after cpu0 calling dequeue_skb() and before cpu0 calling qdisc_run_end(). Lockless qdisc has below another concurrent problem when tx_action is involved: cpu0(serving tx_action) cpu1 cpu2 . . . . q->enqueue . . qdisc_run_begin() . . dequeue_skb() . . . q->enqueue . . . . sch_direct_xmit() . . . qdisc_run_begin() . . return and do nothing . . . clear __QDISC_STATE_SCHED . . qdisc_run_begin() . . return and do nothing . . . . . . qdisc_run_end() . This patch fixes the above data race by: 1. Get the flag before doing spin_trylock(). 2. If the first spin_trylock() return false and the flag is not set before the first spin_trylock(), Set the flag and retry another spin_trylock() in case other CPU may not see the new flag after it releases the lock. 3. reschedule if the flags is set after the lock is released at the end of qdisc_run_end(). For tx_action case, the flags is also set when cpu1 is at the end if qdisc_run_end(), so tx_action will be rescheduled again to dequeue the skb enqueued by cpu2. Only clear the flag before retrying a dequeuing when dequeuing returns NULL in order to reduce the overhead of the above double spin_trylock() and __netif_schedule() calling. The performance impact of this patch, tested using pktgen and dummy netdev with pfifo_fast qdisc attached: threads without+this_patch with+this_patch delta 1 2.61Mpps 2.60Mpps -0.3% 2 3.97Mpps 3.82Mpps -3.7% 4 5.62Mpps 5.59Mpps -0.5% 8 2.78Mpps 2.77Mpps -0.3% 16 2.22Mpps 2.22Mpps -0.0% Fixes: 6b3ba9146fe6 ("net: sched: allow qdiscs to handle locking") Signed-off-by: Yunsheng Lin <linyunsheng(a)huawei.com> --- V3: fix a compile error and a few comment typo, remove the __QDISC_STATE_DEACTIVATED checking, and update the performance data. V2: Avoid the overhead of fixing the data race as much as possible. --- include/net/sch_generic.h | 38 +++++++++++++++++++++++++++++++++++++- net/sched/sch_generic.c | 12 ++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index f7a6e14..e3f46eb 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -36,6 +36,7 @@ struct qdisc_rate_table { enum qdisc_state_t { __QDISC_STATE_SCHED, __QDISC_STATE_DEACTIVATED, + __QDISC_STATE_NEED_RESCHEDULE, }; struct qdisc_size_table { @@ -159,8 +160,38 @@ static inline bool qdisc_is_empty(const struct Qdisc *qdisc) static inline bool qdisc_run_begin(struct Qdisc *qdisc) { if (qdisc->flags & TCQ_F_NOLOCK) { + bool dont_retry = test_bit(__QDISC_STATE_NEED_RESCHEDULE, + &qdisc->state); + + if (spin_trylock(&qdisc->seqlock)) + goto nolock_empty; + + /* If the flag is set before doing the spin_trylock() and + * the above spin_trylock() return false, it means other cpu + * holding the lock will do dequeuing for us, or it wil see + * the flag set after releasing lock and reschedule the + * net_tx_action() to do the dequeuing. + */ + if (dont_retry) + return false; + + /* We could do set_bit() before the first spin_trylock(), + * and avoid doing second spin_trylock() completely, then + * we could have multi cpus doing the set_bit(). Here use + * dont_retry to avoid doing the set_bit() and the second + * spin_trylock(), which has 5% performance improvement than + * doing the set_bit() before the first spin_trylock(). + */ + set_bit(__QDISC_STATE_NEED_RESCHEDULE, + &qdisc->state); + + /* Retry again in case other CPU may not see the new flag + * after it releases the lock at the end of qdisc_run_end(). + */ if (!spin_trylock(&qdisc->seqlock)) return false; + +nolock_empty: WRITE_ONCE(qdisc->empty, false); } else if (qdisc_is_running(qdisc)) { return false; @@ -176,8 +207,13 @@ static inline bool qdisc_run_begin(struct Qdisc *qdisc) static inline void qdisc_run_end(struct Qdisc *qdisc) { write_seqcount_end(&qdisc->running); - if (qdisc->flags & TCQ_F_NOLOCK) + if (qdisc->flags & TCQ_F_NOLOCK) { spin_unlock(&qdisc->seqlock); + + if (unlikely(test_bit(__QDISC_STATE_NEED_RESCHEDULE, + &qdisc->state))) + __netif_schedule(qdisc); + } } static inline bool qdisc_may_bulk(const struct Qdisc *qdisc) diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c index 44991ea..4953430 100644 --- a/net/sched/sch_generic.c +++ b/net/sched/sch_generic.c @@ -640,8 +640,10 @@ static struct sk_buff *pfifo_fast_dequeue(struct Qdisc *qdisc) { struct pfifo_fast_priv *priv = qdisc_priv(qdisc); struct sk_buff *skb = NULL; + bool need_retry = true; int band; +retry: for (band = 0; band < PFIFO_FAST_BANDS && !skb; band++) { struct skb_array *q = band2list(priv, band); @@ -652,6 +654,16 @@ static struct sk_buff *pfifo_fast_dequeue(struct Qdisc *qdisc) } if (likely(skb)) { qdisc_update_stats_at_dequeue(qdisc, skb); + } else if (need_retry && + test_and_clear_bit(__QDISC_STATE_NEED_RESCHEDULE, + &qdisc->state)) { + /* do another enqueuing after clearing the flag to + * avoid calling __netif_schedule(). + */ + smp_mb__after_atomic(); + need_retry = false; + + goto retry; } else { WRITE_ONCE(qdisc->empty, true); } -- 2.7.4

2 6

[RFC PATCH v6 4/4] scheduler: Add cluster scheduler level for x86
by Barry Song 20 Apr '21

20 Apr '21

From: Tim Chen <tim.c.chen(a)linux.intel.com> There are x86 CPU architectures (e.g. Jacobsville) where L2 cahce is shared among a cluster of cores instead of being exclusive to one single core. To prevent oversubscription of L2 cache, load should be balanced between such L2 clusters, especially for tasks with no shared data. Also with cluster scheduling policy where tasks are woken up in the same L2 cluster, we will benefit from keeping tasks related to each other and likely sharing data in the same L2 cluster. Add CPU masks of CPUs sharing the L2 cache so we can build such L2 cluster scheduler domain. Signed-off-by: Tim Chen <tim.c.chen(a)linux.intel.com> Signed-off-by: Barry Song <song.bao.hua(a)hisilicon.com> --- -v6: * added topology_cluster_cpumask() for x86, code provided by Tim. arch/x86/Kconfig | 8 ++++++++ arch/x86/include/asm/smp.h | 7 +++++++ arch/x86/include/asm/topology.h | 2 ++ arch/x86/kernel/cpu/cacheinfo.c | 1 + arch/x86/kernel/cpu/common.c | 3 +++ arch/x86/kernel/smpboot.c | 43 ++++++++++++++++++++++++++++++++++++++++- 6 files changed, 63 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2792879..d597de2 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1002,6 +1002,14 @@ config NR_CPUS This is purely to save memory: each supported CPU adds about 8KB to the kernel image. +config SCHED_CLUSTER + bool "Cluster scheduler support" + default n + help + Cluster scheduler support improves the CPU scheduler's decision + making when dealing with machines that have clusters of CPUs + sharing L2 cache. If unsure say N here. + config SCHED_SMT def_bool y if SMP diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index c0538f8..9cbc4ae 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -16,7 +16,9 @@ DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_die_map); /* cpus sharing the last level cache: */ DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_llc_shared_map); +DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_l2c_shared_map); DECLARE_PER_CPU_READ_MOSTLY(u16, cpu_llc_id); +DECLARE_PER_CPU_READ_MOSTLY(u16, cpu_l2c_id); DECLARE_PER_CPU_READ_MOSTLY(int, cpu_number); static inline struct cpumask *cpu_llc_shared_mask(int cpu) @@ -24,6 +26,11 @@ static inline struct cpumask *cpu_llc_shared_mask(int cpu) return per_cpu(cpu_llc_shared_map, cpu); } +static inline struct cpumask *cpu_l2c_shared_mask(int cpu) +{ + return per_cpu(cpu_l2c_shared_map, cpu); +} + DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_cpu_to_apicid); DECLARE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_acpiid); DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_bios_cpu_apicid); diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 9239399..800fa48 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -103,6 +103,7 @@ static inline void setup_node_to_cpumask_map(void) { } #include <asm-generic/topology.h> extern const struct cpumask *cpu_coregroup_mask(int cpu); +extern const struct cpumask *cpu_clustergroup_mask(int cpu); #define topology_logical_package_id(cpu) (cpu_data(cpu).logical_proc_id) #define topology_physical_package_id(cpu) (cpu_data(cpu).phys_proc_id) @@ -114,6 +115,7 @@ static inline void setup_node_to_cpumask_map(void) { } #ifdef CONFIG_SMP #define topology_die_cpumask(cpu) (per_cpu(cpu_die_map, cpu)) +#define topology_cluster_cpumask(cpu) (cpu_clustergroup_mask(cpu)) #define topology_core_cpumask(cpu) (per_cpu(cpu_core_map, cpu)) #define topology_sibling_cpumask(cpu) (per_cpu(cpu_sibling_map, cpu)) diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinfo.c index 3ca9be4..0d03a71 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -846,6 +846,7 @@ void init_intel_cacheinfo(struct cpuinfo_x86 *c) l2 = new_l2; #ifdef CONFIG_SMP per_cpu(cpu_llc_id, cpu) = l2_id; + per_cpu(cpu_l2c_id, cpu) = l2_id; #endif } diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index ab640ab..0ba282d 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -78,6 +78,9 @@ /* Last level cache ID of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(u16, cpu_llc_id) = BAD_APICID; +/* L2 cache ID of each logical CPU */ +DEFINE_PER_CPU_READ_MOSTLY(u16, cpu_l2c_id) = BAD_APICID; + /* correctly size the local cpu masks */ void __init setup_cpu_local_masks(void) { diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 02813a7..c85ffa8 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -101,6 +101,8 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_llc_shared_map); +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_l2c_shared_map); + /* Per CPU bogomips and other parameters */ DEFINE_PER_CPU_READ_MOSTLY(struct cpuinfo_x86, cpu_info); EXPORT_PER_CPU_SYMBOL(cpu_info); @@ -501,6 +503,21 @@ static bool match_llc(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) return topology_sane(c, o, "llc"); } +static bool match_l2c(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) +{ + int cpu1 = c->cpu_index, cpu2 = o->cpu_index; + + /* Do not match if we do not have a valid APICID for cpu: */ + if (per_cpu(cpu_l2c_id, cpu1) == BAD_APICID) + return false; + + /* Do not match if L2 cache id does not match: */ + if (per_cpu(cpu_l2c_id, cpu1) != per_cpu(cpu_l2c_id, cpu2)) + return false; + + return topology_sane(c, o, "l2c"); +} + /* * Unlike the other levels, we do not enforce keeping a * multicore group inside a NUMA node. If this happens, we will @@ -522,7 +539,7 @@ static bool match_die(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) } -#if defined(CONFIG_SCHED_SMT) || defined(CONFIG_SCHED_MC) +#if defined(CONFIG_SCHED_SMT) || defined(CONFIG_SCHED_CLUSTER) || defined(CONFIG_SCHED_MC) static inline int x86_sched_itmt_flags(void) { return sysctl_sched_itmt_enabled ? SD_ASYM_PACKING : 0; @@ -540,12 +557,21 @@ static int x86_smt_flags(void) return cpu_smt_flags() | x86_sched_itmt_flags(); } #endif +#ifdef CONFIG_SCHED_CLUSTER +static int x86_cluster_flags(void) +{ + return cpu_cluster_flags() | x86_sched_itmt_flags(); +} +#endif #endif static struct sched_domain_topology_level x86_numa_in_package_topology[] = { #ifdef CONFIG_SCHED_SMT { cpu_smt_mask, x86_smt_flags, SD_INIT_NAME(SMT) }, #endif +#ifdef CONFIG_SCHED_CLUSTER + { cpu_clustergroup_mask, x86_cluster_flags, SD_INIT_NAME(CLS) }, +#endif #ifdef CONFIG_SCHED_MC { cpu_coregroup_mask, x86_core_flags, SD_INIT_NAME(MC) }, #endif @@ -556,6 +582,9 @@ static int x86_smt_flags(void) #ifdef CONFIG_SCHED_SMT { cpu_smt_mask, x86_smt_flags, SD_INIT_NAME(SMT) }, #endif +#ifdef CONFIG_SCHED_CLUSTER + { cpu_clustergroup_mask, x86_cluster_flags, SD_INIT_NAME(CLS) }, +#endif #ifdef CONFIG_SCHED_MC { cpu_coregroup_mask, x86_core_flags, SD_INIT_NAME(MC) }, #endif @@ -583,6 +612,7 @@ void set_cpu_sibling_map(int cpu) if (!has_mp) { cpumask_set_cpu(cpu, topology_sibling_cpumask(cpu)); cpumask_set_cpu(cpu, cpu_llc_shared_mask(cpu)); + cpumask_set_cpu(cpu, cpu_l2c_shared_mask(cpu)); cpumask_set_cpu(cpu, topology_core_cpumask(cpu)); cpumask_set_cpu(cpu, topology_die_cpumask(cpu)); c->booted_cores = 1; @@ -598,6 +628,8 @@ void set_cpu_sibling_map(int cpu) if ((i == cpu) || (has_mp && match_llc(c, o))) link_mask(cpu_llc_shared_mask, cpu, i); + if ((i == cpu) || (has_mp && match_l2c(c, o))) + link_mask(cpu_l2c_shared_mask, cpu, i); } /* @@ -649,6 +681,11 @@ const struct cpumask *cpu_coregroup_mask(int cpu) return cpu_llc_shared_mask(cpu); } +const struct cpumask *cpu_clustergroup_mask(int cpu) +{ + return cpu_l2c_shared_mask(cpu); +} + static void impress_friends(void) { int cpu; @@ -1332,6 +1369,7 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) zalloc_cpumask_var(&per_cpu(cpu_core_map, i), GFP_KERNEL); zalloc_cpumask_var(&per_cpu(cpu_die_map, i), GFP_KERNEL); zalloc_cpumask_var(&per_cpu(cpu_llc_shared_map, i), GFP_KERNEL); + zalloc_cpumask_var(&per_cpu(cpu_l2c_shared_map, i), GFP_KERNEL); } /* @@ -1556,7 +1594,10 @@ static void remove_siblinginfo(int cpu) cpumask_clear_cpu(cpu, topology_sibling_cpumask(sibling)); for_each_cpu(sibling, cpu_llc_shared_mask(cpu)) cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling)); + for_each_cpu(sibling, cpu_l2c_shared_mask(cpu)) + cpumask_clear_cpu(cpu, cpu_l2c_shared_mask(sibling)); cpumask_clear(cpu_llc_shared_mask(cpu)); + cpumask_clear(cpu_l2c_shared_mask(cpu)); cpumask_clear(topology_sibling_cpumask(cpu)); cpumask_clear(topology_core_cpumask(cpu)); cpumask_clear(topology_die_cpumask(cpu)); -- 1.8.3.1

1 0

[PATCH net-next 0/3] net: hns3: misc updates for -next
by Huazhong Tan 20 Apr '21

20 Apr '21

This series includes some misc updates for the HNS3 ethernet driver. Huazhong Tan (3): net: hns3: remove a duplicate pf reset counting net: hns3: cleanup inappropriate spaces in struct hlcgevf_tqp_stats net: hns3: change the value of the SEPARATOR_VALUE macro in hclgevf_main.c drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c | 2 -- drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.c | 2 +- drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.h | 4 ++-- 3 files changed, 3 insertions(+), 5 deletions(-) -- 2.7.4

2 4

Re: [PATCH V4] app/testpmd: support Tx mbuf free on demand cmd
by Ferruh Yigit 19 Apr '21

19 Apr '21

On 4/19/2021 1:36 PM, Lijun Ou wrote: > From: Chengwen Feng <fengchengwen(a)huawei.com> > > This patch support tx_done_cleanup command: > tx_done_cleanup port (port_id) (queue_id) (free_cnt) > Instead of creating a new root level, 'tx_done_cleanup' command, what do you think to use existing port command, something like: "port cleanup <port_id> txq <queue_id> <free_cnt>" > Users must make sure there are no concurrent access to the same Tx > queue (like rte_eth_tx_burst, rte_eth_dev_tx_queue_stop and so on) > this command executed. > > Signed-off-by: Chengwen Feng <fengchengwen(a)huawei.com> > Signed-off-by: Lijun Ou <oulijun(a)huawei.com> > --- > V3->V4: > - revert the V3 scheme. > > V2->V3: > - The command implementation is changed so that the queuestate does > not depend on the command execution. > > V1->V2: > - use Tx instead of TX > - add note in doc > --- > app/test-pmd/cmdline.c | 84 +++++++++++++++++++++++++++++ > doc/guides/rel_notes/release_21_05.rst | 2 + > doc/guides/testpmd_app_ug/testpmd_funcs.rst | 11 ++++ > 3 files changed, 97 insertions(+) > > diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c > index 5bf1497..b43964d 100644 > --- a/app/test-pmd/cmdline.c > +++ b/app/test-pmd/cmdline.c > @@ -675,6 +675,9 @@ static void cmd_help_long_parsed(void *parsed_result, > "set port (port_id) ptype_mask (ptype_mask)\n" > " set packet types classification for a specific port\n\n" > > + "tx_done_cleanup (port_id) (queue_id) (free_cnt)\n" > + " Cleanup a Tx queue's mbuf on a port\n\n" > + If 'free_cnt' is '0', driver should free all possible mbufs, can you add this detail to help? > "set port (port_id) queue-region region_id (value) " > "queue_start_index (value) queue_num (value)\n" > " Set a queue region on a port\n\n" > @@ -16915,6 +16918,86 @@ cmdline_parse_inst_t cmd_showport_macs = { > }, > }; > > +/* *** tx_done_cleanup *** */ > +struct cmd_tx_done_cleanup_result { > + cmdline_fixed_string_t clean; > + cmdline_fixed_string_t port; > + uint16_t port_id; > + uint16_t queue_id; > + uint32_t free_cnt; > +}; > + > +static void > +cmd_tx_done_cleanup_parsed(void *parsed_result, > + __rte_unused struct cmdline *cl, > + __rte_unused void *data) > +{ > + struct cmd_tx_done_cleanup_result *res = parsed_result; > + uint16_t port_id = res->port_id; > + uint16_t queue_id = res->queue_id; > + uint32_t free_cnt = res->free_cnt; > + struct rte_eth_txq_info qinfo; > + int ret; > + > + if (rte_eth_tx_queue_info_get(port_id, queue_id, &qinfo)) { > + printf("Failed to get port %u Tx queue %u info!\n", > + port_id, queue_id); > + return; > + } > + > + if (qinfo.queue_state != RTE_ETH_QUEUE_STATE_STARTED) { > + printf("TX queue %u not started!\n", queue_id); > + return; > + } > + > + /* > + * rte_eth_tx_done_cleanup is a dataplane API, user must make sure > + * there are no concurrent access to the same Tx queue (like > + * rte_eth_tx_burst, rte_eth_dev_tx_queue_stop and so on) when this API > + * called. > + */ Should there be a check to see if testpmd forwarding is stared or not? This API shouldn't be allowed when forwarding is started. > + ret = rte_eth_tx_done_cleanup(port_id, queue_id, free_cnt); > + if (ret < 0) { > + printf("Failed to cleanup mbuf for port %u Tx queue %u " > + "error desc: %s(%d)\n", > + port_id, queue_id, strerror(-ret), ret); > + return; > + } > + > + printf("Cleanup port %u Tx queue %u mbuf nums: %u\n", > + port_id, queue_id, ret); > +} > + > +cmdline_parse_token_string_t cmd_tx_done_cleanup_clean = > + TOKEN_STRING_INITIALIZER(struct cmd_tx_done_cleanup_result, clean, > + "tx_done_cleanup"); > +cmdline_parse_token_string_t cmd_tx_done_cleanup_port = > + TOKEN_STRING_INITIALIZER(struct cmd_tx_done_cleanup_result, port, > + "port"); > +cmdline_parse_token_num_t cmd_tx_done_cleanup_port_id = > + TOKEN_NUM_INITIALIZER(struct cmd_tx_done_cleanup_result, port_id, > + UINT16); > +cmdline_parse_token_num_t cmd_tx_done_cleanup_queue_id = > + TOKEN_NUM_INITIALIZER(struct cmd_tx_done_cleanup_result, queue_id, > + UINT16); > +cmdline_parse_token_num_t cmd_tx_done_cleanup_free_cnt = > + TOKEN_NUM_INITIALIZER(struct cmd_tx_done_cleanup_result, free_cnt, > + UINT32); > + > +cmdline_parse_inst_t cmd_tx_done_cleanup = { > + .f = cmd_tx_done_cleanup_parsed, > + .data = NULL, > + .help_str = "tx_done_cleanup port <port_id> <queue_id> <free_cnt>", > + .tokens = { > + (void *)&cmd_tx_done_cleanup_clean, > + (void *)&cmd_tx_done_cleanup_port, > + (void *)&cmd_tx_done_cleanup_port_id, > + (void *)&cmd_tx_done_cleanup_queue_id, > + (void *)&cmd_tx_done_cleanup_free_cnt, > + NULL, > + }, > +}; > + > /* ******************************************************************************** */ > > /* list of instructions */ > @@ -17040,6 +17123,7 @@ cmdline_parse_ctx_t main_ctx[] = { > (cmdline_parse_inst_t *)&cmd_config_rss_reta, > (cmdline_parse_inst_t *)&cmd_showport_reta, > (cmdline_parse_inst_t *)&cmd_showport_macs, > + (cmdline_parse_inst_t *)&cmd_tx_done_cleanup, > (cmdline_parse_inst_t *)&cmd_config_burst, > (cmdline_parse_inst_t *)&cmd_config_thresh, > (cmdline_parse_inst_t *)&cmd_config_threshold, > diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst > index 4538bb9..554499f 100644 > --- a/doc/guides/rel_notes/release_21_05.rst > +++ b/doc/guides/rel_notes/release_21_05.rst > @@ -217,6 +217,8 @@ New Features > ``show port (port_id) rxq (queue_id) desc used count`` > * Added command to dump internal representation information of single flow. > ``flow dump (port_id) rule (rule_id)`` > + * Added command to cleanup a Tx queue's mbuf on a port. > + ``tx_done_cleanup port <port_id> <queue_id> <free_cnt>`` > > * **Updated ipsec-secgw sample application.** > > diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst > index a736e7d..6472647 100644 > --- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst > +++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst > @@ -270,6 +270,17 @@ and ready to be processed by the driver on a given RX queue:: > > testpmd> show port (port_id) rxq (queue_id) desc used count > > +cleanup txq mbufs > +~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Request the driver to free mbufs currently cached by the driver for a given port's > +Tx queue:: > + testpmd> tx_done_cleanup port (port_id) (queue_id) (free_cnt) > + > +.. note:: > + This command is dangerous, users must make sure there are no cucurrent access to > + the same Tx queue (link rte_eth_tx_burst, rte_eth_dev_tx_queue_stop and so on). > + > show config > ~~~~~~~~~~~ > >

1 0

[PATCH] app/testpmd: support Tx mbuf free on demand cmd
by Lijun Ou 19 Apr '21

19 Apr '21

From: Chengwen Feng <fengchengwen(a)huawei.com> This patch support tx_done_cleanup command: tx_done_cleanup port (port_id) (queue_id) (free_cnt) User must make sure there are no concurrent access to the same Tx queue (like rte_eth_tx_burst, rte_eth_dev_tx_queue_stop and so on) when this command executed. Signed-off-by: Chengwen Feng <fengchengwen(a)huawei.com> Signed-off-by: Lijun Ou <oulijun(a)huawei.com> --- app/test-pmd/cmdline.c | 91 +++++++++++++++++++++++++++++ doc/guides/rel_notes/release_21_05.rst | 2 + doc/guides/testpmd_app_ug/testpmd_funcs.rst | 7 +++ 3 files changed, 100 insertions(+) diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index 14110eb..832ae70 100644 --- a/app/test-pmd/cmdline.c +++ b/app/test-pmd/cmdline.c @@ -36,6 +36,7 @@ #include <rte_pci.h> #include <rte_ether.h> #include <rte_ethdev.h> +#include <rte_ethdev_driver.h> #include <rte_string_fns.h> #include <rte_devargs.h> #include <rte_flow.h> @@ -675,6 +676,9 @@ static void cmd_help_long_parsed(void *parsed_result, "set port (port_id) ptype_mask (ptype_mask)\n" " set packet types classification for a specific port\n\n" + "tx_done_cleanup (port_id) (queue_id) (free_cnt)\n" + " Cleanup a tx queue's mbuf on a port\n\n" + "set port (port_id) queue-region region_id (value) " "queue_start_index (value) queue_num (value)\n" " Set a queue region on a port\n\n" @@ -16910,6 +16914,92 @@ cmdline_parse_inst_t cmd_showport_macs = { }, }; +/* *** tx_done_cleanup *** */ +struct cmd_tx_done_cleanup_result { + cmdline_fixed_string_t clean; + cmdline_fixed_string_t port; + uint16_t port_id; + uint16_t queue_id; + uint32_t free_cnt; +}; + +static void +cmd_tx_done_cleanup_parsed(void *parsed_result, + __rte_unused struct cmdline *cl, + __rte_unused void *data) +{ + struct cmd_tx_done_cleanup_result *res = parsed_result; + struct rte_eth_dev *dev; + uint16_t port_id = res->port_id; + uint16_t queue_id = res->queue_id; + uint32_t free_cnt = res->free_cnt; + int ret; + + if (!rte_eth_dev_is_valid_port(port_id)) { + printf("Invalid port_id %u\n", port_id); + return; + } + + dev = &rte_eth_devices[port_id]; + if (queue_id >= dev->data->nb_tx_queues) { + printf("Invalid TX queue_id %u\n", queue_id); + return; + } + + if (dev->data->tx_queue_state[queue_id] != + RTE_ETH_QUEUE_STATE_STARTED) { + printf("TX queue_id %u not started!\n", queue_id); + return; + } + + /* + * rte_eth_tx_done_cleanup is a dataplane API, user must make sure + * there are no concurrent access to the same Tx queue (like + * rte_eth_tx_burst, rte_eth_dev_tx_queue_stop and so on) when this API + * called. + */ + ret = rte_eth_tx_done_cleanup(port_id, queue_id, free_cnt); + if (ret < 0) { + printf("Failed to cleanup mbuf for port %u TX queue %u " + "error desc: %s(%d)\n", + port_id, queue_id, strerror(-ret), ret); + return; + } + + printf("Cleanup port %u TX queue %u mbuf nums: %u\n", + port_id, queue_id, ret); +} + +cmdline_parse_token_string_t cmd_tx_done_cleanup_clean = + TOKEN_STRING_INITIALIZER(struct cmd_tx_done_cleanup_result, clean, + "tx_done_cleanup"); +cmdline_parse_token_string_t cmd_tx_done_cleanup_port = + TOKEN_STRING_INITIALIZER(struct cmd_tx_done_cleanup_result, port, + "port"); +cmdline_parse_token_num_t cmd_tx_done_cleanup_port_id = + TOKEN_NUM_INITIALIZER(struct cmd_tx_done_cleanup_result, port_id, + UINT16); +cmdline_parse_token_num_t cmd_tx_done_cleanup_queue_id = + TOKEN_NUM_INITIALIZER(struct cmd_tx_done_cleanup_result, queue_id, + UINT16); +cmdline_parse_token_num_t cmd_tx_done_cleanup_free_cnt = + TOKEN_NUM_INITIALIZER(struct cmd_tx_done_cleanup_result, free_cnt, + UINT32); + +cmdline_parse_inst_t cmd_tx_done_cleanup = { + .f = cmd_tx_done_cleanup_parsed, + .data = NULL, + .help_str = "tx_done_cleanup port <port_id> <queue_id> <free_cnt>", + .tokens = { + (void *)&cmd_tx_done_cleanup_clean, + (void *)&cmd_tx_done_cleanup_port, + (void *)&cmd_tx_done_cleanup_port_id, + (void *)&cmd_tx_done_cleanup_queue_id, + (void *)&cmd_tx_done_cleanup_free_cnt, + NULL, + }, +}; + /* ******************************************************************************** */ /* list of instructions */ @@ -17035,6 +17125,7 @@ cmdline_parse_ctx_t main_ctx[] = { (cmdline_parse_inst_t *)&cmd_config_rss_reta, (cmdline_parse_inst_t *)&cmd_showport_reta, (cmdline_parse_inst_t *)&cmd_showport_macs, + (cmdline_parse_inst_t *)&cmd_tx_done_cleanup, (cmdline_parse_inst_t *)&cmd_config_burst, (cmdline_parse_inst_t *)&cmd_config_thresh, (cmdline_parse_inst_t *)&cmd_config_threshold, diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst index 23f7f0b..8077573 100644 --- a/doc/guides/rel_notes/release_21_05.rst +++ b/doc/guides/rel_notes/release_21_05.rst @@ -69,6 +69,8 @@ New Features * Added command to display Rx queue used descriptor count. ``show port (port_id) rxq (queue_id) desc used count`` + * Added command to cleanup a Tx queue's mbuf on a port. + ``tx_done_cleanup port <port_id> <queue_id> <free_cnt>`` Removed Items diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst index f59eb8a..39281f5 100644 --- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst +++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst @@ -272,6 +272,13 @@ and ready to be processed by the driver on a given RX queue:: testpmd> show port (port_id) rxq (queue_id) desc used count +cleanup txq mbufs +~~~~~~~~~~~~~~~~~~~~~~~~ + +Request the driver to free mbufs currently cached by the driver for a given port's +Tx queue:: + testpmd> tx_done_cleanup port (port_id) (queue_id) (free_cnt) + show config ~~~~~~~~~~~ -- 2.7.4

7 22

Re: [RFC PATCH 2/3] vfio/hisilicon: register the driver to vfio
by Jason Gunthorpe 19 Apr '21

19 Apr '21

On Mon, Apr 19, 2021 at 08:24:40PM +0800, liulongfang wrote: > > I'm also confused how this works securely at all, as a general rule a > > VFIO PCI driver cannot access the MMIO memory of the function it is > > planning to assign to the guest. There is a lot of danger that the > > guest could access that MMIO space one way or another. > > VF's MMIO memory is divided into two parts, one is the guest part, > and the other is the live migration part. They do not affect each other, > so there is no security problem. AFAIK there are several scenarios where a guest can access this MMIO memory using DMA even if it is not mapped into the guest for CPU access. > If pci_release_mem_regions() is not added here, > The guests using pci_request_mem_regions() will return an error. > Then, the guest will not be able to obtain the MMIO address of the VF. Which is why VFIO has this protection to prevent sharing MMIO regions on the VF assigned to the guest Jason

1 0

Re: [PATCH v7] ethdev: add queue state when retrieve queue information
by Thomas Monjalon 19 Apr '21

19 Apr '21

19/04/2021 10:57, Lijun Ou: > Currently, upper-layer application could get queue state only > through pointers such as dev->data->tx_queue_state[queue_id], > this is not the recommended way to access it. So this patch > add get queue state when call rte_eth_rx_queue_info_get and > rte_eth_tx_queue_info_get API. > > Note: After add queue_state field, the 'struct rte_eth_rxq_info' size > remains 128B, and the 'struct rte_eth_txq_info' size remains 64B, so > it could be ABI compatible. > > Signed-off-by: Chengwen Feng <fengchengwen(a)huawei.com> > Signed-off-by: Lijun Ou <oulijun(a)huawei.com> > Acked-by: Konstantin Ananyev <konstantin.ananyev(a)intel.com> Acked-by: Thomas Monjalon <thomas(a)monjalon.net>

2 1