January 2021 - Linuxarm - mailweb.openeuler.org

[RFC PATCH v3 0/2] scheduler: expose the topology of clusters and add cluster scheduler
by Barry Song 14 Apr '21

14 Apr '21

ARM64 server chip Kunpeng 920 has 6 clusters in each NUMA node, and each cluster has 4 cpus. All clusters share L3 cache data while each cluster has local L3 tag. On the other hand, each cluster will share some internal system bus. This means cache is much more affine inside one cluster than across clusters. +-----------------------------------+ +---------+ | +------+ +------+ +---------------------------+ | | | CPU0 | | cpu1 | | +-----------+ | | | +------+ +------+ | | | | | | +----+ L3 | | | | +------+ +------+ cluster | | tag | | | | | CPU2 | | CPU3 | | | | | | | +------+ +------+ | +-----------+ | | | | | | +-----------------------------------+ | | +-----------------------------------+ | | | +------+ +------+ +--------------------------+ | | | | | | | +-----------+ | | | +------+ +------+ | | | | | | | | L3 | | | | +------+ +------+ +----+ tag | | | | | | | | | | | | | | +------+ +------+ | +-----------+ | | | | | | +-----------------------------------+ | L3 | | data | +-----------------------------------+ | | | +------+ +------+ | +-----------+ | | | | | | | | | | | | | +------+ +------+ +----+ L3 | | | | | | tag | | | | +------+ +------+ | | | | | | | | | | ++ +-----------+ | | | +------+ +------+ |---------------------------+ | +-----------------------------------| | | +-----------------------------------| | | | +------+ +------+ +---------------------------+ | | | | | | | +-----------+ | | | +------+ +------+ | | | | | | +----+ L3 | | | | +------+ +------+ | | tag | | | | | | | | | | | | | | +------+ +------+ | +-----------+ | | | | | | +-----------------------------------+ | | +-----------------------------------+ | | | +------+ +------+ +--------------------------+ | | | | | | | +-----------+ | | | +------+ +------+ | | | | | Through the following small program, you can see the performance impact of running it in one cluster and across two clusters: struct foo { int x; int y; } f; void *thread1_fun(void *param) { int s = 0; for (int i = 0; i < 0xfffffff; i++) s += f.x; } void *thread2_fun(void *param) { int s = 0; for (int i = 0; i < 0xfffffff; i++) f.y++; } int main(int argc, char **argv) { pthread_t tid1, tid2; pthread_create(&tid1, NULL, thread1_fun, NULL); pthread_create(&tid2, NULL, thread2_fun, NULL); pthread_join(tid1, NULL); pthread_join(tid2, NULL); } While running this program in one cluster, it takes: $ time taskset -c 0,1 ./a.out real 0m0.832s user 0m1.649s sys 0m0.004s As a contrast, it takes much more time if we run the same program in two clusters: $ time taskset -c 0,4 ./a.out real 0m1.133s user 0m1.960s sys 0m0.000s 0.832/1.133 = 73%, it is a huge difference. Also, hackbench running on 4 cpus in single one cluster and 4 cpus in different clusters also shows a large contrast: * inside a cluster: root@ubuntu:~# taskset -c 0,1,2,3 hackbench -p -T -l 20000 -g 1 Running in threaded mode with 1 groups using 40 file descriptors each (== 40 tasks) Each sender will pass 20000 messages of 100 bytes Time: 4.285 * across clusters: root@ubuntu:~# taskset -c 0,4,8,12 hackbench -p -T -l 20000 -g 1 Running in threaded mode with 1 groups using 40 file descriptors each (== 40 tasks) Each sender will pass 20000 messages of 100 bytes Time: 5.524 The score is 4.285 vs 5.524, shorter time means better performance. All these testing implies that we should let the Linux scheduler use this topology to make better load balancing and WAKE_AFFINE decisions. However, the current scheduler totally has no idea of clusters. This patchset exposed the cluster topology first, then added the sched domain for cluster. While it is named as "cluster", architectures and machines can define the exact meaning of cluster as long as they have some resources sharing under llc and they can leverage the affinity of this resource to achive better scheduling performance. -v3: - rebased againest 5.11-rc2 - with respect to the comments of Valentin Schneider, Peter Zijlstra, Vincent Guittot and Mel Gorman etc. * moved the scheduler changes from arm64 to the common place for all architectures. * added SD_SHARE_CLS_RESOURCES sd_flags specifying the sched_domain where select_idle_cpu() should begin to scan from * removed redundant select_idle_cluster() function since all code is in select_idle_cpu() now. it also avoided scanning cluster cpus twice in v2 code; * redo the hackbench in one numa after the above changes Valentin suggested that select_idle_cpu() could begin to scan from domain with SD_SHARE_PKG_RESOURCES. Changing like this might be too aggressive and limit the spreading of tasks. Thus, this patch lets the architectures and machines to decide where to start by adding a new SD_SHARE_CLS_RESOURCES. Barry Song (1): scheduler: add scheduler level for clusters Jonathan Cameron (1): topology: Represent clusters of CPUs within a die. Documentation/admin-guide/cputopology.rst | 26 +++++++++++--- arch/arm64/Kconfig | 7 ++++ arch/arm64/kernel/topology.c | 2 ++ drivers/acpi/pptt.c | 60 +++++++++++++++++++++++++++++++ drivers/base/arch_topology.c | 14 ++++++++ drivers/base/topology.c | 10 ++++++ include/linux/acpi.h | 5 +++ include/linux/arch_topology.h | 5 +++ include/linux/sched/sd_flags.h | 9 +++++ include/linux/sched/topology.h | 7 ++++ include/linux/topology.h | 13 +++++++ kernel/sched/fair.c | 27 ++++++++++---- kernel/sched/topology.c | 6 ++++ 13 files changed, 181 insertions(+), 10 deletions(-) -- 2.7.4

7 19

[PATCH v2 0/2] EDAC/ghes: Add EDAC device for reporting the CPU cache error count
by Shiju Jose 17 Mar '21

17 Mar '21

CPU cache corrected errors are detected occasionally on few of our ARM64 hardware boards. Though it is rare, the probability of the CPU cache errors frequently occurring can't be avoided. The earlier failure detection by monitoring the cache corrected errors for the frequent occurrences and taking preventive action could prevent more serious hardware faults. On Intel architectures, cache corrected errors are reported and the affected cores are offlined in the architecture specific method. http://www.mcelog.org/cache.html However for the firmware-first error reporting, specifically on ARM64 architecture, there is no provision present for reporting the cache corrected error count to the user-space and taking preventive action such as offline the affected cores. For this purpose, it was suggested to create the CPU EDAC device for the CPU caches for reporting the cache error count for the firmware-first error reporting. User-space application could monitor the recorded corrected error count for the earlier hardware failure detection and could take preventive action, such as offline the corresponding CPU core/s. Changes: RFC V1 -> RFC V2: 1. Fixed feedback by Boris. 1.1. Added reason of this patch. 1.2. Changed CPU errors to CPU cache errors in the drivers/edac/Kconfig 1.3 Changed EDAC cache list to percpu variables. 1.4 Changed configuration depends on ARM64. 1.5. Moved discovery of cacheinfo to ghes_scan_system(). 2. Changes in the descriptions. Shiju Jose (2): EDAC/ghes: Add EDAC device for reporting the CPU cache errors ACPI / APEI: Add reporting ARM64 CPU cache corrected error count Documentation/ABI/testing/sysfs-devices-edac | 15 ++ drivers/acpi/apei/ghes.c | 76 +++++++- drivers/edac/Kconfig | 12 ++ drivers/edac/ghes_edac.c | 186 +++++++++++++++++++ include/acpi/ghes.h | 27 +++ include/linux/cper.h | 4 + 6 files changed, 316 insertions(+), 4 deletions(-) -- 2.17.1

3 5

Re: [PATCH] lib/logic_pio: Fix overlap check for pio registery
by John Garry 15 Mar '21

15 Mar '21

On 21/12/2020 13:04, Jiahui Cen wrote: >> On 21/12/2020 03:24, Jiahui Cen wrote: >>> Hi John, >>> >>> On 2020/12/18 18:40, John Garry wrote: >>>> On 18/12/2020 06:23, Jiahui Cen wrote: >>>>> Since the [start, end) is a half-open interval, a range with the end equal >>>>> to the start of another range should not be considered as overlapped. >>>>> >>>>> Signed-off-by: Jiahui Cen<cenjiahui(a)huawei.com> >>>>> --- >>>>> lib/logic_pio.c | 2 +- >>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>> >>>>> diff --git a/lib/logic_pio.c b/lib/logic_pio.c >>>>> index f32fe481b492..445d611f1dc1 100644 >>>>> --- a/lib/logic_pio.c >>>>> +++ b/lib/logic_pio.c >>>>> @@ -57,7 +57,7 @@ int logic_pio_register_range(struct logic_pio_hwaddr *new_range) >>>>> new_range->flags == LOGIC_PIO_CPU_MMIO) { >>>>> /* for MMIO ranges we need to check for overlap */ >>>>> if (start >= range->hw_start + range->size || >>>>> - end < range->hw_start) { >>>>> + end <= range->hw_start) { >>>> It looks like your change is correct, but should not really have an impact in practice since: >>>> a: BIOSes generally list ascending IO port CPU addresses >>>> b. there is space between IO port CPU address regions >>>> >>>> Have you seen a problem here? >>>> >>> No serious problem. I found it just when I was working on adding support of >>> pci expander bridge for Arm in QEMU. I found the IO window of some extended >>> root bus could not be registered when I inserted the extended buses' _CRS >>> info into DSDT table in the x86 way, which does not sort the buses. >>> >>> Though root buses should be sorted in QEMU, would it be better to accept >>> those non-ascending IO windows? >>> >> ok, so it seems that you have seen a real problem, and this issue is not just detected by code analysis. >> >>> BTW, for b, it seems to be no space between IO windows of different root buses >>> generated by EDK2. Or maybe I missed something obvious. >> I don't know about that. Anyway, your change looks ok. >> >> Reviewed-by: John Garry<john.garry(a)huawei.com> >> >> BTW, for your virt env, will there be requirement to unregister PCI MMIO ranges? Currently we don't see that in non-virt world. >> > Thanks for your review. > > And currently there is no such a requirement in my virt env. > I am not sure what happened to this patch, but I plan on sending some patches in this area soon - do you want me to include this one? Thanks, John

2 2

[RFC V2] app/testpmd: support multi-process
by Lijun Ou 05 Mar '21

05 Mar '21

This patch adds multi-process support for testpmd. The test cmd example as follows: the primary cmd: ./testpmd -w xxx --file-prefix=xx -l 0-1 -n 2 -- -i\ --rxq=16 --txq=16 --num-procs=2 --proc-id=0 the secondary cmd: ./testpmd -w xxx --file-prefix=xx -l 2-3 -n 2 -- -i\ --rxq=16 --txq=16 --num-procs=2 --proc-id=1 Signed-off-by: Min Hu (Connor) <humin29(a)huawei.com> Signed-off-by: Lijun Ou <oulijun(a)huawei.com> --- app/test-pmd/cmdline.c | 6 ++- app/test-pmd/config.c | 9 +++- app/test-pmd/parameters.c | 9 ++++ app/test-pmd/testpmd.c | 135 +++++++++++++++++++++++++++++----------------- app/test-pmd/testpmd.h | 7 +++ 5 files changed, 114 insertions(+), 52 deletions(-) diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index 89034c8..48af5cd 100644 --- a/app/test-pmd/cmdline.c +++ b/app/test-pmd/cmdline.c @@ -71,8 +71,6 @@ #include "cmdline_tm.h" #include "bpf_cmd.h" -static struct cmdline *testpmd_cl; - static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue); /* *** Help command with introduction. *** */ @@ -17124,6 +17122,10 @@ prompt(void) if (testpmd_cl == NULL) return; cmdline_interact(testpmd_cl); + if (unlikely(f_quit == 1)) { + dup2(testpmd_fd_copy, testpmd_cl->s_in); + close(testpmd_fd_copy); + } cmdline_stdin_exit(testpmd_cl); } diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c index 0e2b9f7..f065008 100644 --- a/app/test-pmd/config.c +++ b/app/test-pmd/config.c @@ -3100,6 +3100,8 @@ rss_fwd_config_setup(void) queueid_t rxq; queueid_t nb_q; streamid_t sm_id; + int start; + int end; nb_q = nb_rxq; if (nb_q > nb_txq) @@ -3117,7 +3119,10 @@ rss_fwd_config_setup(void) init_fwd_streams(); setup_fwd_config_of_each_lcore(&cur_fwd_config); - rxp = 0; rxq = 0; + start = proc_id * nb_q / num_procs; + end = start + nb_q / num_procs; + rxp = 0; + rxq = start; for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) { struct fwd_stream *fs; @@ -3134,6 +3139,8 @@ rss_fwd_config_setup(void) continue; rxp = 0; rxq++; + if (rxq >= end) + rxq = start; } } diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index df5eb10..ac63854 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -45,6 +45,8 @@ #include <rte_flow.h> #include "testpmd.h" +#define PARAM_PROC_ID "proc-id" +#define PARAM_NUM_PROCS "num-procs" static void usage(char* progname) @@ -603,6 +605,8 @@ launch_args_parse(int argc, char** argv) { "rx-mq-mode", 1, 0, 0 }, { "record-core-cycles", 0, 0, 0 }, { "record-burst-stats", 0, 0, 0 }, + { PARAM_NUM_PROCS, 1, 0, 0 }, + { PARAM_PROC_ID, 1, 0, 0 }, { 0, 0, 0, 0 }, }; @@ -1356,6 +1360,11 @@ launch_args_parse(int argc, char** argv) record_core_cycles = 1; if (!strcmp(lgopts[opt_idx].name, "record-burst-stats")) record_burst_stats = 1; + + if (strncmp(lgopts[opt_idx].name, PARAM_NUM_PROCS, 8) == 0) + num_procs = atoi(optarg); + if (strncmp(lgopts[opt_idx].name, PARAM_PROC_ID, 7) == 0) + proc_id = atoi(optarg); break; case 'h': usage(argv[0]); diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index c256e71..3abd080 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -63,6 +63,8 @@ #include "testpmd.h" +int testpmd_fd_copy = 500; /* the copy of STDIN_FILENO */ + #ifndef MAP_HUGETLB /* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */ #define HUGE_FLAG (0x40000) @@ -125,6 +127,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS]; */ uint8_t rxring_numa[RTE_MAX_ETHPORTS]; +int proc_id = 0; +unsigned num_procs = 1; + /* * Store specified sockets on which TX ring to be used by ports * is allocated. @@ -978,16 +983,26 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf, /* wrapper to rte_mempool_create() */ TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n", rte_mbuf_best_mempool_ops()); - rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf, - mb_mempool_cache, 0, mbuf_seg_size, socket_id); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_mp = rte_pktmbuf_pool_create(pool_name, + nb_mbuf, mb_mempool_cache, 0, + mbuf_seg_size, socket_id); + else + rte_mp = rte_mempool_lookup(pool_name); + break; } case MP_ALLOC_ANON: { - rte_mp = rte_mempool_create_empty(pool_name, nb_mbuf, - mb_size, (unsigned int) mb_mempool_cache, - sizeof(struct rte_pktmbuf_pool_private), - socket_id, mempool_flags); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_mp = rte_mempool_create_empty(pool_name, + nb_mbuf, mb_size, + (unsigned int)mb_mempool_cache, + sizeof(struct rte_pktmbuf_pool_private), + socket_id, mempool_flags); + else + rte_mp = rte_mempool_lookup(pool_name); + if (rte_mp == NULL) goto err; @@ -1017,9 +1032,13 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf, TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n", rte_mbuf_best_mempool_ops()); - rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf, - mb_mempool_cache, 0, mbuf_seg_size, - heap_socket); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_mp = rte_pktmbuf_pool_create(pool_name, + nb_mbuf, mb_mempool_cache, 0, + mbuf_seg_size, heap_socket); + else + rte_mp = rte_mempool_lookup(pool_name); + break; } case MP_ALLOC_XBUF: @@ -2503,21 +2522,28 @@ start_port(portid_t pid) return -1; } /* configure port */ - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq, + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + diag = rte_eth_dev_configure(pi, + nb_rxq + nb_hairpinq, nb_txq + nb_hairpinq, &(port->dev_conf)); - if (diag != 0) { - if (rte_atomic16_cmpset(&(port->port_status), - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0) - printf("Port %d can not be set back " - "to stopped\n", pi); - printf("Fail to configure port %d\n", pi); - /* try to reconfigure port next time */ - port->need_reconfig = 1; - return -1; + if (diag != 0) { + if (rte_atomic16_cmpset( + &(port->port_status), + RTE_PORT_HANDLING, + RTE_PORT_STOPPED) == 0) + printf("Port %d can not be set " + "back to stopped\n", pi); + printf("Fail to configure port %d\n", + pi); + /* try to reconfigure port next time */ + port->need_reconfig = 1; + return -1; + } } } - if (port->need_reconfig_queues > 0) { + if (port->need_reconfig_queues > 0 && + rte_eal_process_type() == RTE_PROC_PRIMARY) { port->need_reconfig_queues = 0; /* setup tx queues */ for (qi = 0; qi < nb_txq; qi++) { @@ -2618,15 +2644,18 @@ start_port(portid_t pid) cnt_pi++; /* start port */ - if (rte_eth_dev_start(pi) < 0) { - printf("Fail to start port %d\n", pi); - - /* Fail to setup rx queue, return */ - if (rte_atomic16_cmpset(&(port->port_status), - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0) - printf("Port %d can not be set back to " - "stopped\n", pi); - continue; + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + diag = rte_eth_dev_start(pi); + if (diag < 0) { + printf("Fail to start port %d\n", pi); + + /* Fail to setup rx queue, return */ + if (rte_atomic16_cmpset(&(port->port_status), + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0) + printf("Port %d can not be set back to " + "stopped\n", pi); + continue; + } } if (rte_atomic16_cmpset(&(port->port_status), @@ -2755,7 +2784,7 @@ stop_port(portid_t pid) if (port->flow_list) port_flow_flush(pi); - if (rte_eth_dev_stop(pi) != 0) + if (rte_eal_process_type() == RTE_PROC_PRIMARY && rte_eth_dev_stop(pi) != 0) RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n", pi); @@ -2824,8 +2853,10 @@ close_port(portid_t pid) continue; } - port_flow_flush(pi); - rte_eth_dev_close(pi); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + port_flow_flush(pi); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_eth_dev_close(pi); } remove_invalid_ports(); @@ -3089,7 +3120,7 @@ pmd_test_exit(void) } } for (i = 0 ; i < RTE_DIM(mempools) ; i++) { - if (mempools[i]) + if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i]) rte_mempool_free(mempools[i]); } @@ -3537,6 +3568,10 @@ init_port_dcb_config(portid_t pid, int retval; uint16_t i; + if (num_procs > 1) { + printf("The multi-process feature doesn't support dcb.\n"); + return -ENOTSUP; + } rte_port = &ports[pid]; memset(&port_conf, 0, sizeof(struct rte_eth_conf)); @@ -3635,13 +3670,6 @@ init_port(void) } static void -force_quit(void) -{ - pmd_test_exit(); - prompt_exit(); -} - -static void print_stats(void) { uint8_t i; @@ -3672,12 +3700,16 @@ signal_handler(int signum) if (latencystats_enabled != 0) rte_latencystats_uninit(); #endif - force_quit(); /* Set flag to indicate the force termination. */ f_quit = 1; - /* exit with the expected status */ - signal(signum, SIG_DFL); - kill(getpid(), signum); + if (interactive == 1) { + dup2(testpmd_cl->s_in, testpmd_fd_copy); + close(testpmd_cl->s_in); + } else { + dup2(0, testpmd_fd_copy); + close(0); + } + } } @@ -3702,10 +3734,6 @@ main(int argc, char** argv) rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n", rte_strerror(rte_errno)); - if (rte_eal_process_type() == RTE_PROC_SECONDARY) - rte_exit(EXIT_FAILURE, - "Secondary process type not supported.\n"); - ret = register_eth_event_callback(); if (ret != 0) rte_exit(EXIT_FAILURE, "Cannot register for ethdev events"); @@ -3801,8 +3829,10 @@ main(int argc, char** argv) } } - if (!no_device_start && start_port(RTE_PORT_ALL) != 0) + if (!no_device_start && start_port(RTE_PORT_ALL) != 0) { + pmd_test_exit(); rte_exit(EXIT_FAILURE, "Start ports failed\n"); + } /* set all ports to promiscuous mode by default */ RTE_ETH_FOREACH_DEV(port_id) { @@ -3848,6 +3878,8 @@ main(int argc, char** argv) } prompt(); pmd_test_exit(); + if (unlikely(f_quit == 1)) + prompt_exit(); } else #endif { @@ -3883,6 +3915,11 @@ main(int argc, char** argv) printf("Press enter to exit\n"); rc = read(0, &c, 1); pmd_test_exit(); + if (unlikely(f_quit == 1)) { + dup2(testpmd_fd_copy, 0); + close(testpmd_fd_copy); + prompt_exit(); + } if (rc < 0) return 1; } diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index 5f23162..8629c57 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -13,6 +13,7 @@ #include <rte_gso.h> #include <cmdline.h> #include <sys/queue.h> +#include "cmdline.h" #define RTE_PORT_ALL (~(portid_t)0x0) @@ -24,6 +25,10 @@ #define RTE_PORT_CLOSED (uint16_t)2 #define RTE_PORT_HANDLING (uint16_t)3 +uint8_t f_quit; +int testpmd_fd_copy; +struct cmdline *testpmd_cl; + /* * It is used to allocate the memory for hash key. * The hash key size is NIC dependent. @@ -421,6 +426,8 @@ extern uint64_t noisy_lkup_mem_sz; extern uint64_t noisy_lkup_num_writes; extern uint64_t noisy_lkup_num_reads; extern uint64_t noisy_lkup_num_reads_writes; +extern int proc_id; +extern unsigned num_procs; extern uint8_t dcb_config; extern uint8_t dcb_test; -- 2.7.4

2 2

Re: [PATCH v13 00/15] SMMUv3 Nested Stage Setup (IOMMU part)
by Shameerali Kolothum Thodi 22 Feb '21

22 Feb '21

Hi Eric, > -----Original Message----- > From: Eric Auger [mailto:eric.auger@redhat.com] > Sent: 18 November 2020 11:22 > To: eric.auger.pro(a)gmail.com; eric.auger(a)redhat.com; > iommu(a)lists.linux-foundation.org; linux-kernel(a)vger.kernel.org; > kvm(a)vger.kernel.org; kvmarm(a)lists.cs.columbia.edu; will(a)kernel.org; > joro(a)8bytes.org; maz(a)kernel.org; robin.murphy(a)arm.com; > alex.williamson(a)redhat.com > Cc: jean-philippe(a)linaro.org; zhangfei.gao(a)linaro.org; > zhangfei.gao(a)gmail.com; vivek.gautam(a)arm.com; Shameerali Kolothum > Thodi <shameerali.kolothum.thodi(a)huawei.com>; > jacob.jun.pan(a)linux.intel.com; yi.l.liu(a)intel.com; tn(a)semihalf.com; > nicoleotsuka(a)gmail.com; yuzenghui <yuzenghui(a)huawei.com> > Subject: [PATCH v13 00/15] SMMUv3 Nested Stage Setup (IOMMU part) > > This series brings the IOMMU part of HW nested paging support > in the SMMUv3. The VFIO part is submitted separately. > > The IOMMU API is extended to support 2 new API functionalities: > 1) pass the guest stage 1 configuration > 2) pass stage 1 MSI bindings > > Then those capabilities gets implemented in the SMMUv3 driver. > > The virtualizer passes information through the VFIO user API > which cascades them to the iommu subsystem. This allows the guest > to own stage 1 tables and context descriptors (so-called PASID > table) while the host owns stage 2 tables and main configuration > structures (STE). I am seeing an issue with Guest testpmd run with this series. I have two different setups and testpmd works fine with the first one but not with the second. 1). Guest doesn't have kernel driver built-in for pass-through dev. root@ubuntu:/# lspci -v ... 00:02.0 Ethernet controller: Huawei Technologies Co., Ltd. Device a22e (rev 21) Subsystem: Huawei Technologies Co., Ltd. Device 0000 Flags: fast devsel Memory at 8000100000 (64-bit, prefetchable) [disabled] [size=64K] Memory at 8000000000 (64-bit, prefetchable) [disabled] [size=1M] Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [a0] MSI-X: Enable- Count=67 Masked- Capabilities: [b0] Power Management version 3 Capabilities: [100] Access Control Services Capabilities: [300] Transaction Processing Hints root@ubuntu:/# echo vfio-pci > /sys/bus/pci/devices/0000:00:02.0/driver_override root@ubuntu:/# echo 0000:00:02.0 > /sys/bus/pci/drivers_probe root@ubuntu:/mnt/dpdk/build/app# ./testpmd -w 0000:00:02.0 --file-prefix socket0 -l 0-1 -n 2 -- -i EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/socket0/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-32768kB EAL: No available hugepages reported in hugepages-64kB EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_hns3_vf (19e5:a22e) device: 0000:00:02.0 (socket 0) EAL: No legacy callbacks, legacy socket not created Interactive-mode selected testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 8E:A6:8C:43:43:45 Checking link statuses... Done testpmd> 2). Guest have kernel driver built-in for pass-through dev. root@ubuntu:/# lspci -v ... 00:02.0 Ethernet controller: Huawei Technologies Co., Ltd. Device a22e (rev 21) Subsystem: Huawei Technologies Co., Ltd. Device 0000 Flags: bus master, fast devsel, latency 0 Memory at 8000100000 (64-bit, prefetchable) [size=64K] Memory at 8000000000 (64-bit, prefetchable) [size=1M] Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [a0] MSI-X: Enable+ Count=67 Masked- Capabilities: [b0] Power Management version 3 Capabilities: [100] Access Control Services Capabilities: [300] Transaction Processing Hints Kernel driver in use: hns3 root@ubuntu:/# echo vfio-pci > /sys/bus/pci/devices/0000:00:02.0/driver_override root@ubuntu:/# echo 0000:00:02.0 > /sys/bus/pci/drivers/hns3/unbind root@ubuntu:/# echo 0000:00:02.0 > /sys/bus/pci/drivers_probe root@ubuntu:/mnt/dpdk/build/app# ./testpmd -w 0000:00:02.0 --file-prefix socket0 -l 0-1 -n 2 -- -i EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/socket0/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-32768kB EAL: No available hugepages reported in hugepages-64kB EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_hns3_vf (19e5:a22e) device: 0000:00:02.0 (socket 0) 0000:00:02.0 hns3_get_mbx_resp(): VF could not get mbx(11,0) head(1) tail(0) lost(1) from PF in_irq:0 hns3vf_get_queue_info(): Failed to get tqp info from PF: -62 hns3vf_init_vf(): Failed to fetch configuration: -62 hns3vf_dev_init(): Failed to init vf: -62 EAL: Releasing pci mapped resource for 0000:00:02.0 EAL: Calling pci_unmap_resource for 0000:00:02.0 at 0x1100800000 EAL: Calling pci_unmap_resource for 0000:00:02.0 at 0x1100810000 EAL: Requested device 0000:00:02.0 cannot be used EAL: Bus (pci) probe failed. EAL: No legacy callbacks, legacy socket not created testpmd: No probed ethernet devices Interactive-mode selected testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Done testpmd> And in this case, smmu(host) reports a translation fault, [ 6542.670624] arm-smmu-v3 arm-smmu-v3.2.auto: event 0x10 received: [ 6542.670630] arm-smmu-v3 arm-smmu-v3.2.auto: 0x00007d1200000010 [ 6542.670631] arm-smmu-v3 arm-smmu-v3.2.auto: 0x000012000000007c [ 6542.670633] arm-smmu-v3 arm-smmu-v3.2.auto: 0x00000000fffef040 [ 6542.670634] arm-smmu-v3 arm-smmu-v3.2.auto: 0x00000000fffef000 Tested with Intel 82599 card(ixgbevf) as well. but same errror. Not able to root cause the problem yet. With the hope that, this is related to tlb entries not being invlaidated properly, I tried explicitly issuing CMD_TLBI_NSNH_ALL and CMD_CFGI_CD_ALL just before the STE update, but no luck yet :( Please let me know if I am missing something here or has any clue if you can replicate this on your setup. Thanks, Shameer > > Best Regards > > Eric > > This series can be found at: > https://github.com/eauger/linux/tree/5.10-rc4-2stage-v13 > (including the VFIO part in his last version: v11) > > The series includes a patch from Jean-Philippe. It is better to > review the original patch: > [PATCH v8 2/9] iommu/arm-smmu-v3: Maintain a SID->device structure > > The VFIO series is sent separately. > > History: > > v12 -> v13: > - fixed compilation issue with CONFIG_ARM_SMMU_V3_SVA > reported by Shameer. This urged me to revisit patch 4 into > iommu/smmuv3: Allow s1 and s2 configs to coexist where > s1_cfg and s2_cfg are not dynamically allocated anymore. > Instead I use a new set field in existing structs > - fixed 2 others config checks > - Updated "iommu/arm-smmu-v3: Maintain a SID->device structure" > according to the last version > > v11 -> v12: > - rebase on top of v5.10-rc4 > > Eric Auger (14): > iommu: Introduce attach/detach_pasid_table API > iommu: Introduce bind/unbind_guest_msi > iommu/smmuv3: Allow s1 and s2 configs to coexist > iommu/smmuv3: Get prepared for nested stage support > iommu/smmuv3: Implement attach/detach_pasid_table > iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs > iommu/smmuv3: Implement cache_invalidate > dma-iommu: Implement NESTED_MSI cookie > iommu/smmuv3: Nested mode single MSI doorbell per domain enforcement > iommu/smmuv3: Enforce incompatibility between nested mode and HW MSI > regions > iommu/smmuv3: Implement bind/unbind_guest_msi > iommu/smmuv3: Report non recoverable faults > iommu/smmuv3: Accept configs with more than one context descriptor > iommu/smmuv3: Add PASID cache invalidation per PASID > > Jean-Philippe Brucker (1): > iommu/arm-smmu-v3: Maintain a SID->device structure > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 659 > ++++++++++++++++++-- > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 103 ++- > drivers/iommu/dma-iommu.c | 142 ++++- > drivers/iommu/iommu.c | 105 ++++ > include/linux/dma-iommu.h | 16 + > include/linux/iommu.h | 41 ++ > include/uapi/linux/iommu.h | 54 ++ > 7 files changed, 1042 insertions(+), 78 deletions(-) > > -- > 2.21.3

2 3

[PATCH v4 00/12] add IRQF_NO_AUTOEN for request_irq
by Barry Song 10 Feb '21

10 Feb '21

This patchset added IRQF_NO_AUTOEN for request_irq() and converted drivers/input to this new API. Other drivers will be handled afterwards. -v4: remove the irq_settings magic for NOAUTOEN Barry Song (12): genirq: add IRQF_NO_AUTOEN for request_irq Input: ar1021 - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: atmel_mxt_ts - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: melfas_mip4 - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: bu21029_ts - request_irq by IRQF_NO_AUTOEN and remove irq_set_status_flags Input: stmfts - request_irq by IRQF_NO_AUTOEN and remove irq_set_status_flags Input: zinitix - request_irq by IRQF_NO_AUTOEN and remove irq_set_status_flags Input: mms114 - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: wm831x-ts - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: cyttsp - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: tegra-kbc - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: tca6416-keypad - request_irq by IRQF_NO_AUTOEN and remove disable_irq drivers/input/keyboard/tca6416-keypad.c | 3 +-- drivers/input/keyboard/tegra-kbc.c | 5 ++--- drivers/input/touchscreen/ar1021_i2c.c | 5 +---- drivers/input/touchscreen/atmel_mxt_ts.c | 5 ++--- drivers/input/touchscreen/bu21029_ts.c | 4 ++-- drivers/input/touchscreen/cyttsp_core.c | 5 ++--- drivers/input/touchscreen/melfas_mip4.c | 5 ++--- drivers/input/touchscreen/mms114.c | 4 ++-- drivers/input/touchscreen/stmfts.c | 3 +-- drivers/input/touchscreen/wm831x-ts.c | 3 +-- drivers/input/touchscreen/zinitix.c | 4 ++-- include/linux/interrupt.h | 3 +++ kernel/irq/manage.c | 8 +++++++- 13 files changed, 28 insertions(+), 29 deletions(-) -- 2.25.1

2 13

[PATCH v2 for-next] RDMA/hns: Add support of direct wqe
by Weihang Li 09 Feb '21

09 Feb '21

From: Yixing Liu <liuyixing1(a)huawei.com> Direct wqe is a mechanism to fill wqe directly into the hardware. In the case of light load, the wqe will be filled into pcie bar space of the hardware, this will reduce one memory access operation and therefore reduce the latency. Signed-off-by: Yixing Liu <liuyixing1(a)huawei.com> Signed-off-by: Lang Cheng <chenglang(a)huawei.com> Signed-off-by: Weihang Li <liweihang(a)huawei.com> --- Changes since v1: * Delete an extra blank line. * Link: https://patchwork.kernel.org/project/linux-rdma/patch/1611395717-11081-1-gi… drivers/infiniband/hw/hns/hns_roce_device.h | 6 ++++ drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 44 ++++++++++++++++++++++++++++- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 13 +++++++++ 3 files changed, 62 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h index f62851f..907bf71 100644 --- a/drivers/infiniband/hw/hns/hns_roce_device.h +++ b/drivers/infiniband/hw/hns/hns_roce_device.h @@ -90,6 +90,7 @@ #define HNS_ROCE_MAX_PORTS 6 #define HNS_ROCE_GID_SIZE 16 #define HNS_ROCE_SGE_SIZE 16 +#define HNS_ROCE_DWQE_SIZE 65536 #define HNS_ROCE_HOP_NUM_0 0xff @@ -643,6 +644,10 @@ struct hns_roce_work { u32 queue_num; }; +enum { + HNS_ROCE_QP_CAP_DIRECT_WQE = BIT(5), +}; + struct hns_roce_qp { struct ib_qp ibqp; struct hns_roce_wq rq; @@ -984,6 +989,7 @@ struct hns_roce_dev { struct mutex pgdir_mutex; int irq[HNS_ROCE_MAX_IRQ_NUM]; u8 __iomem *reg_base; + void __iomem *mem_base; struct hns_roce_caps caps; struct xarray qp_table_xa; diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c index a5bbfb1..8ae3317 100644 --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c @@ -503,6 +503,8 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp, if (ret) return ret; + qp->sl = to_hr_ah(ud_wr(wr)->ah)->av.sl; + set_extend_sge(qp, wr->sg_list, &curr_idx, valid_num_sge); /* @@ -635,6 +637,8 @@ static inline void update_sq_db(struct hns_roce_dev *hr_dev, V2_DB_BYTE_4_TAG_S, qp->doorbell_qpn); roce_set_field(sq_db.byte_4, V2_DB_BYTE_4_CMD_M, V2_DB_BYTE_4_CMD_S, HNS_ROCE_V2_SQ_DB); + /* indicates data on new BAR, 0 : SQ doorbell, 1 : DWQE */ + roce_set_bit(sq_db.byte_4, V2_DB_FLAG_S, 0); roce_set_field(sq_db.parameter, V2_DB_PARAMETER_IDX_M, V2_DB_PARAMETER_IDX_S, qp->sq.head); roce_set_field(sq_db.parameter, V2_DB_PARAMETER_SL_M, @@ -644,6 +648,38 @@ static inline void update_sq_db(struct hns_roce_dev *hr_dev, } } +static void hns_roce_write512(struct hns_roce_dev *hr_dev, u64 *val, + u64 __iomem *dest) +{ +#define HNS_ROCE_WRITE_TIMES 8 + struct hns_roce_v2_priv *priv = (struct hns_roce_v2_priv *)hr_dev->priv; + struct hnae3_handle *handle = priv->handle; + const struct hnae3_ae_ops *ops = handle->ae_algo->ops; + int i; + + if (!hr_dev->dis_db && !ops->get_hw_reset_stat(handle)) + for (i = 0; i < HNS_ROCE_WRITE_TIMES; i++) + writeq_relaxed(*(val + i), dest + i); +} + +static void write_dwqe(struct hns_roce_dev *hr_dev, struct hns_roce_qp *qp, + void *wqe) +{ + struct hns_roce_v2_rc_send_wqe *rc_sq_wqe = wqe; + + /* All kinds of DirectWQE have the same header field layout */ + roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_FLAG_S, 1); + roce_set_field(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_DB_SL_L_M, + V2_RC_SEND_WQE_BYTE_4_DB_SL_L_S, qp->sl); + roce_set_field(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_DB_SL_H_M, + V2_RC_SEND_WQE_BYTE_4_DB_SL_H_S, qp->sl >> 2); + roce_set_field(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_M, + V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_S, qp->sq.head); + + hns_roce_write512(hr_dev, wqe, hr_dev->mem_base + + HNS_ROCE_DWQE_SIZE * qp->ibqp.qp_num); +} + static int hns_roce_v2_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, const struct ib_send_wr **bad_wr) @@ -710,7 +746,12 @@ static int hns_roce_v2_post_send(struct ib_qp *ibqp, qp->next_sge = sge_idx; /* Memory barrier */ wmb(); - update_sq_db(hr_dev, qp); + + if (nreq == 1 && qp->sq.head == qp->sq.tail + 1 && + (qp->en_flags & HNS_ROCE_QP_CAP_DIRECT_WQE)) + write_dwqe(hr_dev, qp, wqe); + else + update_sq_db(hr_dev, qp); } spin_unlock_irqrestore(&qp->sq.lock, flags); @@ -6273,6 +6314,7 @@ static void hns_roce_hw_v2_get_cfg(struct hns_roce_dev *hr_dev, /* Get info from NIC driver. */ hr_dev->reg_base = handle->rinfo.roce_io_base; + hr_dev->mem_base = handle->rinfo.roce_mem_base; hr_dev->caps.num_ports = 1; hr_dev->iboe.netdevs[0] = handle->rinfo.netdev; hr_dev->iboe.phy_port[0] = 0; diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.h b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h index 69bc072..add1816 100644 --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.h +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h @@ -1098,6 +1098,8 @@ struct hns_roce_v2_mpt_entry { #define V2_DB_BYTE_4_CMD_S 24 #define V2_DB_BYTE_4_CMD_M GENMASK(27, 24) +#define V2_DB_FLAG_S 31 + #define V2_DB_PARAMETER_IDX_S 0 #define V2_DB_PARAMETER_IDX_M GENMASK(15, 0) @@ -1194,6 +1196,15 @@ struct hns_roce_v2_rc_send_wqe { #define V2_RC_SEND_WQE_BYTE_4_OPCODE_S 0 #define V2_RC_SEND_WQE_BYTE_4_OPCODE_M GENMASK(4, 0) +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_L_S 5 +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_L_M GENMASK(6, 5) + +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_H_S 13 +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_H_M GENMASK(14, 13) + +#define V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_S 15 +#define V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_M GENMASK(30, 15) + #define V2_RC_SEND_WQE_BYTE_4_OWNER_S 7 #define V2_RC_SEND_WQE_BYTE_4_CQE_S 8 @@ -1216,6 +1227,8 @@ struct hns_roce_v2_rc_send_wqe { #define V2_RC_FRMR_WQE_BYTE_4_LW_S 23 +#define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31 + #define V2_RC_SEND_WQE_BYTE_16_XRC_SRQN_S 0 #define V2_RC_SEND_WQE_BYTE_16_XRC_SRQN_M GENMASK(23, 0) -- 2.8.1

2 1

[PATCH for-next 00/12] RDMA/hns: Several fixes and cleanups of RQ/SRQ
by Weihang Li 09 Feb '21

09 Feb '21

There are some issues when using SRQ on HIP08/HIP09, the first part of this series is some fixes on them. In addition, the codes about RQ/SRQ including the creation and post recv flow are a bit hard to understand, they need to be refactored. Lang Cheng (2): RDMA/hns: Allocate one more recv SGE for HIP08 RDMA/hns: Use new interfaces to write SRQC Wenpeng Liang (8): RDMA/hns: Bugfix for checking whether the srq is full when post wr RDMA/hns: Force srq_limit to 0 when creating SRQ RDMA/hns: Fixed wrong judgments in the goto branch RDMA/hns: Remove the reserved WQE of SRQ RDMA/hns: Refactor hns_roce_create_srq() RDMA/hns: Refactor code about SRQ Context RDMA/hns: Refactor hns_roce_v2_post_srq_recv() RDMA/hns: Add verification of QP type when post_recv Xi Wang (2): RDMA/hns: Refactor post recv flow RDMA/hns: Clear remaining unused sges when post_recv drivers/infiniband/hw/hns/hns_roce_device.h | 16 +- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 407 +++++++++++++++------------- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 72 +++-- drivers/infiniband/hw/hns/hns_roce_main.c | 3 +- drivers/infiniband/hw/hns/hns_roce_qp.c | 37 ++- drivers/infiniband/hw/hns/hns_roce_srq.c | 329 ++++++++++++---------- 6 files changed, 510 insertions(+), 354 deletions(-) -- 2.8.1

2 13

[PATCH v2 RFC 0/7] RDMA/hns: Add support for Dynamic Context Attachment
by Weihang Li 07 Feb '21

07 Feb '21

The hip09 introduces the DCA(Dynamic Context Attachment) feature which supports many RC QPs to share the WQE buffer in a memory pool. If a QP enables DCA feature, the WQE's buffer will not be allocated when creating but when the users start to post WRs. This will reduce the memory consumption when there are too many QPs are inactive. Changes since v1: * Replace all GFP_ATOMIC with GFP_NOWAIT, because the former may use emergency pool if no regular memory can be found. * Change size of cap_flags of alloc_ucontext_resp from 32 to 64 to avoid a potential problem when pass it back to the userspace. * Move definition of HNS_ROCE_CAP_FLAG_DCA_MODE to hns-abi.h. * Rename free_mem_states() to free_dca_states() in #1. * Link: https://patchwork.kernel.org/project/linux-rdma/cover/1610706138-4219-1-git… Xi Wang (7): RDMA/hns: Introduce DCA for RC QP RDMA/hns: Add method for shrinking DCA memory pool RDMA/hns: Configure DCA mode for the userspace QP RDMA/hns: Add method for attaching WQE buffer RDMA/hns: Setup the configuration of WQE addressing to QPC RDMA/hns: Add method to detach WQE buffer RDMA/hns: Add method to query WQE buffer's address drivers/infiniband/hw/hns/Makefile | 2 +- drivers/infiniband/hw/hns/hns_roce_dca.c | 1264 +++++++++++++++++++++++++++ drivers/infiniband/hw/hns/hns_roce_dca.h | 68 ++ drivers/infiniband/hw/hns/hns_roce_device.h | 31 + drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 223 ++++- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 3 + drivers/infiniband/hw/hns/hns_roce_main.c | 27 +- drivers/infiniband/hw/hns/hns_roce_qp.c | 119 ++- include/uapi/rdma/hns-abi.h | 64 ++ 9 files changed, 1754 insertions(+), 47 deletions(-) create mode 100644 drivers/infiniband/hw/hns/hns_roce_dca.c create mode 100644 drivers/infiniband/hw/hns/hns_roce_dca.h -- 2.8.1

3 9

[PATCH] perf metricgroup: Fix system PMU metrics
by John Garry 05 Feb '21

05 Feb '21

Joakim reports that getting "perf stat" for multiple system PMU metrics segfaults: ./perf stat -a -I 1000 -M imx8mm_ddr_write.all,imx8mm_ddr_write.all Segmentation fault While the same works without issue for a single metric. The logic in metricgroup__add_metric_sys_event_iter() is broken, in that add_metric() @m argument should be NULL for each new metric. Fix by not passing a holder for that, and rather make local in metricgroup__add_metric_sys_event_iter(). Fixes: be335ec28efa ("perf metricgroup: Support adding metrics for system PMUs") Reported-by: Joakim Zhang <qiangqing.zhang(a)nxp.com> Signed-off-by: John Garry <john.garry(a)huawei.com> diff --git a/tools/perf/util/metricgroup.c b/tools/perf/util/metricgroup.c index ee94d3e8dd65..2e60ee170abc 100644 --- a/tools/perf/util/metricgroup.c +++ b/tools/perf/util/metricgroup.c @@ -766,7 +766,6 @@ int __weak arch_get_runtimeparam(struct pmu_event *pe __maybe_unused) struct metricgroup_add_iter_data { struct list_head *metric_list; const char *metric; - struct metric **m; struct expr_ids *ids; int *ret; bool *has_match; @@ -1058,12 +1057,13 @@ static int metricgroup__add_metric_sys_event_iter(struct pmu_event *pe, void *data) { struct metricgroup_add_iter_data *d = data; + struct metric *m = NULL; int ret; if (!match_pe_metric(pe, d->metric)) return 0; - ret = add_metric(d->metric_list, pe, d->metric_no_group, d->m, NULL, d->ids); + ret = add_metric(d->metric_list, pe, d->metric_no_group, &m, NULL, d->ids); if (ret) return ret; @@ -1114,7 +1114,6 @@ static int metricgroup__add_metric(const char *metric, bool metric_no_group, .metric_list = &list, .metric = metric, .metric_no_group = metric_no_group, - .m = &m, .ids = &ids, .has_match = &has_match, .ret = &ret, -- 2.26.2

4 15

[PATCH 0/3] update doc for hns3
by Lijun Ou 03 Feb '21

03 Feb '21

Here series add some updates for hns3 rst and TM feature note for hns3 as well as FEC feature notes. Lijun Ou (2): doc: update release notes for hns3 doc: fix hns3 rst Min Hu (Connor) (1): doc: add FEC in NIC features doc/guides/nics/features.rst | 14 ++++++++++++++ doc/guides/nics/features/default.ini | 1 + doc/guides/nics/hns3.rst | 2 +- doc/guides/rel_notes/release_21_02.rst | 4 ++++ 4 files changed, 20 insertions(+), 1 deletion(-) -- 2.7.4

4 28

[PATCH] perf test: Add parse-metric memory bandwidth testcase
by John Garry 03 Feb '21

03 Feb '21

Event duration_time in a metric expression requires special handling. Improve test coverage by including a metric whose expression includes duration_time. The actual metric is a copied from the L1D_Cache_Fill_BW metric on my broadwell machine. Signed-off-by: John Garry <john.garry(a)huawei.com> --- Based on acme perf/core + "perf metricgroup: Fix for metrics containing duration_time" diff --git a/tools/perf/tests/parse-metric.c b/tools/perf/tests/parse-metric.c index ce7be37f0d88..6dc1db1626ad 100644 --- a/tools/perf/tests/parse-metric.c +++ b/tools/perf/tests/parse-metric.c @@ -69,6 +69,10 @@ static struct pmu_event pme_test[] = { .metric_expr = "1/m3", .metric_name = "M3", }, +{ + .metric_expr = "64 * l1d.replacement / 1000000000 / duration_time", + .metric_name = "L1D_Cache_Fill_BW", +}, { .name = NULL, } @@ -107,6 +111,8 @@ static void load_runtime_stat(struct runtime_stat *st, struct evlist *evlist, evlist__for_each_entry(evlist, evsel) { count = find_value(evsel->name, vals); perf_stat__update_shadow_stats(evsel, count, 0, st); + if (!strcmp(evsel->name, "duration_time")) + update_stats(&walltime_nsecs_stats, count); } } @@ -321,6 +327,23 @@ static int test_recursion_fail(void) return 0; } +static int test_memory_bandwidth(void) +{ + double ratio; + struct value vals[] = { + { .event = "l1d.replacement", .val = 4000000 }, + { .event = "duration_time", .val = 200000000 }, + { .event = NULL, }, + }; + + TEST_ASSERT_VAL("failed to compute metric", + compute_metric("L1D_Cache_Fill_BW", vals, &ratio) == 0); + TEST_ASSERT_VAL("L1D_Cache_Fill_BW, wrong ratio", + 1.28 == ratio); + + return 0; +} + static int test_metric_group(void) { double ratio1, ratio2; @@ -353,5 +376,6 @@ int test__parse_metric(struct test *test __maybe_unused, int subtest __maybe_unu TEST_ASSERT_VAL("DCache_L2 failed", test_dcache_l2() == 0); TEST_ASSERT_VAL("recursion fail failed", test_recursion_fail() == 0); TEST_ASSERT_VAL("test metric group", test_metric_group() == 0); + TEST_ASSERT_VAL("Memory bandwidth", test_memory_bandwidth() == 0); return 0; } -- 2.26.2

3 2

[PATCH 00/14] Misc updates for hns3
by Lijun Ou 02 Feb '21

02 Feb '21

This series add a new feature LSC event report and two bugfixes as well as some cleanup patches. Chengchang Tang (2): net/hns3: reconstruct the Rx interrupt map net/hns3: fix interrupt resources in Rx interrupt mode Chengwen Feng (1): net/hns3: support LSC event report Huisong Li (2): net/hns3: encapsulate dfx stats in Rx/Tx datapatch net/hns3: move queue stats to xstats Lijun Ou (9): net/hns3: use array instead of switch-case net/hns3: move judgment conditions to separated functions net/hns3: extract common judgments for all FDIR type net/hns3: refactor reset event report function net/hns3: fix memory leak with secondary process exit net/hns3: rename RSS functions net/hns3: adjust some comments net/hns3: remove unnecessary parentheses net/hns3: use %d instead of %u for enum variable drivers/net/hns3/hns3_cmd.c | 54 ++-- drivers/net/hns3/hns3_cmd.h | 9 +- drivers/net/hns3/hns3_ethdev.c | 208 ++++++++++------ drivers/net/hns3/hns3_ethdev.h | 4 +- drivers/net/hns3/hns3_ethdev_vf.c | 111 ++++++--- drivers/net/hns3/hns3_flow.c | 180 ++++++-------- drivers/net/hns3/hns3_mbx.c | 14 +- drivers/net/hns3/hns3_regs.c | 2 +- drivers/net/hns3/hns3_regs.h | 24 +- drivers/net/hns3/hns3_rss.c | 12 +- drivers/net/hns3/hns3_rss.h | 4 +- drivers/net/hns3/hns3_rxtx.c | 60 +++-- drivers/net/hns3/hns3_rxtx.h | 148 ++++++----- drivers/net/hns3/hns3_rxtx_vec_neon.h | 2 +- drivers/net/hns3/hns3_stats.c | 450 ++++++++++++++++++++++++++-------- drivers/net/hns3/hns3_stats.h | 15 +- 16 files changed, 840 insertions(+), 457 deletions(-) -- 2.7.4

4 20

[PATCH V2 net-next 0/2] net: hns3: updates for -next
by Huazhong Tan 02 Feb '21

02 Feb '21

This patchset adds dump tm info of nodes, priority and qset in debugfs. Three debugfs files tm_nodes, tm_priority and tm_qset are created in new tm directory, and use cat command to dump their info, for examples: $ cat tm_nodes BASE_ID MAX_NUM PG 0 8 PRI 0 8 QSET 0 8 QUEUE 0 1024 $ cat tm_priority ID MODE DWRR C_IR_B C_IR_U C_IR_S C_BS_B C_BS_S C_FLAG C_RATE(Mbps) P_IR_B P_IR_U P_IR_S P_BS_B P_BS_S P_FLAG P_RATE(Mbps) 0000 dwrr 100 0 0 0 5 20 0 0 150 7 0 5 20 0 0 0001 sp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0002 sp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0003 sp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0004 sp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0005 sp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0006 sp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0007 sp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 $ cat tm_qset ID MAP_PRI LINK_VLD MODE DWRR 0000 0 1 dwrr 100 0001 0 0 sp 0 0002 0 0 sp 0 0003 0 0 sp 0 0004 0 0 sp 0 0005 0 0 sp 0 0006 0 0 sp 0 change log: V2: add readonly files for dump all nodes, priority and qset info suggested by Jakub Kicinski. previous version: V1: https://patchwork.kernel.org/project/netdevbpf/patch/1610694569-43099-1-git… Guangbin Huang (2): net: hns3: add interfaces to query information of tm priority/qset net: hns3: add debugfs support for tm nodes, priority and qset info drivers/net/ethernet/hisilicon/hns3/hnae3.h | 8 + drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c | 55 +++++- .../net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.h | 1 + .../ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c | 153 +++++++++++++++++ .../ethernet/hisilicon/hns3/hns3pf/hclge_main.c | 1 + .../ethernet/hisilicon/hns3/hns3pf/hclge_main.h | 2 + .../net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c | 186 +++++++++++++++++++++ .../net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.h | 48 +++++- 8 files changed, 452 insertions(+), 2 deletions(-) -- 2.7.4

3 5

[PATCH 0/2] Add check of 4-byte when parsing SFDP 4bait table
by Yicong Yang 01 Feb '21

01 Feb '21

Add check of 4-byte address mode support when parsing SFDP 4bait. Some flash will provide a 4bait table and the spi-nor core will convert the address mode to 4-byte without checking whether it's actually supported or not by the controller. For example, the 16M s25fs128s1 provides the 4bait and will be convert to 4-byte address mode, which makes it unusable on hisi-sfc-v3xx on hip08 platform. Add the check before convert the address mode when parsing the 4bait will solve this issue. Yicong Yang (2): mtd: spi-nor: check 4-byte address support when parsing 4bait spi: hisi-sfc-v3xx: add address mode check drivers/mtd/spi-nor/sfdp.c | 48 +++++++++++++++++++++++++++++++++++++++++ drivers/spi/spi-hisi-sfc-v3xx.c | 25 ++++++++++++++++++++- 2 files changed, 72 insertions(+), 1 deletion(-) -- 2.8.1

3 5

[PATCH v3 00/12] add IRQF_NO_AUTOEN for request_irq
by Barry Song 01 Feb '21

01 Feb '21

This patchset added IRQF_NO_AUTOEN for request_irq() and converted drivers/input to this new API. Other drivers will be handled afterwards. -v3: added examples with respect to Greg's comment Barry Song (12): genirq: add IRQF_NO_AUTOEN for request_irq Input: ar1021 - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: atmel_mxt_ts - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: melfas_mip4 - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: bu21029_ts - request_irq by IRQF_NO_AUTOEN and remove irq_set_status_flags Input: stmfts - request_irq by IRQF_NO_AUTOEN and remove irq_set_status_flags Input: zinitix - request_irq by IRQF_NO_AUTOEN and remove irq_set_status_flags Input: mms114 - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: wm831x-ts - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: cyttsp - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: tegra-kbc - request_irq by IRQF_NO_AUTOEN and remove disable_irq Input: tca6416-keypad - request_irq by IRQF_NO_AUTOEN and remove disable_irq drivers/input/keyboard/tca6416-keypad.c | 3 +-- drivers/input/keyboard/tegra-kbc.c | 5 ++--- drivers/input/touchscreen/ar1021_i2c.c | 5 +---- drivers/input/touchscreen/atmel_mxt_ts.c | 5 ++--- drivers/input/touchscreen/bu21029_ts.c | 4 ++-- drivers/input/touchscreen/cyttsp_core.c | 5 ++--- drivers/input/touchscreen/melfas_mip4.c | 5 ++--- drivers/input/touchscreen/mms114.c | 4 ++-- drivers/input/touchscreen/stmfts.c | 3 +-- drivers/input/touchscreen/wm831x-ts.c | 3 +-- drivers/input/touchscreen/zinitix.c | 4 ++-- include/linux/interrupt.h | 3 +++ kernel/irq/manage.c | 8 ++++++++ kernel/irq/settings.h | 10 ++++++++++ 14 files changed, 39 insertions(+), 28 deletions(-) -- 2.25.1

6 19

[GIT PULL v2] arm64: dts: hisilicon dts updates for v5.12
by Wei Xu 30 Jan '21

30 Jan '21

Hi ARM-SoC team, Please consider to pull the following changes. Thanks! Best Regards, Wei --- The following changes since commit 5c8fe583cce542aa0b84adc939ce85293de36e5e: Linux 5.11-rc1 (2020-12-27 15:30:22 -0800) are available in the git repository at: git://github.com/hisilicon/linux-hisi.git tags/hisi-arm64-dt-for-5.12v2 for you to fetch changes up to b6e141eec86b730dfeddb888af7d3e0247530ef1: arm64: dts: hisilicon: hi3670.dtsi: add I2C settings (2021-01-29 16:40:07 +0800) ---------------------------------------------------------------- ARM64: DT: Hisilicon ARM64 DT updates for 5.12 - Further cleanups of the hisilicon DTS to align with the dtschema - Add or update the I2C, pinctrl and reset nodes for Hikey970 ---------------------------------------------------------------- Mauro Carvalho Chehab (3): arm64: dts: hisilicon: hi3670.dtsi: add iomcu_rst arm64: dts: hisilicon: hikey970-pinctrl.dtsi: add missing pinctrl settings arm64: dts: hisilicon: hi3670.dtsi: add I2C settings Zhen Lei (6): arm64: dts: hisilicon: separate each group of data in the property "ranges" arm64: dts: hisilicon: place clock-names "bus" before "core" arm64: dts: hisilicon: normalize the node name of the module thermal arm64: dts: hisilicon: normalize the node name of the localbus arm64: dts: hisilicon: avoid irrelevant nodes being mistakenly identified as PHY nodes arm64: dts: hisilicon: delete unused property smmu-cb-memtype arch/arm64/boot/dts/hisilicon/hi3660.dtsi | 6 +- arch/arm64/boot/dts/hisilicon/hi3670.dtsi | 77 +++ arch/arm64/boot/dts/hisilicon/hi3798cv200.dtsi | 8 +- arch/arm64/boot/dts/hisilicon/hi6220.dtsi | 8 +- .../arm64/boot/dts/hisilicon/hikey970-pinctrl.dtsi | 632 ++++++++++++++++++++- arch/arm64/boot/dts/hisilicon/hip05.dtsi | 2 +- arch/arm64/boot/dts/hisilicon/hip06.dtsi | 6 +- arch/arm64/boot/dts/hisilicon/hip07.dtsi | 9 +- 8 files changed, 714 insertions(+), 34 deletions(-)

2 1

question about bonding mode 4
by moyufeng 30 Jan '21

30 Jan '21

Hi Team, I have a question about bonding. During testing bonding mode 4 scenarios, I find that there is a very low probability that the pointer is null. The following information is displayed: [99359.795934] bond0: (slave eth13.2001): Port 2 did not find a suitable aggregator [99359.796960] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000020 [99359.798127] Mem abort info: [99359.798526] ESR = 0x96000004 [99359.798938] EC = 0x25: DABT (current EL), IL = 32 bits [99359.799673] SET = 0, FnV = 0 [99359.800106] EA = 0, S1PTW = 0 [99359.800554] Data abort info: [99359.800952] ISV = 0, ISS = 0x00000004 [99359.801522] CM = 0, WnR = 0 [99359.801970] user pgtable: 4k pages, 48-bit VAs, pgdp=00000000c64e6000 [99359.802876] [0000000000000020] pgd=0000000000000000 [99359.803555] Internal error: Oops: 96000004 [#1] PREEMPT SMP [99359.804369] Modules linked in: bonding hns3(-) hclgevf hnae3 [last unloaded: bonding] [99359.805494] CPU: 1 PID: 951 Comm: kworker/u10:2 Not tainted 5.7.0-rc4+ #1 [99359.806455] Hardware name: linux,dummy-virt (DT) [99359.807107] Workqueue: bond0 bond_3ad_state_machine_handler [bonding] [99359.808056] pstate: 60c00005 (nZCv daif +PAN +UAO) [99359.808722] pc : bond_3ad_state_machine_handler+0x7fc/0xdb8 [bonding] [99359.809652] lr : bond_3ad_state_machine_handler+0x7f4/0xdb8 [bonding] [99359.810535] sp : ffff80001882bd20 [99359.811012] x29: ffff80001882bd20 x28: ffff000085939a38 [99359.811791] x27: ffff00008649bb68 x26: 00000000aaaaaaab [99359.812871] x25: ffff800009401000 x24: ffff800009408de4 [99359.814049] x23: ffff80001882bd98 x22: ffff00008649b880 [99359.815210] x21: 0000000000000000 x20: ffff000085939a00 [99359.816401] x19: ffff00008649b880 x18: ffff800012572988 [99359.817637] x17: 0000000000000000 x16: 0000000000000000 [99359.818870] x15: ffff80009882b987 x14: 726f746167657267 [99359.820090] x13: 676120656c626174 x12: 697573206120646e [99359.821374] x11: 696620746f6e2064 x10: 696420322074726f [99359.822659] x9 : 50203a2931303032 x8 : 0000000000081391 [99359.823891] x7 : ffff8000108e3ad0 x6 : ffff8000128858bb [99359.825109] x5 : 0000000000000000 x4 : 0000000000000000 [99359.826262] x3 : 00000000ffffffff x2 : 906b329bb5362a00 [99359.827394] x1 : 906b329bb5362a00 x0 : 0000000000000000 [99359.828540] Call trace: [99359.829071] bond_3ad_state_machine_handler+0x7fc/0xdb8 [bonding] [99359.830367] process_one_work+0x15c/0x4a0 [99359.831216] worker_thread+0x50/0x478 [99359.832022] kthread+0x130/0x160 [99359.832716] ret_from_fork+0x10/0x18 [99359.833487] Code: 910c0021 95f704bb f9403f80 b5ffe300 (f9401000) [99359.834742] ---[ end trace c7a8e02914afc4e0 ]--- [99359.835817] Kernel panic - not syncing: Fatal exception in interrupt [99359.837334] SMP: stopping secondary CPUs [99359.838277] Kernel Offset: disabled [99359.839086] CPU features: 0x080002,22208218 [99359.840053] Memory Limit: none [99359.840783] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- The test procedure is as follows: 1. Configure bonding and set it to mode 4. echo "4" > /sys/class/net/bond0/bonding/mode ifconfig bond0 up 2. Configure two VLANs and add them to the bonding in step 1. vconfig add eth0 2001 vconfig add eth1 2001 ifenslave bond0 eth0.2001 eth1.2001 3. Unload the network device driver and bonding driver. rmmod hns3 rmmod hclge rmmod hnae3 rmmod bonding.ko 4. Repeat the preceding steps for a long time. By checking the logic in ad_port_selection_logic(), I find that if enter the branch "Port %d did not find a suitable aggregator", the value of port->aggregator will be NULL, causing the problem. So I'd like to ask what circumstances will be involved in this branch, and what should be done in this case? The detailed code analysis is as follows: static void ad_port_selection_logic(struct port *port, bool *update_slave_arr) { [...] /* if the port is connected to other aggregator, detach it */ if (port->aggregator) { /* detach the port from its former aggregator */ temp_aggregator = port->aggregator; for (curr_port = temp_aggregator->lag_ports; curr_port; last_port = curr_port, curr_port = curr_port->next_port_in_aggregator) { if (curr_port == port) { temp_aggregator->num_of_ports--; /* if it is the first port attached to the * aggregator */ if (!last_port) { temp_aggregator->lag_ports = port->next_port_in_aggregator; } else { /* not the first port attached to the * aggregator */ last_port->next_port_in_aggregator = port->next_port_in_aggregator; } /* clear the port's relations to this * aggregator */ port->aggregator = NULL; ----analysis: set port->aggregator NULL at the beginning of ad_port_selection_logic(). port->next_port_in_aggregator = NULL; port->actor_port_aggregator_identifier = 0; slave_dbg(bond->dev, port->slave->dev, "Port %d left LAG %d\n", port->actor_port_number, temp_aggregator->aggregator_identifier); /* if the aggregator is empty, clear its * parameters, and set it ready to be attached */ if (!temp_aggregator->lag_ports) ad_clear_agg(temp_aggregator); break; } } if (!curr_port) { /* meaning: the port was related to an aggregator * but was not on the aggregator port list */ net_warn_ratelimited("%s: (slave %s): Warning: Port %d was related to aggregator %d but was not on its port list\n", port->slave->bond->dev->name, port->slave->dev->name, port->actor_port_number, port->aggregator->aggregator_identifier); } } /* search on all aggregators for a suitable aggregator for this port */ bond_for_each_slave(bond, slave, iter) { aggregator = &(SLAVE_AD_INFO(slave)->aggregator); /* keep a free aggregator for later use(if needed) */ if (!aggregator->lag_ports) { if (!free_aggregator) free_aggregator = aggregator; ----analysis: Save free_aggregator if found a free aggregator. But in this case, no free aggregator is available. continue; } /* check if current aggregator suits us */ if (((aggregator->actor_oper_aggregator_key == port->actor_oper_port_key) && /* if all parameters match AND */ MAC_ADDRESS_EQUAL(&(aggregator->partner_system), &(port->partner_oper.system)) && (aggregator->partner_system_priority == port->partner_oper.system_priority) && (aggregator->partner_oper_aggregator_key == port->partner_oper.key) ) && ((!MAC_ADDRESS_EQUAL(&(port->partner_oper.system), &(null_mac_addr)) && /* partner answers */ !aggregator->is_individual) /* but is not individual OR */ ) ) { /* attach to the founded aggregator */ port->aggregator = aggregator; ----analysis: If a proper aggregator is found, port->aggregator is assigned here. port->actor_port_aggregator_identifier = port->aggregator->aggregator_identifier; port->next_port_in_aggregator = aggregator->lag_ports; port->aggregator->num_of_ports++; aggregator->lag_ports = port; slave_dbg(bond->dev, slave->dev, "Port %d joined LAG %d (existing LAG)\n", port->actor_port_number, port->aggregator->aggregator_identifier); /* mark this port as selected */ port->sm_vars |= AD_PORT_SELECTED; found = 1; ----analysis: Set found to 1 if port->aggregator is assigned. break; } } /* the port couldn't find an aggregator - attach it to a new * aggregator */ if (!found) { if (free_aggregator) { /* assign port a new aggregator */ port->aggregator = free_aggregator; ----analysis: No proper free_aggregator is found. Therefore, port->aggregator cannot be assigned here. port->actor_port_aggregator_identifier = port->aggregator->aggregator_identifier; /* update the new aggregator's parameters * if port was responsed from the end-user */ if (port->actor_oper_port_key & AD_DUPLEX_KEY_MASKS) /* if port is full duplex */ port->aggregator->is_individual = false; else port->aggregator->is_individual = true; port->aggregator->actor_admin_aggregator_key = port->actor_admin_port_key; port->aggregator->actor_oper_aggregator_key = port->actor_oper_port_key; port->aggregator->partner_system = port->partner_oper.system; port->aggregator->partner_system_priority = port->partner_oper.system_priority; port->aggregator->partner_oper_aggregator_key = port->partner_oper.key; port->aggregator->receive_state = 1; port->aggregator->transmit_state = 1; port->aggregator->lag_ports = port; port->aggregator->num_of_ports++; /* mark this port as selected */ port->sm_vars |= AD_PORT_SELECTED; slave_dbg(bond->dev, port->slave->dev, "Port %d joined LAG %d (new LAG)\n", port->actor_port_number, port->aggregator->aggregator_identifier); } else { slave_err(bond->dev, port->slave->dev, "Port %d did not find a suitable aggregator\n", port->actor_port_number); ----analysis: The fault scenario goes to this branch, and port->aggregator remains NULL. } } /* if all aggregator's ports are READY_N == TRUE, set ready=TRUE * in all aggregator's ports, else set ready=FALSE in all * aggregator's ports */ __set_agg_ports_ready(port->aggregator, __agg_ports_are_ready(port->aggregator)); ----analysis: port->aggregator is still NULL, which causes problem. aggregator = __get_first_agg(port); ad_agg_selection_logic(aggregator, update_slave_arr); if (!port->aggregator->is_active) port->actor_oper_port_state &= ~LACP_STATE_SYNCHRONIZATION; } Thanks.

2 3

[PATCH for-next] RDMA/hns: Add support of direct wqe
by Weihang Li 30 Jan '21

30 Jan '21

From: Yixing Liu <liuyixing1(a)huawei.com> Direct wqe is a mechanism to fill wqe directly into the hardware. In the case of light load, the wqe will be filled into pcie bar space of the hardware, this will reduce one memory access operation and therefore reduce the latency. Signed-off-by: Yixing Liu <liuyixing1(a)huawei.com> Signed-off-by: Lang Cheng <chenglang(a)huawei.com> Signed-off-by: Weihang Li <liweihang(a)huawei.com> --- drivers/infiniband/hw/hns/hns_roce_device.h | 6 ++++ drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 44 ++++++++++++++++++++++++++++- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 14 +++++++++ 3 files changed, 63 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h index c46b330..6abf7d8 100644 --- a/drivers/infiniband/hw/hns/hns_roce_device.h +++ b/drivers/infiniband/hw/hns/hns_roce_device.h @@ -90,6 +90,7 @@ #define HNS_ROCE_MAX_PORTS 6 #define HNS_ROCE_GID_SIZE 16 #define HNS_ROCE_SGE_SIZE 16 +#define HNS_ROCE_DWQE_SIZE 65536 #define HNS_ROCE_HOP_NUM_0 0xff @@ -644,6 +645,10 @@ struct hns_roce_work { u32 queue_num; }; +enum { + HNS_ROCE_QP_CAP_DIRECT_WQE = BIT(5), +}; + struct hns_roce_qp { struct ib_qp ibqp; struct hns_roce_wq rq; @@ -986,6 +991,7 @@ struct hns_roce_dev { struct mutex pgdir_mutex; int irq[HNS_ROCE_MAX_IRQ_NUM]; u8 __iomem *reg_base; + void __iomem *mem_base; struct hns_roce_caps caps; struct xarray qp_table_xa; diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c index 4c06889..0d26079 100644 --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c @@ -503,6 +503,8 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp, if (ret) return ret; + qp->sl = to_hr_ah(ud_wr(wr)->ah)->av.sl; + set_extend_sge(qp, wr->sg_list, &curr_idx, valid_num_sge); /* @@ -635,6 +637,8 @@ static inline void update_sq_db(struct hns_roce_dev *hr_dev, V2_DB_BYTE_4_TAG_S, qp->doorbell_qpn); roce_set_field(sq_db.byte_4, V2_DB_BYTE_4_CMD_M, V2_DB_BYTE_4_CMD_S, HNS_ROCE_V2_SQ_DB); + /* indicates data on new BAR, 0 : SQ doorbell, 1 : DWQE */ + roce_set_bit(sq_db.byte_4, V2_DB_FLAG_S, 0); roce_set_field(sq_db.parameter, V2_DB_PARAMETER_IDX_M, V2_DB_PARAMETER_IDX_S, qp->sq.head); roce_set_field(sq_db.parameter, V2_DB_PARAMETER_SL_M, @@ -644,6 +648,38 @@ static inline void update_sq_db(struct hns_roce_dev *hr_dev, } } +static void hns_roce_write512(struct hns_roce_dev *hr_dev, u64 *val, + u64 __iomem *dest) +{ +#define HNS_ROCE_WRITE_TIMES 8 + struct hns_roce_v2_priv *priv = (struct hns_roce_v2_priv *)hr_dev->priv; + struct hnae3_handle *handle = priv->handle; + const struct hnae3_ae_ops *ops = handle->ae_algo->ops; + int i; + + if (!hr_dev->dis_db && !ops->get_hw_reset_stat(handle)) + for (i = 0; i < HNS_ROCE_WRITE_TIMES; i++) + writeq_relaxed(*(val + i), dest + i); +} + +static void write_dwqe(struct hns_roce_dev *hr_dev, struct hns_roce_qp *qp, + void *wqe) +{ + struct hns_roce_v2_rc_send_wqe *rc_sq_wqe = wqe; + + /* All kinds of DirectWQE have the same header field layout */ + roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_FLAG_S, 1); + roce_set_field(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_DB_SL_L_M, + V2_RC_SEND_WQE_BYTE_4_DB_SL_L_S, qp->sl); + roce_set_field(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_DB_SL_H_M, + V2_RC_SEND_WQE_BYTE_4_DB_SL_H_S, qp->sl >> 2); + roce_set_field(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_M, + V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_S, qp->sq.head); + + hns_roce_write512(hr_dev, wqe, hr_dev->mem_base + + HNS_ROCE_DWQE_SIZE * qp->ibqp.qp_num); +} + static int hns_roce_v2_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, const struct ib_send_wr **bad_wr) @@ -710,7 +746,12 @@ static int hns_roce_v2_post_send(struct ib_qp *ibqp, qp->next_sge = sge_idx; /* Memory barrier */ wmb(); - update_sq_db(hr_dev, qp); + + if (nreq == 1 && qp->sq.head == qp->sq.tail + 1 && + (qp->en_flags & HNS_ROCE_QP_CAP_DIRECT_WQE)) + write_dwqe(hr_dev, qp, wqe); + else + update_sq_db(hr_dev, qp); } spin_unlock_irqrestore(&qp->sq.lock, flags); @@ -6286,6 +6327,7 @@ static void hns_roce_hw_v2_get_cfg(struct hns_roce_dev *hr_dev, /* Get info from NIC driver. */ hr_dev->reg_base = handle->rinfo.roce_io_base; + hr_dev->mem_base = handle->rinfo.roce_mem_base; hr_dev->caps.num_ports = 1; hr_dev->iboe.netdevs[0] = handle->rinfo.netdev; hr_dev->iboe.phy_port[0] = 0; diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.h b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h index bdaccf8..badc1c9 100644 --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.h +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h @@ -97,6 +97,7 @@ #define HNS_ROCE_V2_PAGE_SIZE_SUPPORTED 0xFFFFF000 #define HNS_ROCE_V2_MAX_INNER_MTPT_NUM 2 #define HNS_ROCE_INVALID_LKEY 0x100 + #define HNS_ROCE_CMQ_TX_TIMEOUT 30000 #define HNS_ROCE_V2_UC_RC_SGE_NUM_IN_WQE 2 #define HNS_ROCE_V2_RSV_QPS 8 @@ -1059,6 +1060,8 @@ struct hns_roce_v2_mpt_entry { #define V2_DB_BYTE_4_CMD_S 24 #define V2_DB_BYTE_4_CMD_M GENMASK(27, 24) +#define V2_DB_FLAG_S 31 + #define V2_DB_PARAMETER_IDX_S 0 #define V2_DB_PARAMETER_IDX_M GENMASK(15, 0) @@ -1155,6 +1158,15 @@ struct hns_roce_v2_rc_send_wqe { #define V2_RC_SEND_WQE_BYTE_4_OPCODE_S 0 #define V2_RC_SEND_WQE_BYTE_4_OPCODE_M GENMASK(4, 0) +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_L_S 5 +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_L_M GENMASK(6, 5) + +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_H_S 13 +#define V2_RC_SEND_WQE_BYTE_4_DB_SL_H_M GENMASK(14, 13) + +#define V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_S 15 +#define V2_RC_SEND_WQE_BYTE_4_WQE_INDEX_M GENMASK(30, 15) + #define V2_RC_SEND_WQE_BYTE_4_OWNER_S 7 #define V2_RC_SEND_WQE_BYTE_4_CQE_S 8 @@ -1177,6 +1189,8 @@ struct hns_roce_v2_rc_send_wqe { #define V2_RC_FRMR_WQE_BYTE_4_LW_S 23 +#define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31 + #define V2_RC_SEND_WQE_BYTE_16_XRC_SRQN_S 0 #define V2_RC_SEND_WQE_BYTE_16_XRC_SRQN_M GENMASK(23, 0) -- 2.8.1

2 1

[PATCH 0/5] hisi_sas: More misc patches
by John Garry 30 Jan '21

30 Jan '21

This is a collection of misc patches picked up during the latest dev cycle, targeted at 5.12 . Features include: - Some tidy-up from after recent change to expose HW queues on v2 HW - Add trace FIFO DFX debugfs support - Flush wq for driver removal - Add ability to enable debugfs support as a kernel config option Thanks! John Garry (2): scsi: hisi_sas: Remove deferred probe check in hisi_sas_v2_probe() scsi: hisi_sas: Don't check .nr_hw_queues in hisi_sas_task_prep() Luo Jiaxing (3): scsi: hisi_sas: Enable debugfs support by default scsi: hisi_sas: Flush workqueue in hisi_sas_v3_remove() scsi: hisi_sas: Add trace FIFO debugfs support drivers/scsi/hisi_sas/Kconfig | 6 + drivers/scsi/hisi_sas/hisi_sas.h | 15 ++ drivers/scsi/hisi_sas/hisi_sas_main.c | 19 +- drivers/scsi/hisi_sas/hisi_sas_v2_hw.c | 12 -- drivers/scsi/hisi_sas/hisi_sas_v3_hw.c | 252 +++++++++++++++++++++++++ 5 files changed, 286 insertions(+), 18 deletions(-) -- 2.26.2

2 7

Question about the mechanism of RoCEv2 VLAN validation
by liweihang 29 Jan '21

29 Jan '21

Hi All, I'm confusing at the VLAN validation mechanism of RoCEv2. Assuming that we have two nodes with an HCA that supports RoCE v2. And we add a VLAN (id = 1) on each nodes, but they are of different network segments. The IP and VLAN configuration are as follows: NODE_A NODE_B +--------+---------------+------+ +--------+---------------+------+ | device | IP | VLAN | | device | IP | VLAN | +--------+---------------+------+ +--------+---------------+------+ | eth0 | 192.168.97.1 | 0 | /----| eth0 | 192.168.100.2 | 0 | +--------+---------------+------+ / +--------+---------------+------+ | eth0.1 | 192.168.100.3 | 1 |---/ | eth0.1 | 192.168.98.2 | 1 | +--------+---------------+------+ +--------+---------------+------+ Now I try to ping eth0 on NODE_B from eth0.1 on NODE_A, of cource it fails becauce these devices are using different VLAN ID. Then I do some tests on RoCE, the first one is a simple RC send test: NODE_A: ib_send_bw -d mlx5_0 -x 5 (the sgid 5 belongs to eth0.1) NODE_B: ib_send_bw -d mlx5_0 -x 3 <server ip> (the sgid 3 belongs to eth0) The result is as expected, the RoCEv2 packet with unmatched VLAN ID was dropped. I think the reason is that for RC service, the VLAN information of a QP is recorded in QPC, and the HCA can check it when receiving a packet. But when I run a simple UD send test: NODE_A: ib_send_bw -d mlx5_0 -x 5 -c UD (the sgid 5 belongs to eth0.1) NODE_B: ib_send_bw -d mlx5_0 -x 3 -c UD <server ip> (the sgid 3 belongs to eth0) The test ends without error successfully. So the question is, RoCEv2 is based on Ethernet, shouldn't a RoCEv2 node check the VLAN ID of every incoming packets? UD is connectionless-oriented, an UD QP won't record VLAN info in it's QPC, so how to achieve the checking mechanism? Or a UD QP should just ignore the unmatched VLAN ID? I didn't find any info from the IB specification, I'd appreciate it if someone could help explain it. Thanks Weihang

2 2

[RFC PATCH v2] sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2
by Barry Song 29 Jan '21

29 Jan '21

As long as NUMA diameter > 2, building sched_domain by sibling's child domain will definitely create a sched_domain with sched_group which will span out of the sched_domain +------+ +------+ +-------+ +------+ | node | 12 |node | 20 | node | 12 |node | | 0 +---------+1 +--------+ 2 +-------+3 | +------+ +------+ +-------+ +------+ domain0 node0 node1 node2 node3 domain1 node0+1 node0+1 node2+3 node2+3 + domain2 node0+1+2 | group: node0+1 | group:node2+3 <-------------------+ when node2 is added into the domain2 of node0, kernel is using the child domain of node2's domain2, which is domain1(node2+3). Node 3 is outside the span of the domain including node0+1+2. This will make load_balance() and select_task_rq_fair() run on wrong CPU and base on avg load. here we move to use the *child* domain of the *child* domain of node2's domain2 to build the sched_group. +------+ +------+ +-------+ +------+ | node | 12 |node | 20 | node | 12 |node | | 0 +---------+1 +--------+ 2 +-------+3 | +------+ +------+ +-------+ +------+ domain0 node0 node1 +- node2 node3 | domain1 node0+1 node0+1 | node2+3 node2+3 | domain2 node0+1+2 | group: node0+1 | group:node2 <-------------------+ A tricky thing is that we shouldn't use the first cpu of node2 as the unique ID for the sched_group generated by grandchild, otherwise, when this cpu becomes the balance_cpu of another sched_group of cpus other than node0, our sched_group generated by grandchild will have same ID with another sched_group generated by child in init_overlap_sched_group(): build_balance_mask(sd, sg, mask); cpu = cpumask_first_and(sched_group_span(sg), mask); sg->sgc = *per_cpu_ptr(sdd->sgc, cpu); So here we move to use the second cpu. But we can put the cpu back during load_balance by using sched_group_span rather than group_balance_mask. For the corner case, if NUMA has only one CPU, we will get to a warning WARN_ON_ONCE here: static void init_overlap_sched_group(struct sched_domain *sd, struct sched_group *sg) { if (atomic_inc_return(&sg->sgc->ref) == 1) cpumask_copy(group_balance_mask(sg), mask); else WARN_ON_ONCE(!cpumask_equal(group_balance_mask(sg), mask)); } But It is pretty much not a real case. Tested by the below topology: qemu-system-aarch64 -M virt -nographic \ -smp cpus=8 \ -numa node,cpus=0-1,nodeid=0 \ -numa node,cpus=2-3,nodeid=1 \ -numa node,cpus=4-5,nodeid=2 \ -numa node,cpus=6-7,nodeid=3 \ -numa dist,src=0,dst=1,val=12 \ -numa dist,src=0,dst=2,val=20 \ -numa dist,src=0,dst=3,val=22 \ -numa dist,src=1,dst=2,val=22 \ -numa dist,src=2,dst=3,val=12 \ -numa dist,src=1,dst=3,val=24 \ -m 4G -cpu cortex-a57 -kernel arch/arm64/boot/Image w/o patch, we get lots of "groups don't span domain->span": [ 0.802139] CPU0 attaching sched-domain(s): [ 0.802193] domain-0: span=0-1 level=MC [ 0.802443] groups: 0:{ span=0 cap=1013 }, 1:{ span=1 cap=979 } [ 0.802693] domain-1: span=0-3 level=NUMA [ 0.802731] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 } [ 0.802811] domain-2: span=0-5 level=NUMA [ 0.802829] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 } [ 0.802881] ERROR: groups don't span domain->span [ 0.803058] domain-3: span=0-7 level=NUMA [ 0.803080] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 } [ 0.804055] CPU1 attaching sched-domain(s): [ 0.804072] domain-0: span=0-1 level=MC [ 0.804096] groups: 1:{ span=1 cap=979 }, 0:{ span=0 cap=1013 } [ 0.804152] domain-1: span=0-3 level=NUMA [ 0.804170] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 } [ 0.804219] domain-2: span=0-5 level=NUMA [ 0.804236] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 } [ 0.804302] ERROR: groups don't span domain->span [ 0.804520] domain-3: span=0-7 level=NUMA [ 0.804546] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 } [ 0.804677] CPU2 attaching sched-domain(s): [ 0.804687] domain-0: span=2-3 level=MC [ 0.804705] groups: 2:{ span=2 cap=934 }, 3:{ span=3 cap=1009 } [ 0.804754] domain-1: span=0-3 level=NUMA [ 0.804772] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 } [ 0.804820] domain-2: span=0-5 level=NUMA [ 0.804836] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 } [ 0.804944] ERROR: groups don't span domain->span [ 0.805108] domain-3: span=0-7 level=NUMA [ 0.805134] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 } [ 0.805223] CPU3 attaching sched-domain(s): [ 0.805232] domain-0: span=2-3 level=MC [ 0.805249] groups: 3:{ span=3 cap=1009 }, 2:{ span=2 cap=934 } [ 0.805319] domain-1: span=0-3 level=NUMA [ 0.805336] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 } [ 0.805383] domain-2: span=0-5 level=NUMA [ 0.805399] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 } [ 0.805458] ERROR: groups don't span domain->span [ 0.805605] domain-3: span=0-7 level=NUMA [ 0.805626] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 } [ 0.805712] CPU4 attaching sched-domain(s): [ 0.805721] domain-0: span=4-5 level=MC [ 0.805738] groups: 4:{ span=4 cap=984 }, 5:{ span=5 cap=924 } [ 0.805787] domain-1: span=4-7 level=NUMA [ 0.805803] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 } [ 0.805851] domain-2: span=0-1,4-7 level=NUMA [ 0.805867] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 } [ 0.805915] ERROR: groups don't span domain->span [ 0.806108] domain-3: span=0-7 level=NUMA [ 0.806130] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 } [ 0.806214] CPU5 attaching sched-domain(s): [ 0.806222] domain-0: span=4-5 level=MC [ 0.806240] groups: 5:{ span=5 cap=924 }, 4:{ span=4 cap=984 } [ 0.806841] domain-1: span=4-7 level=NUMA [ 0.806866] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 } [ 0.806934] domain-2: span=0-1,4-7 level=NUMA [ 0.806953] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 } [ 0.807004] ERROR: groups don't span domain->span [ 0.807312] domain-3: span=0-7 level=NUMA [ 0.807386] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 } [ 0.807686] CPU6 attaching sched-domain(s): [ 0.807710] domain-0: span=6-7 level=MC [ 0.807750] groups: 6:{ span=6 cap=1017 }, 7:{ span=7 cap=1012 } [ 0.807840] domain-1: span=4-7 level=NUMA [ 0.807870] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 } [ 0.807952] domain-2: span=0-1,4-7 level=NUMA [ 0.807985] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 } [ 0.808045] ERROR: groups don't span domain->span [ 0.808257] domain-3: span=0-7 level=NUMA [ 0.808571] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6125 }, 2:{ span=0-5 mask=2-3 cap=5899 } [ 0.808848] CPU7 attaching sched-domain(s): [ 0.808860] domain-0: span=6-7 level=MC [ 0.808880] groups: 7:{ span=7 cap=1012 }, 6:{ span=6 cap=1017 } [ 0.808953] domain-1: span=4-7 level=NUMA [ 0.808974] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 } [ 0.809034] domain-2: span=0-1,4-7 level=NUMA [ 0.809055] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 } [ 0.809128] ERROR: groups don't span domain->span [ 0.810361] domain-3: span=0-7 level=NUMA [ 0.810400] groups: 6:{ span=0-1,4-7 mask=6-7 cap=5961 }, 2:{ span=0-5 mask=2-3 cap=5903 } w/ patch, we get no "groups don't span domain->span": [ 0.868907] CPU0 attaching sched-domain(s): [ 0.868962] domain-0: span=0-1 level=MC [ 0.869179] groups: 0:{ span=0 cap=1013 }, 1:{ span=1 cap=983 } [ 0.869405] domain-1: span=0-3 level=NUMA [ 0.869438] groups: 0:{ span=0-1 cap=1996 }, 2:{ span=2-3 cap=2006 } [ 0.869542] domain-2: span=0-5 level=NUMA [ 0.869559] groups: 0:{ span=0-3 cap=4002 }, 5:{ span=4-5 cap=2048 } [ 0.869603] domain-3: span=0-7 level=NUMA [ 0.869618] groups: 0:{ span=0-5 mask=0-1 cap=5980 }, 6:{ span=4-7 mask=6-7 cap=4016 } [ 0.870303] CPU1 attaching sched-domain(s): [ 0.870314] domain-0: span=0-1 level=MC [ 0.870334] groups: 1:{ span=1 cap=983 }, 0:{ span=0 cap=1013 } [ 0.870381] domain-1: span=0-3 level=NUMA [ 0.870396] groups: 0:{ span=0-1 cap=1996 }, 2:{ span=2-3 cap=2006 } [ 0.870440] domain-2: span=0-5 level=NUMA [ 0.870454] groups: 0:{ span=0-3 cap=4002 }, 5:{ span=4-5 cap=2048 } [ 0.870507] domain-3: span=0-7 level=NUMA [ 0.870530] groups: 0:{ span=0-5 mask=0-1 cap=5980 }, 6:{ span=4-7 mask=6-7 cap=4016 } [ 0.870611] CPU2 attaching sched-domain(s): [ 0.870619] domain-0: span=2-3 level=MC [ 0.870634] groups: 2:{ span=2 cap=1007 }, 3:{ span=3 cap=999 } [ 0.870677] domain-1: span=0-3 level=NUMA [ 0.870691] groups: 2:{ span=2-3 cap=2006 }, 0:{ span=0-1 cap=1996 } [ 0.870734] domain-2: span=0-5 level=NUMA [ 0.870748] groups: 2:{ span=0-3 mask=2-3 cap=4054 }, 5:{ span=4-5 cap=2048 } [ 0.870795] domain-3: span=0-7 level=NUMA [ 0.870809] groups: 2:{ span=0-5 mask=2-3 cap=6032 }, 6:{ span=0-1,4-7 mask=6-7 cap=6064 } [ 0.870913] CPU3 attaching sched-domain(s): [ 0.870921] domain-0: span=2-3 level=MC [ 0.870936] groups: 3:{ span=3 cap=999 }, 2:{ span=2 cap=1007 } [ 0.870979] domain-1: span=0-3 level=NUMA [ 0.870993] groups: 2:{ span=2-3 cap=2006 }, 0:{ span=0-1 cap=1996 } [ 0.871035] domain-2: span=0-5 level=NUMA [ 0.871049] groups: 2:{ span=0-3 mask=2-3 cap=4054 }, 5:{ span=4-5 cap=2048 } [ 0.871096] domain-3: span=0-7 level=NUMA [ 0.871110] groups: 2:{ span=0-5 mask=2-3 cap=6032 }, 6:{ span=0-1,4-7 mask=6-7 cap=6064 } [ 0.871177] CPU4 attaching sched-domain(s): [ 0.871185] domain-0: span=4-5 level=MC [ 0.871200] groups: 4:{ span=4 cap=977 }, 5:{ span=5 cap=1001 } [ 0.871243] domain-1: span=4-7 level=NUMA [ 0.871257] groups: 4:{ span=4-5 cap=1978 }, 6:{ span=6-7 cap=1968 } [ 0.871300] domain-2: span=0-1,4-7 level=NUMA [ 0.871314] groups: 4:{ span=4-7 cap=3946 }, 1:{ span=0-1 cap=2048 } [ 0.871356] domain-3: span=0-7 level=NUMA [ 0.871370] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5994 }, 2:{ span=0-3 mask=2-3 cap=4054 } [ 0.871436] CPU5 attaching sched-domain(s): [ 0.871443] domain-0: span=4-5 level=MC [ 0.871457] groups: 5:{ span=5 cap=1001 }, 4:{ span=4 cap=977 } [ 0.871512] domain-1: span=4-7 level=NUMA [ 0.871893] groups: 4:{ span=4-5 cap=1978 }, 6:{ span=6-7 cap=1968 } [ 0.871949] domain-2: span=0-1,4-7 level=NUMA [ 0.871966] groups: 4:{ span=4-7 cap=3946 }, 1:{ span=0-1 cap=2048 } [ 0.872010] domain-3: span=0-7 level=NUMA [ 0.872025] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5994 }, 2:{ span=0-3 mask=2-3 cap=4054 } [ 0.872115] CPU6 attaching sched-domain(s): [ 0.872123] domain-0: span=6-7 level=MC [ 0.872139] groups: 6:{ span=6 cap=993 }, 7:{ span=7 cap=975 } [ 0.872186] domain-1: span=4-7 level=NUMA [ 0.872202] groups: 6:{ span=6-7 cap=1968 }, 4:{ span=4-5 cap=1978 } [ 0.872246] domain-2: span=0-1,4-7 level=NUMA [ 0.872260] groups: 6:{ span=4-7 mask=6-7 cap=4016 }, 1:{ span=0-1 cap=2048 } [ 0.872309] domain-3: span=0-7 level=NUMA [ 0.872323] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6064 }, 2:{ span=0-5 mask=2-3 cap=6032 } [ 0.872392] CPU7 attaching sched-domain(s): [ 0.872399] domain-0: span=6-7 level=MC [ 0.872414] groups: 7:{ span=7 cap=975 }, 6:{ span=6 cap=993 } [ 0.872458] domain-1: span=4-7 level=NUMA [ 0.872472] groups: 6:{ span=6-7 cap=1968 }, 4:{ span=4-5 cap=1978 } [ 0.872662] domain-2: span=0-1,4-7 level=NUMA [ 0.872685] groups: 6:{ span=4-7 mask=6-7 cap=4016 }, 1:{ span=0-1 cap=2048 } [ 0.872737] domain-3: span=0-7 level=NUMA [ 0.872752] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6064 }, 2:{ span=0-5 mask=2-3 cap=6032 } Reported-by: Valentin Schneider <valentin.schneider(a)arm.com> Cc: Meelis Roos <mroos(a)linux.ee> Signed-off-by: Barry Song <song.bao.hua(a)hisilicon.com> --- -v2: fix the span of sched_group rather than hacking the load balance to compensate the topology problem(thanks for the comments of Vincent and Valentin) v1 link: https://lore.kernel.org/lkml/20210115203632.34396-1-song.bao.hua@hisilicon.… kernel/sched/fair.c | 9 ++++- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 84 ++++++++++++++++++++++++++--------------- 3 files changed, 63 insertions(+), 31 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f183dba4961e..3ce815045793 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9538,6 +9538,7 @@ static int should_we_balance(struct lb_env *env) { struct sched_group *sg = env->sd->groups; int cpu; + struct cpumask *mask; /* * Ensure the balancing environment is consistent; can happen @@ -9553,8 +9554,14 @@ static int should_we_balance(struct lb_env *env) if (env->idle == CPU_NEWLY_IDLE) return 1; + /* + * in build_balance_mask(), we removed the first cpu from the + * balance_mask of the sched_group generated by grandchild; + * the first cpu should be able to pull task if it is idle. + */ + mask = sg->from_grandchild ? sched_group_span(sg) : group_balance_mask(sg); /* Try to find first idle CPU */ - for_each_cpu_and(cpu, group_balance_mask(sg), env->cpus) { + for_each_cpu_and(cpu, mask, env->cpus) { if (!idle_cpu(cpu)) continue; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 12ada79d40f3..3ad282fdc5b1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1508,6 +1508,7 @@ struct sched_group { unsigned int group_weight; struct sched_group_capacity *sgc; int asym_prefer_cpu; /* CPU of highest priority in group */ + int from_grandchild; /* this sched_group comes from grandchild */ /* * The CPUs this group covers. diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 5d3675c7a76b..23099f6dc034 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -93,7 +93,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level, group->sgc->id, cpumask_pr_args(sched_group_span(group))); - if ((sd->flags & SD_OVERLAP) && + if ((sd->flags & SD_OVERLAP) && !group->from_grandchild && !cpumask_equal(group_balance_mask(group), sched_group_span(group))) { printk(KERN_CONT " mask=%*pbl", cpumask_pr_args(group_balance_mask(group))); @@ -723,35 +723,6 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu) for (tmp = sd; tmp; tmp = tmp->parent) numa_distance += !!(tmp->flags & SD_NUMA); - /* - * FIXME: Diameter >=3 is misrepresented. - * - * Smallest diameter=3 topology is: - * - * node 0 1 2 3 - * 0: 10 20 30 40 - * 1: 20 10 20 30 - * 2: 30 20 10 20 - * 3: 40 30 20 10 - * - * 0 --- 1 --- 2 --- 3 - * - * NUMA-3 0-3 N/A N/A 0-3 - * groups: {0-2},{1-3} {1-3},{0-2} - * - * NUMA-2 0-2 0-3 0-3 1-3 - * groups: {0-1},{1-3} {0-2},{2-3} {1-3},{0-1} {2-3},{0-2} - * - * NUMA-1 0-1 0-2 1-3 2-3 - * groups: {0},{1} {1},{2},{0} {2},{3},{1} {3},{2} - * - * NUMA-0 0 1 2 3 - * - * The NUMA-2 groups for nodes 0 and 3 are obviously buggered, as the - * group span isn't a subset of the domain span. - */ - WARN_ONCE(numa_distance > 2, "Shortest NUMA path spans too many nodes\n"); - sched_domain_debug(sd, cpu); rq_attach_root(rq, rd); @@ -906,6 +877,7 @@ build_balance_mask(struct sched_domain *sd, struct sched_group *sg, struct cpuma cpumask_clear(mask); for_each_cpu(i, sg_span) { + int from_grandchild = 0; sibling = *per_cpu_ptr(sdd->sd, i); /* @@ -916,10 +888,27 @@ build_balance_mask(struct sched_domain *sd, struct sched_group *sg, struct cpuma if (!sibling->child) continue; + while (sibling->child && + !cpumask_subset(sched_domain_span(sibling->child), + sched_domain_span(sd))) { + sibling = sibling->child; + from_grandchild = 1; + } + /* If we would not end up here, we can't continue from here */ if (!cpumask_equal(sg_span, sched_domain_span(sibling->child))) continue; + sg->from_grandchild = from_grandchild; + /* + * use cpu other than the first one as the unique ID of this group + * otherwise, the ID will conflict with another sched_group coming + * from child rather than grandchild + */ + if (from_grandchild && i == cpumask_first(sg_span) && + cpumask_weight(sg_span) > 1) + continue; + cpumask_set_cpu(i, mask); } @@ -1015,6 +1004,41 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu) if (!cpumask_test_cpu(i, sched_domain_span(sibling))) continue; + /* + * for NUMA diameter >= 3, building sched_domain by sibling's + * child's child domain to prevent sched_group from spanning + * out of sched_domain + * if we don't do this, Diameter >=3 is misrepresented: + * + * Smallest diameter=3 topology is: + * + * node 0 1 2 3 + * 0: 10 20 30 40 + * 1: 20 10 20 30 + * 2: 30 20 10 20 + * 3: 40 30 20 10 + * + * 0 --- 1 --- 2 --- 3 + * + * NUMA-3 0-3 N/A N/A 0-3 + * groups: {0-2},{1-3} {1-3},{0-2} + * + * NUMA-2 0-2 0-3 0-3 1-3 + * groups: {0-1},{1-3} {0-2},{2-3} {1-3},{0-1} {2-3},{0-2} + * + * NUMA-1 0-1 0-2 1-3 2-3 + * groups: {0},{1} {1},{2},{0} {2},{3},{1} {3},{2} + * + * NUMA-0 0 1 2 3 + * + * The NUMA-2 groups for nodes 0 and 3 are obviously buggered, as the + * group span isn't a subset of the domain span. + */ + while (sibling->child && + !cpumask_subset(sched_domain_span(sibling->child), + span)) + sibling = sibling->child; + sg = build_group_from_child_sched_domain(sibling, cpu); if (!sg) goto fail; -- 2.25.1

2 1

[PATCH v2 0/4] perf vendor events arm64: Some tidy-up/refactoring
by John Garry 29 Jan '21

29 Jan '21

There is much event duplication in the common and uarch events for A76 and Ampere eMag support, so factor out into a common JSON. Since the wording for events may differ between CPU datasheet and the architecture reference manual, the current wording is kept. This is unless there are minor differences. In addition, event names are renamed to be same as architecture reference manual, to keep commonality. Also a minor fix is included for the Ampere eMag JSON. Differences to v1: - tidy some ',' misplaced John Garry (4): perf vendor events arm64: Fix Ampere eMag event typo perf vendor events arm64: Add common and uarch event JSON perf vendor events arm64: Reference common and uarch events for Ampere eMag perf vendor events arm64: Reference common and uarch events for A76 .../arch/arm64/ampere/emag/branch.json | 8 +- .../arch/arm64/ampere/emag/bus.json | 5 +- .../arch/arm64/ampere/emag/cache.json | 58 +--- .../arch/arm64/ampere/emag/clock.json | 4 +- .../arch/arm64/ampere/emag/exception.json | 10 +- .../arch/arm64/ampere/emag/instruction.json | 34 +-- .../arch/arm64/ampere/emag/memory.json | 11 +- .../arch/arm64/arm/cortex-a76-n1/branch.json | 12 +- .../arch/arm64/arm/cortex-a76-n1/bus.json | 19 +- .../arch/arm64/arm/cortex-a76-n1/cache.json | 118 +++------ .../arm64/arm/cortex-a76-n1/exception.json | 10 +- .../arm64/arm/cortex-a76-n1/instruction.json | 45 +--- .../arch/arm64/arm/cortex-a76-n1/memory.json | 6 +- .../arch/arm64/arm/cortex-a76-n1/other.json | 4 +- .../arm64/arm/cortex-a76-n1/pipeline.json | 12 +- .../arm64/armv8-common-and-microarch.json | 248 ++++++++++++++++++ 16 files changed, 356 insertions(+), 248 deletions(-) create mode 100644 tools/perf/pmu-events/arch/arm64/armv8-common-and-microarch.json -- 2.26.2

2 5

[PATCH v2 for-next 0/3] RDMA/hns: Refactor codes about memory
by Weihang Li 29 Jan '21

29 Jan '21

This series refactors memory related codes, including MR, MTR(Memory Translate Region) and MPT(Memory Protection Table). Changes since v1: * Add static keyword for hns_roce_mr_free(). * Link: https://patchwork.kernel.org/project/linux-rdma/cover/1611236882-41498-1-gi… Lang Cheng (2): RDMA/hns: Optimize the MR registration process RDMA/hns: Use new interface to set MPT related fields Xi Wang (1): RDMA/hns: Refactor the MTR creation flow drivers/infiniband/hw/hns/hns_roce_common.h | 22 ++ drivers/infiniband/hw/hns/hns_roce_cq.c | 1 - drivers/infiniband/hw/hns/hns_roce_device.h | 5 +- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 77 +++--- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 39 +++ drivers/infiniband/hw/hns/hns_roce_mr.c | 401 +++++++++++----------------- drivers/infiniband/hw/hns/hns_roce_qp.c | 1 - drivers/infiniband/hw/hns/hns_roce_srq.c | 2 - 8 files changed, 257 insertions(+), 291 deletions(-) -- 2.8.1

2 4

[PATCH] iommu: Check dev->iommu in iommu_dev_xxx functions
by Shameer Kolothum 28 Jan '21

28 Jan '21

The device iommu probe/attach might have failed leaving dev->iommu to NULL and device drivers may still invoke these functions resulting a crash in iommu vendor driver code. Hence make sure we check that. Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> --- drivers/iommu/iommu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index ffeebda8d6de..cb68153c5cc0 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2867,7 +2867,7 @@ bool iommu_dev_has_feature(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_has_feat) + if (dev->iommu && ops && ops->dev_has_feat) return ops->dev_has_feat(dev, feat); return false; @@ -2878,7 +2878,7 @@ int iommu_dev_enable_feature(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_enable_feat) + if (dev->iommu && ops && ops->dev_enable_feat) return ops->dev_enable_feat(dev, feat); return -ENODEV; @@ -2894,7 +2894,7 @@ int iommu_dev_disable_feature(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_disable_feat) + if (dev->iommu && ops && ops->dev_disable_feat) return ops->dev_disable_feat(dev, feat); return -EBUSY; @@ -2905,7 +2905,7 @@ bool iommu_dev_feature_enabled(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_feat_enabled) + if (dev->iommu && ops && ops->dev_feat_enabled) return ops->dev_feat_enabled(dev, feat); return false; -- 2.17.1

4 4