January 2021 - Linuxarm - mailweb.openeuler.org

Question about the mechanism of RoCEv2 VLAN validation
by liweihang 29 Jan '21

29 Jan '21

Hi All, I'm confusing at the VLAN validation mechanism of RoCEv2. Assuming that we have two nodes with an HCA that supports RoCE v2. And we add a VLAN (id = 1) on each nodes, but they are of different network segments. The IP and VLAN configuration are as follows: NODE_A NODE_B +--------+---------------+------+ +--------+---------------+------+ | device | IP | VLAN | | device | IP | VLAN | +--------+---------------+------+ +--------+---------------+------+ | eth0 | 192.168.97.1 | 0 | /----| eth0 | 192.168.100.2 | 0 | +--------+---------------+------+ / +--------+---------------+------+ | eth0.1 | 192.168.100.3 | 1 |---/ | eth0.1 | 192.168.98.2 | 1 | +--------+---------------+------+ +--------+---------------+------+ Now I try to ping eth0 on NODE_B from eth0.1 on NODE_A, of cource it fails becauce these devices are using different VLAN ID. Then I do some tests on RoCE, the first one is a simple RC send test: NODE_A: ib_send_bw -d mlx5_0 -x 5 (the sgid 5 belongs to eth0.1) NODE_B: ib_send_bw -d mlx5_0 -x 3 <server ip> (the sgid 3 belongs to eth0) The result is as expected, the RoCEv2 packet with unmatched VLAN ID was dropped. I think the reason is that for RC service, the VLAN information of a QP is recorded in QPC, and the HCA can check it when receiving a packet. But when I run a simple UD send test: NODE_A: ib_send_bw -d mlx5_0 -x 5 -c UD (the sgid 5 belongs to eth0.1) NODE_B: ib_send_bw -d mlx5_0 -x 3 -c UD <server ip> (the sgid 3 belongs to eth0) The test ends without error successfully. So the question is, RoCEv2 is based on Ethernet, shouldn't a RoCEv2 node check the VLAN ID of every incoming packets? UD is connectionless-oriented, an UD QP won't record VLAN info in it's QPC, so how to achieve the checking mechanism? Or a UD QP should just ignore the unmatched VLAN ID? I didn't find any info from the IB specification, I'd appreciate it if someone could help explain it. Thanks Weihang

2 2

[RFC PATCH v2] sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2
by Barry Song 29 Jan '21

29 Jan '21

As long as NUMA diameter > 2, building sched_domain by sibling's child domain will definitely create a sched_domain with sched_group which will span out of the sched_domain +------+ +------+ +-------+ +------+ | node | 12 |node | 20 | node | 12 |node | | 0 +---------+1 +--------+ 2 +-------+3 | +------+ +------+ +-------+ +------+ domain0 node0 node1 node2 node3 domain1 node0+1 node0+1 node2+3 node2+3 + domain2 node0+1+2 | group: node0+1 | group:node2+3 <-------------------+ when node2 is added into the domain2 of node0, kernel is using the child domain of node2's domain2, which is domain1(node2+3). Node 3 is outside the span of the domain including node0+1+2. This will make load_balance() and select_task_rq_fair() run on wrong CPU and base on avg load. here we move to use the *child* domain of the *child* domain of node2's domain2 to build the sched_group. +------+ +------+ +-------+ +------+ | node | 12 |node | 20 | node | 12 |node | | 0 +---------+1 +--------+ 2 +-------+3 | +------+ +------+ +-------+ +------+ domain0 node0 node1 +- node2 node3 | domain1 node0+1 node0+1 | node2+3 node2+3 | domain2 node0+1+2 | group: node0+1 | group:node2 <-------------------+ A tricky thing is that we shouldn't use the first cpu of node2 as the unique ID for the sched_group generated by grandchild, otherwise, when this cpu becomes the balance_cpu of another sched_group of cpus other than node0, our sched_group generated by grandchild will have same ID with another sched_group generated by child in init_overlap_sched_group(): build_balance_mask(sd, sg, mask); cpu = cpumask_first_and(sched_group_span(sg), mask); sg->sgc = *per_cpu_ptr(sdd->sgc, cpu); So here we move to use the second cpu. But we can put the cpu back during load_balance by using sched_group_span rather than group_balance_mask. For the corner case, if NUMA has only one CPU, we will get to a warning WARN_ON_ONCE here: static void init_overlap_sched_group(struct sched_domain *sd, struct sched_group *sg) { if (atomic_inc_return(&sg->sgc->ref) == 1) cpumask_copy(group_balance_mask(sg), mask); else WARN_ON_ONCE(!cpumask_equal(group_balance_mask(sg), mask)); } But It is pretty much not a real case. Tested by the below topology: qemu-system-aarch64 -M virt -nographic \ -smp cpus=8 \ -numa node,cpus=0-1,nodeid=0 \ -numa node,cpus=2-3,nodeid=1 \ -numa node,cpus=4-5,nodeid=2 \ -numa node,cpus=6-7,nodeid=3 \ -numa dist,src=0,dst=1,val=12 \ -numa dist,src=0,dst=2,val=20 \ -numa dist,src=0,dst=3,val=22 \ -numa dist,src=1,dst=2,val=22 \ -numa dist,src=2,dst=3,val=12 \ -numa dist,src=1,dst=3,val=24 \ -m 4G -cpu cortex-a57 -kernel arch/arm64/boot/Image w/o patch, we get lots of "groups don't span domain->span": [ 0.802139] CPU0 attaching sched-domain(s): [ 0.802193] domain-0: span=0-1 level=MC [ 0.802443] groups: 0:{ span=0 cap=1013 }, 1:{ span=1 cap=979 } [ 0.802693] domain-1: span=0-3 level=NUMA [ 0.802731] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 } [ 0.802811] domain-2: span=0-5 level=NUMA [ 0.802829] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 } [ 0.802881] ERROR: groups don't span domain->span [ 0.803058] domain-3: span=0-7 level=NUMA [ 0.803080] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 } [ 0.804055] CPU1 attaching sched-domain(s): [ 0.804072] domain-0: span=0-1 level=MC [ 0.804096] groups: 1:{ span=1 cap=979 }, 0:{ span=0 cap=1013 } [ 0.804152] domain-1: span=0-3 level=NUMA [ 0.804170] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 } [ 0.804219] domain-2: span=0-5 level=NUMA [ 0.804236] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 } [ 0.804302] ERROR: groups don't span domain->span [ 0.804520] domain-3: span=0-7 level=NUMA [ 0.804546] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 } [ 0.804677] CPU2 attaching sched-domain(s): [ 0.804687] domain-0: span=2-3 level=MC [ 0.804705] groups: 2:{ span=2 cap=934 }, 3:{ span=3 cap=1009 } [ 0.804754] domain-1: span=0-3 level=NUMA [ 0.804772] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 } [ 0.804820] domain-2: span=0-5 level=NUMA [ 0.804836] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 } [ 0.804944] ERROR: groups don't span domain->span [ 0.805108] domain-3: span=0-7 level=NUMA [ 0.805134] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 } [ 0.805223] CPU3 attaching sched-domain(s): [ 0.805232] domain-0: span=2-3 level=MC [ 0.805249] groups: 3:{ span=3 cap=1009 }, 2:{ span=2 cap=934 } [ 0.805319] domain-1: span=0-3 level=NUMA [ 0.805336] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 } [ 0.805383] domain-2: span=0-5 level=NUMA [ 0.805399] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 } [ 0.805458] ERROR: groups don't span domain->span [ 0.805605] domain-3: span=0-7 level=NUMA [ 0.805626] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 } [ 0.805712] CPU4 attaching sched-domain(s): [ 0.805721] domain-0: span=4-5 level=MC [ 0.805738] groups: 4:{ span=4 cap=984 }, 5:{ span=5 cap=924 } [ 0.805787] domain-1: span=4-7 level=NUMA [ 0.805803] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 } [ 0.805851] domain-2: span=0-1,4-7 level=NUMA [ 0.805867] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 } [ 0.805915] ERROR: groups don't span domain->span [ 0.806108] domain-3: span=0-7 level=NUMA [ 0.806130] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 } [ 0.806214] CPU5 attaching sched-domain(s): [ 0.806222] domain-0: span=4-5 level=MC [ 0.806240] groups: 5:{ span=5 cap=924 }, 4:{ span=4 cap=984 } [ 0.806841] domain-1: span=4-7 level=NUMA [ 0.806866] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 } [ 0.806934] domain-2: span=0-1,4-7 level=NUMA [ 0.806953] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 } [ 0.807004] ERROR: groups don't span domain->span [ 0.807312] domain-3: span=0-7 level=NUMA [ 0.807386] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 } [ 0.807686] CPU6 attaching sched-domain(s): [ 0.807710] domain-0: span=6-7 level=MC [ 0.807750] groups: 6:{ span=6 cap=1017 }, 7:{ span=7 cap=1012 } [ 0.807840] domain-1: span=4-7 level=NUMA [ 0.807870] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 } [ 0.807952] domain-2: span=0-1,4-7 level=NUMA [ 0.807985] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 } [ 0.808045] ERROR: groups don't span domain->span [ 0.808257] domain-3: span=0-7 level=NUMA [ 0.808571] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6125 }, 2:{ span=0-5 mask=2-3 cap=5899 } [ 0.808848] CPU7 attaching sched-domain(s): [ 0.808860] domain-0: span=6-7 level=MC [ 0.808880] groups: 7:{ span=7 cap=1012 }, 6:{ span=6 cap=1017 } [ 0.808953] domain-1: span=4-7 level=NUMA [ 0.808974] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 } [ 0.809034] domain-2: span=0-1,4-7 level=NUMA [ 0.809055] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 } [ 0.809128] ERROR: groups don't span domain->span [ 0.810361] domain-3: span=0-7 level=NUMA [ 0.810400] groups: 6:{ span=0-1,4-7 mask=6-7 cap=5961 }, 2:{ span=0-5 mask=2-3 cap=5903 } w/ patch, we get no "groups don't span domain->span": [ 0.868907] CPU0 attaching sched-domain(s): [ 0.868962] domain-0: span=0-1 level=MC [ 0.869179] groups: 0:{ span=0 cap=1013 }, 1:{ span=1 cap=983 } [ 0.869405] domain-1: span=0-3 level=NUMA [ 0.869438] groups: 0:{ span=0-1 cap=1996 }, 2:{ span=2-3 cap=2006 } [ 0.869542] domain-2: span=0-5 level=NUMA [ 0.869559] groups: 0:{ span=0-3 cap=4002 }, 5:{ span=4-5 cap=2048 } [ 0.869603] domain-3: span=0-7 level=NUMA [ 0.869618] groups: 0:{ span=0-5 mask=0-1 cap=5980 }, 6:{ span=4-7 mask=6-7 cap=4016 } [ 0.870303] CPU1 attaching sched-domain(s): [ 0.870314] domain-0: span=0-1 level=MC [ 0.870334] groups: 1:{ span=1 cap=983 }, 0:{ span=0 cap=1013 } [ 0.870381] domain-1: span=0-3 level=NUMA [ 0.870396] groups: 0:{ span=0-1 cap=1996 }, 2:{ span=2-3 cap=2006 } [ 0.870440] domain-2: span=0-5 level=NUMA [ 0.870454] groups: 0:{ span=0-3 cap=4002 }, 5:{ span=4-5 cap=2048 } [ 0.870507] domain-3: span=0-7 level=NUMA [ 0.870530] groups: 0:{ span=0-5 mask=0-1 cap=5980 }, 6:{ span=4-7 mask=6-7 cap=4016 } [ 0.870611] CPU2 attaching sched-domain(s): [ 0.870619] domain-0: span=2-3 level=MC [ 0.870634] groups: 2:{ span=2 cap=1007 }, 3:{ span=3 cap=999 } [ 0.870677] domain-1: span=0-3 level=NUMA [ 0.870691] groups: 2:{ span=2-3 cap=2006 }, 0:{ span=0-1 cap=1996 } [ 0.870734] domain-2: span=0-5 level=NUMA [ 0.870748] groups: 2:{ span=0-3 mask=2-3 cap=4054 }, 5:{ span=4-5 cap=2048 } [ 0.870795] domain-3: span=0-7 level=NUMA [ 0.870809] groups: 2:{ span=0-5 mask=2-3 cap=6032 }, 6:{ span=0-1,4-7 mask=6-7 cap=6064 } [ 0.870913] CPU3 attaching sched-domain(s): [ 0.870921] domain-0: span=2-3 level=MC [ 0.870936] groups: 3:{ span=3 cap=999 }, 2:{ span=2 cap=1007 } [ 0.870979] domain-1: span=0-3 level=NUMA [ 0.870993] groups: 2:{ span=2-3 cap=2006 }, 0:{ span=0-1 cap=1996 } [ 0.871035] domain-2: span=0-5 level=NUMA [ 0.871049] groups: 2:{ span=0-3 mask=2-3 cap=4054 }, 5:{ span=4-5 cap=2048 } [ 0.871096] domain-3: span=0-7 level=NUMA [ 0.871110] groups: 2:{ span=0-5 mask=2-3 cap=6032 }, 6:{ span=0-1,4-7 mask=6-7 cap=6064 } [ 0.871177] CPU4 attaching sched-domain(s): [ 0.871185] domain-0: span=4-5 level=MC [ 0.871200] groups: 4:{ span=4 cap=977 }, 5:{ span=5 cap=1001 } [ 0.871243] domain-1: span=4-7 level=NUMA [ 0.871257] groups: 4:{ span=4-5 cap=1978 }, 6:{ span=6-7 cap=1968 } [ 0.871300] domain-2: span=0-1,4-7 level=NUMA [ 0.871314] groups: 4:{ span=4-7 cap=3946 }, 1:{ span=0-1 cap=2048 } [ 0.871356] domain-3: span=0-7 level=NUMA [ 0.871370] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5994 }, 2:{ span=0-3 mask=2-3 cap=4054 } [ 0.871436] CPU5 attaching sched-domain(s): [ 0.871443] domain-0: span=4-5 level=MC [ 0.871457] groups: 5:{ span=5 cap=1001 }, 4:{ span=4 cap=977 } [ 0.871512] domain-1: span=4-7 level=NUMA [ 0.871893] groups: 4:{ span=4-5 cap=1978 }, 6:{ span=6-7 cap=1968 } [ 0.871949] domain-2: span=0-1,4-7 level=NUMA [ 0.871966] groups: 4:{ span=4-7 cap=3946 }, 1:{ span=0-1 cap=2048 } [ 0.872010] domain-3: span=0-7 level=NUMA [ 0.872025] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5994 }, 2:{ span=0-3 mask=2-3 cap=4054 } [ 0.872115] CPU6 attaching sched-domain(s): [ 0.872123] domain-0: span=6-7 level=MC [ 0.872139] groups: 6:{ span=6 cap=993 }, 7:{ span=7 cap=975 } [ 0.872186] domain-1: span=4-7 level=NUMA [ 0.872202] groups: 6:{ span=6-7 cap=1968 }, 4:{ span=4-5 cap=1978 } [ 0.872246] domain-2: span=0-1,4-7 level=NUMA [ 0.872260] groups: 6:{ span=4-7 mask=6-7 cap=4016 }, 1:{ span=0-1 cap=2048 } [ 0.872309] domain-3: span=0-7 level=NUMA [ 0.872323] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6064 }, 2:{ span=0-5 mask=2-3 cap=6032 } [ 0.872392] CPU7 attaching sched-domain(s): [ 0.872399] domain-0: span=6-7 level=MC [ 0.872414] groups: 7:{ span=7 cap=975 }, 6:{ span=6 cap=993 } [ 0.872458] domain-1: span=4-7 level=NUMA [ 0.872472] groups: 6:{ span=6-7 cap=1968 }, 4:{ span=4-5 cap=1978 } [ 0.872662] domain-2: span=0-1,4-7 level=NUMA [ 0.872685] groups: 6:{ span=4-7 mask=6-7 cap=4016 }, 1:{ span=0-1 cap=2048 } [ 0.872737] domain-3: span=0-7 level=NUMA [ 0.872752] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6064 }, 2:{ span=0-5 mask=2-3 cap=6032 } Reported-by: Valentin Schneider <valentin.schneider(a)arm.com> Cc: Meelis Roos <mroos(a)linux.ee> Signed-off-by: Barry Song <song.bao.hua(a)hisilicon.com> --- -v2: fix the span of sched_group rather than hacking the load balance to compensate the topology problem(thanks for the comments of Vincent and Valentin) v1 link: https://lore.kernel.org/lkml/20210115203632.34396-1-song.bao.hua@hisilicon.… kernel/sched/fair.c | 9 ++++- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 84 ++++++++++++++++++++++++++--------------- 3 files changed, 63 insertions(+), 31 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f183dba4961e..3ce815045793 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9538,6 +9538,7 @@ static int should_we_balance(struct lb_env *env) { struct sched_group *sg = env->sd->groups; int cpu; + struct cpumask *mask; /* * Ensure the balancing environment is consistent; can happen @@ -9553,8 +9554,14 @@ static int should_we_balance(struct lb_env *env) if (env->idle == CPU_NEWLY_IDLE) return 1; + /* + * in build_balance_mask(), we removed the first cpu from the + * balance_mask of the sched_group generated by grandchild; + * the first cpu should be able to pull task if it is idle. + */ + mask = sg->from_grandchild ? sched_group_span(sg) : group_balance_mask(sg); /* Try to find first idle CPU */ - for_each_cpu_and(cpu, group_balance_mask(sg), env->cpus) { + for_each_cpu_and(cpu, mask, env->cpus) { if (!idle_cpu(cpu)) continue; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 12ada79d40f3..3ad282fdc5b1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1508,6 +1508,7 @@ struct sched_group { unsigned int group_weight; struct sched_group_capacity *sgc; int asym_prefer_cpu; /* CPU of highest priority in group */ + int from_grandchild; /* this sched_group comes from grandchild */ /* * The CPUs this group covers. diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 5d3675c7a76b..23099f6dc034 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -93,7 +93,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level, group->sgc->id, cpumask_pr_args(sched_group_span(group))); - if ((sd->flags & SD_OVERLAP) && + if ((sd->flags & SD_OVERLAP) && !group->from_grandchild && !cpumask_equal(group_balance_mask(group), sched_group_span(group))) { printk(KERN_CONT " mask=%*pbl", cpumask_pr_args(group_balance_mask(group))); @@ -723,35 +723,6 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu) for (tmp = sd; tmp; tmp = tmp->parent) numa_distance += !!(tmp->flags & SD_NUMA); - /* - * FIXME: Diameter >=3 is misrepresented. - * - * Smallest diameter=3 topology is: - * - * node 0 1 2 3 - * 0: 10 20 30 40 - * 1: 20 10 20 30 - * 2: 30 20 10 20 - * 3: 40 30 20 10 - * - * 0 --- 1 --- 2 --- 3 - * - * NUMA-3 0-3 N/A N/A 0-3 - * groups: {0-2},{1-3} {1-3},{0-2} - * - * NUMA-2 0-2 0-3 0-3 1-3 - * groups: {0-1},{1-3} {0-2},{2-3} {1-3},{0-1} {2-3},{0-2} - * - * NUMA-1 0-1 0-2 1-3 2-3 - * groups: {0},{1} {1},{2},{0} {2},{3},{1} {3},{2} - * - * NUMA-0 0 1 2 3 - * - * The NUMA-2 groups for nodes 0 and 3 are obviously buggered, as the - * group span isn't a subset of the domain span. - */ - WARN_ONCE(numa_distance > 2, "Shortest NUMA path spans too many nodes\n"); - sched_domain_debug(sd, cpu); rq_attach_root(rq, rd); @@ -906,6 +877,7 @@ build_balance_mask(struct sched_domain *sd, struct sched_group *sg, struct cpuma cpumask_clear(mask); for_each_cpu(i, sg_span) { + int from_grandchild = 0; sibling = *per_cpu_ptr(sdd->sd, i); /* @@ -916,10 +888,27 @@ build_balance_mask(struct sched_domain *sd, struct sched_group *sg, struct cpuma if (!sibling->child) continue; + while (sibling->child && + !cpumask_subset(sched_domain_span(sibling->child), + sched_domain_span(sd))) { + sibling = sibling->child; + from_grandchild = 1; + } + /* If we would not end up here, we can't continue from here */ if (!cpumask_equal(sg_span, sched_domain_span(sibling->child))) continue; + sg->from_grandchild = from_grandchild; + /* + * use cpu other than the first one as the unique ID of this group + * otherwise, the ID will conflict with another sched_group coming + * from child rather than grandchild + */ + if (from_grandchild && i == cpumask_first(sg_span) && + cpumask_weight(sg_span) > 1) + continue; + cpumask_set_cpu(i, mask); } @@ -1015,6 +1004,41 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu) if (!cpumask_test_cpu(i, sched_domain_span(sibling))) continue; + /* + * for NUMA diameter >= 3, building sched_domain by sibling's + * child's child domain to prevent sched_group from spanning + * out of sched_domain + * if we don't do this, Diameter >=3 is misrepresented: + * + * Smallest diameter=3 topology is: + * + * node 0 1 2 3 + * 0: 10 20 30 40 + * 1: 20 10 20 30 + * 2: 30 20 10 20 + * 3: 40 30 20 10 + * + * 0 --- 1 --- 2 --- 3 + * + * NUMA-3 0-3 N/A N/A 0-3 + * groups: {0-2},{1-3} {1-3},{0-2} + * + * NUMA-2 0-2 0-3 0-3 1-3 + * groups: {0-1},{1-3} {0-2},{2-3} {1-3},{0-1} {2-3},{0-2} + * + * NUMA-1 0-1 0-2 1-3 2-3 + * groups: {0},{1} {1},{2},{0} {2},{3},{1} {3},{2} + * + * NUMA-0 0 1 2 3 + * + * The NUMA-2 groups for nodes 0 and 3 are obviously buggered, as the + * group span isn't a subset of the domain span. + */ + while (sibling->child && + !cpumask_subset(sched_domain_span(sibling->child), + span)) + sibling = sibling->child; + sg = build_group_from_child_sched_domain(sibling, cpu); if (!sg) goto fail; -- 2.25.1

2 1

[PATCH v2 0/4] perf vendor events arm64: Some tidy-up/refactoring
by John Garry 29 Jan '21

29 Jan '21

There is much event duplication in the common and uarch events for A76 and Ampere eMag support, so factor out into a common JSON. Since the wording for events may differ between CPU datasheet and the architecture reference manual, the current wording is kept. This is unless there are minor differences. In addition, event names are renamed to be same as architecture reference manual, to keep commonality. Also a minor fix is included for the Ampere eMag JSON. Differences to v1: - tidy some ',' misplaced John Garry (4): perf vendor events arm64: Fix Ampere eMag event typo perf vendor events arm64: Add common and uarch event JSON perf vendor events arm64: Reference common and uarch events for Ampere eMag perf vendor events arm64: Reference common and uarch events for A76 .../arch/arm64/ampere/emag/branch.json | 8 +- .../arch/arm64/ampere/emag/bus.json | 5 +- .../arch/arm64/ampere/emag/cache.json | 58 +--- .../arch/arm64/ampere/emag/clock.json | 4 +- .../arch/arm64/ampere/emag/exception.json | 10 +- .../arch/arm64/ampere/emag/instruction.json | 34 +-- .../arch/arm64/ampere/emag/memory.json | 11 +- .../arch/arm64/arm/cortex-a76-n1/branch.json | 12 +- .../arch/arm64/arm/cortex-a76-n1/bus.json | 19 +- .../arch/arm64/arm/cortex-a76-n1/cache.json | 118 +++------ .../arm64/arm/cortex-a76-n1/exception.json | 10 +- .../arm64/arm/cortex-a76-n1/instruction.json | 45 +--- .../arch/arm64/arm/cortex-a76-n1/memory.json | 6 +- .../arch/arm64/arm/cortex-a76-n1/other.json | 4 +- .../arm64/arm/cortex-a76-n1/pipeline.json | 12 +- .../arm64/armv8-common-and-microarch.json | 248 ++++++++++++++++++ 16 files changed, 356 insertions(+), 248 deletions(-) create mode 100644 tools/perf/pmu-events/arch/arm64/armv8-common-and-microarch.json -- 2.26.2

2 5

[PATCH v2 for-next 0/3] RDMA/hns: Refactor codes about memory
by Weihang Li 29 Jan '21

29 Jan '21

This series refactors memory related codes, including MR, MTR(Memory Translate Region) and MPT(Memory Protection Table). Changes since v1: * Add static keyword for hns_roce_mr_free(). * Link: https://patchwork.kernel.org/project/linux-rdma/cover/1611236882-41498-1-gi… Lang Cheng (2): RDMA/hns: Optimize the MR registration process RDMA/hns: Use new interface to set MPT related fields Xi Wang (1): RDMA/hns: Refactor the MTR creation flow drivers/infiniband/hw/hns/hns_roce_common.h | 22 ++ drivers/infiniband/hw/hns/hns_roce_cq.c | 1 - drivers/infiniband/hw/hns/hns_roce_device.h | 5 +- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 77 +++--- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 39 +++ drivers/infiniband/hw/hns/hns_roce_mr.c | 401 +++++++++++----------------- drivers/infiniband/hw/hns/hns_roce_qp.c | 1 - drivers/infiniband/hw/hns/hns_roce_srq.c | 2 - 8 files changed, 257 insertions(+), 291 deletions(-) -- 2.8.1

2 4

[PATCH] iommu: Check dev->iommu in iommu_dev_xxx functions
by Shameer Kolothum 28 Jan '21

28 Jan '21

The device iommu probe/attach might have failed leaving dev->iommu to NULL and device drivers may still invoke these functions resulting a crash in iommu vendor driver code. Hence make sure we check that. Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi(a)huawei.com> --- drivers/iommu/iommu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index ffeebda8d6de..cb68153c5cc0 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2867,7 +2867,7 @@ bool iommu_dev_has_feature(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_has_feat) + if (dev->iommu && ops && ops->dev_has_feat) return ops->dev_has_feat(dev, feat); return false; @@ -2878,7 +2878,7 @@ int iommu_dev_enable_feature(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_enable_feat) + if (dev->iommu && ops && ops->dev_enable_feat) return ops->dev_enable_feat(dev, feat); return -ENODEV; @@ -2894,7 +2894,7 @@ int iommu_dev_disable_feature(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_disable_feat) + if (dev->iommu && ops && ops->dev_disable_feat) return ops->dev_disable_feat(dev, feat); return -EBUSY; @@ -2905,7 +2905,7 @@ bool iommu_dev_feature_enabled(struct device *dev, enum iommu_dev_features feat) { const struct iommu_ops *ops = dev->bus->iommu_ops; - if (ops && ops->dev_feat_enabled) + if (dev->iommu && ops && ops->dev_feat_enabled) return ops->dev_feat_enabled(dev, feat); return false; -- 2.17.1

4 4

[PATCH 0/4] perf vendor events arm64: Some tidy-up/refactoring
by John Garry 28 Jan '21

28 Jan '21

There is much event duplication in the common and uarch events for A76 and Ampere eMag support, so factor out into a common JSON. Since the wording for events may differ between CPU datasheet and the architecture reference manual, the current wording is kept. This is unless there are minor differences. In addition, event names are renamed to be same as architecture reference manual, to keep commonality. Also a minor fix is included for the Ampere eMag JSON. John Garry (4): perf vendor events arm64: Fix Ampere eMag event typo perf vendor events arm64: Add common and uarch event JSON perf vendor events arm64: Reference common and uarch events for Ampere eMag perf vendor events arm64: Reference common and uarch events for A76 .../arch/arm64/ampere/emag/branch.json | 8 +- .../arch/arm64/ampere/emag/bus.json | 5 +- .../arch/arm64/ampere/emag/cache.json | 58 +--- .../arch/arm64/ampere/emag/clock.json | 4 +- .../arch/arm64/ampere/emag/exception.json | 10 +- .../arch/arm64/ampere/emag/instruction.json | 34 +-- .../arch/arm64/ampere/emag/memory.json | 11 +- .../arch/arm64/arm/cortex-a76-n1/branch.json | 12 +- .../arch/arm64/arm/cortex-a76-n1/bus.json | 19 +- .../arch/arm64/arm/cortex-a76-n1/cache.json | 118 +++------ .../arm64/arm/cortex-a76-n1/exception.json | 10 +- .../arm64/arm/cortex-a76-n1/instruction.json | 45 +--- .../arch/arm64/arm/cortex-a76-n1/memory.json | 6 +- .../arch/arm64/arm/cortex-a76-n1/other.json | 4 +- .../arm64/arm/cortex-a76-n1/pipeline.json | 12 +- .../arm64/armv8-common-and-microarch.json | 248 ++++++++++++++++++ 16 files changed, 356 insertions(+), 248 deletions(-) create mode 100644 tools/perf/pmu-events/arch/arm64/armv8-common-and-microarch.json -- 2.26.2

2 7

[PATCH] percpu: set the pointer to NULL after free the memory in free_percpu
by Tian Tao 28 Jan '21

28 Jan '21

set the pointer to NULL after free the memory in free_percpu, so there is no need to set the pointer to NULL in the driver after calling free_percpu again. Signed-off-by: Tian Tao <tiantao6(a)hisilicon.com> --- mm/percpu.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/percpu.c b/mm/percpu.c index f53b89b..af944cc 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -2121,6 +2121,8 @@ void free_percpu(void __percpu *ptr) if (need_balance) pcpu_schedule_balance_work(); + + ptr = NULL; } EXPORT_SYMBOL_GPL(free_percpu); -- 2.7.4

2 1

[PATCH RESEND] dma-mapping: benchmark: fix kernel crash when dma_map_single fails
by Barry Song 28 Jan '21

28 Jan '21

if dma_map_single() fails, kernel will give the below oops since task_struct has been destroyed and we are running into the memory corruption due to use-after-free in kthread_stop(): [ 48.095310] Unable to handle kernel paging request at virtual address 000000c473548040 [ 48.095736] Mem abort info: [ 48.095864] ESR = 0x96000004 [ 48.096025] EC = 0x25: DABT (current EL), IL = 32 bits [ 48.096268] SET = 0, FnV = 0 [ 48.096401] EA = 0, S1PTW = 0 [ 48.096538] Data abort info: [ 48.096659] ISV = 0, ISS = 0x00000004 [ 48.096820] CM = 0, WnR = 0 [ 48.097079] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000104639000 [ 48.098099] [000000c473548040] pgd=0000000000000000, p4d=0000000000000000 [ 48.098832] Internal error: Oops: 96000004 [#1] PREEMPT SMP [ 48.099232] Modules linked in: [ 48.099387] CPU: 0 PID: 2 Comm: kthreadd Tainted: G W [ 48.099887] Hardware name: linux,dummy-virt (DT) [ 48.100078] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--) [ 48.100516] pc : __kmalloc_node+0x214/0x368 [ 48.100944] lr : __kmalloc_node+0x1f4/0x368 [ 48.101458] sp : ffff800011f0bb80 [ 48.101843] x29: ffff800011f0bb80 x28: ffff0000c0098ec0 [ 48.102330] x27: 0000000000000000 x26: 00000000001d4600 [ 48.102648] x25: ffff0000c0098ec0 x24: ffff800011b6a000 [ 48.102988] x23: 00000000ffffffff x22: ffff0000c0098ec0 [ 48.103333] x21: ffff8000101d7a54 x20: 0000000000000dc0 [ 48.103657] x19: ffff0000c0001e00 x18: 0000000000000000 [ 48.104069] x17: 0000000000000000 x16: 0000000000000000 [ 48.105449] x15: 000001aa0304e7b9 x14: 00000000000003b1 [ 48.106401] x13: ffff8000122d5000 x12: ffff80001228d000 [ 48.107296] x11: ffff0000c0154340 x10: 0000000000000000 [ 48.107862] x9 : ffff80000fffffff x8 : ffff0000c473527f [ 48.108326] x7 : ffff800011e62f58 x6 : ffff0000c01c8ed8 [ 48.108778] x5 : ffff0000c0098ec0 x4 : 0000000000000000 [ 48.109223] x3 : 00000000001d4600 x2 : 0000000000000040 [ 48.109656] x1 : 0000000000000001 x0 : ff0000c473548000 [ 48.110104] Call trace: [ 48.110287] __kmalloc_node+0x214/0x368 [ 48.110493] __vmalloc_node_range+0xc4/0x298 [ 48.110805] copy_process+0x2c8/0x15c8 [ 48.111133] kernel_clone+0x5c/0x3c0 [ 48.111373] kernel_thread+0x64/0x90 [ 48.111604] kthreadd+0x158/0x368 [ 48.111810] ret_from_fork+0x10/0x30 [ 48.112336] Code: 17ffffe9 b9402a62 b94008a1 11000421 (f8626802) [ 48.112884] ---[ end trace d4890e21e75419d5 ]--- Signed-off-by: Barry Song <song.bao.hua(a)hisilicon.com> --- kernel/dma/map_benchmark.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/dma/map_benchmark.c b/kernel/dma/map_benchmark.c index b1496e744c68..1b1b8ff875cb 100644 --- a/kernel/dma/map_benchmark.c +++ b/kernel/dma/map_benchmark.c @@ -147,8 +147,10 @@ static int do_map_benchmark(struct map_benchmark_data *map) atomic64_set(&map->sum_sq_unmap, 0); atomic64_set(&map->loops, 0); - for (i = 0; i < threads; i++) + for (i = 0; i < threads; i++) { + get_task_struct(tsk[i]); wake_up_process(tsk[i]); + } msleep_interruptible(map->bparam.seconds * 1000); @@ -183,6 +185,8 @@ static int do_map_benchmark(struct map_benchmark_data *map) } out: + for (i = 0; i < threads; i++) + put_task_struct(tsk[i]); put_device(map->dev); kfree(tsk); return ret; -- 2.25.1

2 1

[PATCH RFC 0/4] Fix arm64 crash for accessing unmapped IO port regions (reboot)
by John Garry 27 Jan '21

27 Jan '21

This is a reboot of my original series to address the problem of drivers for legacy ISA devices accessing unmapped IO port regions on arm64 systems and causing the system to crash. There was another recent report of such an issue [0], and some old ones [1] and [2] for reference. The background is that many systems do not include PCI host controllers, or they do and controller probe may have failed. For these cases, no IO ports are mapped. However, loading drivers for legacy ISA devices can crash the system as there is nothing to stop them accessing those IO ports (which have not been io remap'ed). My original solution tried to keep the kernel alive in these situations by rejecting logical PIO access to PCI IO regions until PCI IO port regions have been mapped. This series goes one step further, by just reserving the complete legacy IO port range in 0x0--0xffff for arm64. The motivation for doing this is to make the request_region() calls for those drivers fail, like this: root@ubuntu:/home/john# insmod mk712.ko [ 3415.575800] mk712: unable to get IO region insmod: ERROR: could not insert module mk712.ko: No such device Otherwise, in theory, those drivers could initiate rogue accesses to mapped IO port regions for other devices and cause corruptions or side-effects. Indeed, those drivers should not be allowed to access IO ports at all in such a system. As a secondary defence, for broken drivers who do not call request_region(), IO port accesses in range 0--0xffff will be ignored, again preserving the system. I am sending as an RFC as I am not sure of any problem with reserving first 0x10000 of IO space like this. There is reserve= commandline argument, which does allow this already. For reference, here's how /proc/ioports looks on my arm64 system with this change: root@ubuntu:/home/john# more /proc/ioports 00010000-0001ffff : PCI Bus 0002:f8 00010000-00010fff : PCI Bus 0002:f9 00010000-00010007 : 0002:f9:00.0 00010000-00010007 : serial 00010008-0001000f : 0002:f9:00.1 00010008-0001000f : serial 00010010-00010017 : 0002:f9:00.2 00010018-0001001f : 0002:f9:00.2 00020000-0002ffff : PCI Bus 0004:88 00030000-0003ffff : PCI Bus 0005:78 00040000-0004ffff : PCI Bus 0006:c0 00050000-0005ffff : PCI Bus 0007:90 00060000-0006ffff : PCI Bus 000a:10 00070000-0007ffff : PCI Bus 000c:20 00080000-0008ffff : PCI Bus 000d:30 [0] https://lore.kernel.org/linux-input/20210112055129.7840-1-song.bao.hua@hisi… [1] https://lore.kernel.org/linux-pci/56F209A9.4040304@huawei.com [2] https://lore.kernel.org/linux-arm-kernel/e6995b4a-184a-d8d4-f4d4-9ce75d8f47… Difference since v4: https://lore.kernel.org/linux-pci/1560262374-67875-1-git-send-email-john.ga… - Reserve legacy ISA region John Garry (4): arm64: io: Introduce IO_SPACE_BASE asm-generic/io.h: Add IO_SPACE_BASE kernel/resource: Make ioport_resource.start configurable logic_pio: Warn on and discard accesses to addresses below IO_SPACE_BASE arch/arm64/include/asm/io.h | 1 + include/asm-generic/io.h | 4 ++++ include/linux/logic_pio.h | 5 +++++ kernel/resource.c | 2 +- lib/logic_pio.c | 20 ++++++++++++++------ 5 files changed, 25 insertions(+), 7 deletions(-) -- 2.26.2

3 8

[PATCH v2 0/2] EDAC/ghes: Add EDAC device for reporting the CPU cache error count
by Shiju Jose 26 Jan '21

26 Jan '21

CPU cache corrected errors are detected occasionally on few of our ARM64 hardware boards. Though it is rare, the probability of the CPU cache errors frequently occurring can't be avoided. The earlier failure detection by monitoring the cache corrected errors for the frequent occurrences and taking preventive action could prevent more serious hardware faults. On Intel architectures, cache corrected errors are reported and the affected cores are offlined in the architecture specific method. http://www.mcelog.org/cache.html However for the firmware-first error reporting, specifically on ARM64 architectures, there is no provision present for reporting the cache corrected error count to the user-space and taking preventive action such as offline the affected cores. For this purpose, it was suggested to create the CPU EDAC device for the CPU caches for reporting the cache error count for the firmware-first error reporting. User-space application could monitor the recorded corrected error count for the earlier hardware failure detection and could take preventive action, such as offline the corresponding CPU core/s. Changes: RFC V1 -> RFC V2: 1. Fixed feedback by Boris. 1.1. Added reason of this patch. 1.2. Changed CPU errors to CPU cache errors in the drivers/edac/Kconfig 1.3 Changed EDAC cache list to percpu variables. 1.4 Changed configuration depends on ARM64. 1.5. Moved discovery of cacheinfo to ghes_scan_system(). 2. Changes in the descriptions. Shiju Jose (2): EDAC/ghes: Add EDAC device for reporting the CPU cache errors ACPI / APEI: Add reporting ARM64 CPU cache corrected error count Documentation/ABI/testing/sysfs-devices-edac | 15 ++ drivers/acpi/apei/ghes.c | 76 +++++++- drivers/edac/Kconfig | 12 ++ drivers/edac/ghes_edac.c | 186 +++++++++++++++++++ include/acpi/ghes.h | 27 +++ include/linux/cper.h | 4 + 6 files changed, 316 insertions(+), 4 deletions(-) -- 2.17.1

1 2