On 25/01/21 03:13, Song Bao Hua (Barry Song) wrote:
As long as NUMA diameter > 2, building sched_domain by sibling's child domain will definitely create a sched_domain with sched_group which will span out of the sched_domain +------+ +------+ +-------+ +------+ | node | 12 |node | 20 | node | 12 |node | | 0 +---------+1 +--------+ 2 +-------+3 | +------+ +------+ +-------+ +------+
domain0 node0 node1 node2 node3
domain1 node0+1 node0+1 node2+3 node2+3 + domain2 node0+1+2 | group: node0+1 | group:node2+3 <-------------------+
when node2 is added into the domain2 of node0, kernel is using the child domain of node2's domain2, which is domain1(node2+3). Node 3 is outside the span of node0+1+2.
Will we move to use the *child* domain of the *child* domain of node2's domain2 to build the sched_group?
I mean: +------+ +------+ +-------+ +------+ | node | 12 |node | 20 | node | 12 |node | | 0 +---------+1 +--------+ 2 +-------+3 | +------+ +------+ +-------+ +------+
domain0 node0 node1 +- node2 node3 | domain1 node0+1 node0+1 | node2+3 node2+3 | domain2 node0+1+2 | group: node0+1 | group:node2 <-------------------+
In this way, it seems we don't have to create a new group as we are just reusing the existing group?
One thing I've been musing over is pretty much this; that is to say we would make all non-local NUMA sched_groups span a single node. This would let us reuse an existing span+sched_group_capacity: the local group of that node at its first NUMA topology level.
Essentially this means getting rid of the overlapping groups, and the balance mask is handled the same way as for !NUMA, i.e. it's the local group span. I've not gone far enough through the thought experiment to see where does it miserably fall apart... It is at the very least violating the expectation that a group span is a child domain's span - here it can be a grand^x children domain's span.
If we take your topology, we currently have:
| tl\node | 0 | 1 | 2 | 3 | |---------+--------------+---------------+---------------+--------------| | NUMA0 | (0)->(1) | (1)->(2)->(0) | (2)->(3)->(1) | (3)->(2) | | NUMA1 | (0-1)->(1-3) | (0-2)->(2-3) | (1-3)->(0-1) | (2-3)->(0-2) | | NUMA2 | (0-2)->(1-3) | N/A | N/A | (1-3)->(0-2) |
With the current overlapping group scheme, we would need to make it look like so:
| tl\node | 0 | 1 | 2 | 3 | |---------+---------------+---------------+---------------+---------------| | NUMA0 | (0)->(1) | (1)->(2)->(0) | (2)->(3)->(1) | (3)->(2) | | NUMA1 | (0-1)->(1-2)* | (0-2)->(2-3) | (1-3)->(0-1) | (2-3)->(1-2)* | | NUMA2 | (0-2)->(1-3) | N/A | N/A | (1-3)->(0-2) |
But as already discussed, that's tricky to make work. With the node-span groups thing, we would turn this into:
| tl\node | 0 | 1 | 2 | 3 | |---------+------------+---------------+---------------+------------| | NUMA0 | (0)->(1) | (1)->(2)->(0) | (2)->(3)->(1) | (3)->(2) | | NUMA1 | (0-1)->(2) | (0-2)->(3) | (1-3)->(0) | (2-3)->(1) | | NUMA2 | (0-2)->(3) | N/A | N/A | (1-3)->(0) |