This series adds net/dynthread, a loadable module that adapts the NAPI processing and the RX queue IRQ placement of a set of net devices to the scheduling behaviour of a target application at runtime. Motivation ---------- Interrupt and softirq processing of network RX competes with latency sensitive applications (e.g. mysqld) for CPU time. When such an application runs for a long time on a CPU without blocking, the softirq work of the NIC completion queues can preempt it on the same CPU and inflate its tail latency. The usual remedies are to steer the RX queue IRQs to other CPUs and/or to move NAPI processing into kernel threads bound to spare CPUs, but a static configuration cannot follow the application: once the target goes idle, is scheduled in short bursts, or is displaced, the reserved CPUs are simply wasted. Approach -------- The module periodically samples the scheduling statistics of all threads matching the appname module parameter and classifies them into one of five statuses (DATA_LOSS, NO_TASK, NOT_RUNNING, FRAGMENTATION, LONG_RUNNING_WITH_INTERRUPTS). Acting on that status it: - restores the RX queue IRQs to their original affinity when the target is gone or barely running; - spreads the IRQs over the NUMA node of their queues when processing in softirq context is acceptable again; - switches the NAPI processing of the configured net devices to kernel threads and binds the NAPI threads (and, optionally, the queue IRQs) to the CPUs of their NUMA node that the target tasks are not allowed to use; - scales the number of CPUs used by those threads with their measured load while the target stays in the long-running status. An unusable sample (counters moved backwards: DATA_LOSS) keeps the previous status, so a single bad sample cannot toggle the mode. Everything is controlled through module parameters, and the original IRQ affinity and NAPI mode are restored when the target disappears, the scheduler is disabled or the module is unloaded. The module is buildable only as a loadable module; the Kconfig entry currently allows ARM64. Patch breakdown --------------- 1/6 net: dynthread: Add dynamic threaded NAPI scheduler framework Wire the new directory into net/Kconfig and net/Makefile, add the NET_DYNTHREAD Kconfig entry and the module skeleton. On load the module takes a reference on each device given with the '#' separated ifname parameter, walks the IRQ descriptor table through the exported irq_get_irq_data(), snapshots desc->action and the effective affinity under desc->lock, and records the IRQ number and the original affinity CPU of every RX queue. IRQs are matched against the device name, its parent device name and the irqname pattern. Also count the NUMA nodes the recorded queues span, drop the device references on unload, and unwind a partially failed init. 2/6 net: dynthread: Monitor the load of the target tasks Add the sampling loop and the status judgement. Every monitored thread whose name matches appname contributes its runtime, voluntary/involuntary context switches and its allowed CPU mask; the masks are unioned to know where the target may run. The sample is compared with the previous one to derive the average runtime per switch, the nvcsw percentage and the CPU load, and to classify it as DATA_LOSS, NO_TASK, NOT_RUNNING, FRAGMENTATION or LONG_RUNNING_WITH_INTERRUPTS. Add the periodic hrtimer/work worker, the dynthreaded enable/disable parameter, a mutex serializing the worker control, a zeroed baseline on every enable, and the last_status tracking used to carry the transition sample. 3/6 net: dynthread: Steer the RX queue IRQs away from the target tasks Manage the affinity of the recorded RX queue IRQs. When the target tasks sit in a single NUMA node and enough CPUs of that node are free, move all queue IRQs there round-robin; when the target is gone or barely running, restore the affinity recorded at load time; when the target is fragmented, spread the IRQs over a subset of their own NUMA node sized from dyn_threaded_irq_cpucore_per. Track the "bound to one node" state so spreading is suppressed while it is active, and clear it on every recovery. All of this is skipped when dyn_threaded_bindirq is off, and the IRQ affinity is restored when the worker is stopped. 4/6 net: dynthread: Switch NAPI to threads on spare CPUs Enable threaded NAPI (dev_set_threaded) on the monitored devices in LONG_RUNNING_WITH_INTERRUPTS and bind each NAPI thread from the NUMA node of its queue IRQ: use the whole node if the target is not there, the target-free CPUs of the node if any, otherwise the first dyn_threaded_corenum CPUs of the node. Optionally bind the queue IRQ to the same mask. Record the CPUs used by the NAPI threads in napi_cpumask for the load evaluation of the next patch. Switch the devices back to softirq mode in NO_TASK, NOT_RUNNING and FRAGMENTATION, when the scheduler is disabled and on module unload; the NAPI and IRQ operations run under the rtnetlink lock. 5/6 net: dynthread: Scale the number of NAPI thread CPUs Sample the kernel_cpustat of the CPUs used by the NAPI threads and compare consecutive periods. Grow dyn_threaded_corenum when the irq/softirq share or the total load of those CPUs is too high, and shrink it when the user share is high or the total load is low, within the lower/upper limit parameters. Damping and anti-fight rules avoid oscillating right after a change or undoing a change made for the opposite reason. The default core count is computed at init from the busiest device's RX queue count divided by the online NUMA nodes and clamped to the limits. Reset the damping state when leaving the long-running status. 6/6 net: dynthread: Add debug switch and instrumentation Add the dyn_threaded_debug module parameter and instrument the periodic sampling, the status transitions, the IRQ affinity decisions, the NAPI thread masks and the CPU-count resizes. The messages are recorded with trace_printk() so they can be read from tracefs without flooding the console; without tracing support they fall back to dynamic debug. The switch is off by default. Amritha Nambiar (2): net: Add queue and napi association net: Add NAPI IRQ support Cai Xinchen (6): net: dynthread: Add dynamic threaded NAPI scheduler framework net: dynthread: Monitor the load of the target tasks net: dynthread: Steer the RX queue IRQs away from the target tasks net: dynthread: Switch NAPI to threads on spare CPUs net: dynthread: Scale the number of NAPI thread CPUs net: dynthread: Add debug switch and instrumentation Carolina Jubran (1): net/mlx5e: Acquire RTNL lock before RQs/SQs activation/deactivation Cosmin Ratiu (1): net/mlx5e: Fix lock order in mlx5e_tx_reporter_ptpsq_unhealthy_recover Joe Damato (1): net/mlx5e: link NAPI instances to queues and IRQs .../net/ethernet/mellanox/mlx5/core/en/ptp.c | 5 +- .../mellanox/mlx5/core/en/reporter_tx.c | 5 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 7 + include/linux/netdevice.h | 21 +- include/net/netdev_rx_queue.h | 2 +- net/Kconfig | 1 + net/Makefile | 1 + net/core/dev.c | 38 + net/dynthread/Kconfig | 13 + net/dynthread/Makefile | 2 + net/dynthread/net_dynthread.c | 1316 +++++++++++++++++ 11 files changed, 1407 insertions(+), 4 deletions(-) create mode 100644 net/dynthread/Kconfig create mode 100644 net/dynthread/Makefile create mode 100644 net/dynthread/net_dynthread.c -- 2.18.0.huawei.25