Since servers have more and more CPUs, and we're running more containers on them, we've been using will-it-scale to test how well ext4 scales. The fallocate2 test (append 8KB to 1MB, truncate to 0, repeat) run concurrently on 64 containers revealed significant contention in block allocation/free, leading to much lower aggregate fallocate OPS compared to a single container (see below). 1 | 2 | 4 | 8 | 16 | 32 | 64 -------|--------|--------|--------|--------|--------|------- 295287 | 70665 | 33865 | 19387 | 10104 | 5588 | 3588 The main bottleneck was the ext4_lock_group(), which both block allocation and free fought over. While the block group for block free is fixed and unoptimizable, the block group for allocation is selectable. Consequently, the ext4_try_lock_group() helper function was added to avoid contention on busy groups, and you can see more in Patch 1. After we fixed the ext4_lock_group bottleneck, another one showed up: s_md_lock. This lock protects different data when allocating and freeing blocks. We got rid of the s_md_lock call in block allocation by making stream allocation work per inode instead of globally. You can find more details in Patch 2. Performance test data follows: CPU: HUAWEI Kunpeng 920 Memory: 480GB Disk: 480GB SSD SATA 3.2 Test: Running will-it-scale/fallocate2 on 64 CPU-bound containers. Observation: Average fallocate operations per container per second. |--------|--------|--------|--------|--------|--------|--------|--------| | - | 1 | 2 | 4 | 8 | 16 | 32 | 64 | |--------|--------|--------|--------|--------|--------|--------|--------| | base | 295287 | 70665 | 33865 | 19387 | 10104 | 5588 | 3588 | |--------|--------|--------|--------|--------|--------|--------|--------| | linear | 286328 | 123102 | 119542 | 90653 | 60344 | 35302 | 23280 | | | -3.0% | 74.20% | 252.9% | 367.5% | 497.2% | 531.6% | 548.7% | |--------|--------|--------|--------|--------|--------|--------|--------| |mb_optim| 292498 | 133305 | 103069 | 61727 | 29702 | 16845 | 10430 | |ize_scan| -0.9% | 88.64% | 204.3% | 218.3% | 193.9% | 201.4% | 190.6% | |--------|--------|--------|--------|--------|--------|--------|--------| Baokun Li (2): ext4: add ext4_try_lock_group() to skip busy groups ext4: move mb_last_[group|start] to ext4_inode_info fs/ext4/ext4.h | 30 ++++++++++++++++++------------ fs/ext4/mballoc.c | 34 ++++++++++++++++++++-------------- fs/ext4/super.c | 2 ++ 3 files changed, 40 insertions(+), 26 deletions(-) -- 2.46.1