Hi Robin,
在 2021/6/7 19:23, Robin Murphy 写道:
On 2021-06-07 03:42, chenxiang wrote:
From: Xiang Chen chenxiang66@hisilicon.com
When rmmod the driver of the last device in the group, cached iovas are not used, and it is better to free them to save memories. And also export function free_rcache_cached_iovas() and iommu_domain_to_iova().
How common is it to use a device a significant amount, then unbind its driver and not use it for long enough to care about? Off-hand I can't think of a particularly realistic use-case which looks like that - the closest I can imagine is unbinding a default kernel driver to replace it with VFIO, but I would expect the set of devices intended for assignment to be distinct from the set of devices used by the host itself, and thus the host driver wouldn't have actually done much to generate a lot of DMA mappings in that initial period. Is my expectation there wrong?
It is possible that user uses the driver for a while and then rmmod it (for example, SAS card is the driver we use always, but sometimes we need to compare the performance with raid card, so we will insmod the raid driver, and rmmod the driver after the test). At this situation, the rcache iovas of raid card are always still even if rmmod the driver (user always rmmod the driver rather than removing the device which will release the group and release all resources).
If there is such a case, how much memory does this actually save in practice? The theoretical worst-case should be roughly 4 * 128 * 6 * sizeof(struct iova) bytes per CPU, which is around 192KB assuming 64-byte kmem cache alignment.
There are 128 CPUs in our ARM64 kunpeng920 board, and i add a debugfs interface drop_rcache of every group in local code which calls function free_rcache_cached_iovas(), and we can see that it saves 1~2M memory after freeing cached iovas.
estuary:/$ free total used free shared buffers cached Mem: 526776720 1336216 525440504 177664 0 177664 -/+ buffers/cache: 1158552 525618168 Swap: 0 0 0 estuary:/sys/kernel/debug/iommu/iovad/iommu_domain29$ echo 1 > drop_rcache estuary:/sys/kernel/debug/iommu/iovad/iommu_domain29$ free total used free shared buffers cached Mem: 526776720 1334672 525442048 177664 0 177664 -/+ buffers/cache: 1157008 525619712 Swap: 0 0 0
The other reason i want to free rcache iovas in suitable place is the IOVA longterm issue[0] (which is unrecoverable), if freeing cached iovas when rmmod the driver or adding a debugfs to do that, i think we can recover it by rmmod the driver and then insmod it or calling the debugfs drop_rcache though it doesn't solve the issue.
[0] https://lore.kernel.org/linux-iommu/1607538189-237944-4-git-send-email-john....
However it seems rather unlikely in practice to have every possible cache entry of every size used, so if saving smaller amounts of memory is a concern wouldn't you also want to explicitly drain the flush queues (16KB per CPU) and maybe look at trying to reclaim the unused pagetable pages from the domain itself - that ~192KB worth of cached IOVAs represents ~32K pages worth of IOVA space, which for an implementation like io-pgtable-arm with the 4KB granule means ~256KB worth of non-leaf pagetables left behind.
Ok, we may consider about those.
I'm not against the idea of having a mechanism to "compact" an idle DMA domain if there are convincing numbers to back it up, but the actual implementation would need to be better than this as well - having the IOMMU core code poking directly into the internals of the iommu-dma abstraction is not the way to go, and exporting anything to modules, which the IOMU core is not, smells suspicious.
Robin.
Xiang Chen (4): iommu/iova: add a function to free all rcached iovas and export it iommu/iova: use function free_rcache_cached_iovas() to free all rcached iovas dma-iommu: add a interface to get iova_domain from iommu domain iommu: free cached iovas when rmmod the driver of the last device in the group
drivers/iommu/dma-iommu.c | 7 +++++++ drivers/iommu/iommu.c | 7 +++++++ drivers/iommu/iova.c | 17 ++++++++++++----- include/linux/dma-iommu.h | 6 ++++++ include/linux/iova.h | 5 +++++ 5 files changed, 37 insertions(+), 5 deletions(-)
Linuxarm mailing list -- linuxarm@openeuler.org To unsubscribe send an email to linuxarm-leave@openeuler.org