On 28/05/2021 08:12, chenxiang wrote:
From: Xiang Chen chenxiang66@hisilicon.com
The first patch is to release those rcache when rmmod the driver of the last device to save memory.
And patch2~6 is add support for IOMMU debugfs related to IOVA as follows: /sys/kernel/debug/iommu/iovad/iommu_domainx
Under iommu_domainx dir, add debugfs file iova_rcache and drop_rcache.
From debugfs file iova_rcache, we can get how many cpu_rcache / share
rcache / iovas are used, and also we can drop those rcache by debugfs file drop_rcache:
For cpu_rcache, [i]=x|y indicates that there are x iova in load iova_magazine and y iova in prev iova_magazine (128 at most). For share rcache, [i]=x indicates that there are x iova_magazines in use.
estuary:/sys/kernel/debug/iommu/iovad/iommu_domain2$
How do we know the relation to the IOMMU group?
And it could be nice to show 'ls -l' output, even if you did mention it, above.
I would also like to see additional info, specifically allocation attempts per rcache range and also allocation attempts which were "too_big" to be cached.
cat iova_rcache [ 272.814457] cpu0 [0]=60|0 [1]=7|0 [2]=32|0 [3]=0|0 [4]=97|0 [5]=104|0
I suppose this is ok, but the output is becoming huge with many CPUs and possibly increasing rcache range.
You possibly could consider breaking it down to sub files or folders, which may be better, like:
ls iovad/iommu_domain2: 0 1 2 3 4 ... too_big
ls iovad/iommu_domain2/0 rcache allocations depot
more iovad/iommu_domain2/0/rcache cpu0: 0|1 cpu1: 2|4 ... cpuMax: 0|0
more iovad/iommu_domain2/0/allocations 1244
more iovad/iommu_domain2/0/depot 28 44 22
ls iovad/iommu_domain2/too_big allocations
As for the rcache file, you could even have separate per-cpu files in a rcache folder, like:
ls iovad/iommu_domain2/0/rcache cpu0 cpu1 cpu2 ...cpu127
more iovad/iommu_domain2/0/rcache/cpu0 0|3
But then we need have more files and folders to examine. Need to find a good balance.
[ 272.820982] cpu1 [0]=0|0 [1]=3|0 [2]=14|0 [3]=76|0 [4]=15|0 [5]=64|0 [ 272.827399] cpu2 [0]=85|128 [1]=84|128 [2]=83|128 [3]=112|128 [4]=22|128 [5]=116|128 [ 272.835197] cpu3 [0]=0|0 [1]=91|0 [2]=101|0 [3]=29|0 [4]=0|0 [5]=36|0 [ 272.841699] cpu4 [0]=0|0 [1]=39|0 [2]=113|0 [3]=75|0 [4]=95|0 [5]=0|0 [ 272.848201] cpu5 [0]=27|0 [1]=48|0 [2]=82|0 [3]=0|0 [4]=19|0 [5]=36|0 [ 272.854702] cpu6 [0]=30|0 [1]=0|0 [2]=48|0 [3]=2|0 [4]=18|0 [5]=56|0 [ 272.861117] cpu7 [0]=27|0 [1]=89|0 [2]=101|0 [3]=59|0 [4]=28|0 [5]=0|0 [ 272.867706] cpu8 [0]=66|0 [1]=114|0 [2]=42|0 [3]=123|0 [4]=96|0 [5]=68|0 [ 272.874466] cpu9 [0]=71|0 [1]=61|0 [2]=28|0 [3]=118|0 [4]=116|0 [5]=41|0 [ 272.881227] cpu10 [0]=83|128 [1]=63|128 [2]=109|128 [3]=79|128 [4]=7|128 [5]=54|128 [ 272.888938] cpu11 [0]=90|0 [1]=34|0 [2]=88|0 [3]=58|0 [4]=20|0 [5]=35|0 [ 272.895611] cpu12 [0]=64|0 [1]=20|0 [2]=18|0 [3]=33|0 [4]=42|0 [5]=22|0 [ 272.902285] cpu13 [0]=17|0 [1]=70|0 [2]=115|0 [3]=59|0 [4]=108|0 [5]=58|0 [ 272.909132] cpu14 [0]=28|0 [1]=18|0 [2]=27|0 [3]=105|0 [4]=65|0 [5]=81|0 [ 272.915892] cpu15 [0]=75|0 [1]=3|0 [2]=73|0 [3]=104|0 [4]=127|0 [5]=102|0 [ 272.922738] cpu16 [0]=54|0 [1]=116|0 [2]=90|0 [3]=31|0 [4]=108|0 [5]=41|0 [ 272.929590] cpu17 [0]=47|0 [1]=82|0 [2]=3|0 [3]=66|0 [4]=68|0 [5]=66|0 [ 272.936179] cpu18 [0]=126|128 [1]=110|128 [2]=48|128 [3]=118|128 [4]=54|128 [5]=73|128 [ 272.944156] cpu19 [0]=31|0 [1]=13|0 [2]=104|0 [3]=45|0 [4]=108|0 [5]=96|0 [ 272.951006] cpu20 [0]=58|0 [1]=113|0 [2]=14|0 [3]=123|0 [4]=52|0 [5]=54|0 [ 272.957856] cpu21 [0]=116|0 [1]=47|0 [2]=96|0 [3]=60|0 [4]=47|0 [5]=126|0 [ 272.964701] cpu22 [0]=84|0 [1]=87|0 [2]=88|0 [3]=68|0 [4]=37|0 [5]=119|0 [ 272.971462] cpu23 [0]=13|0 [1]=63|0 [2]=124|0 [3]=3|0 [4]=7|0 [5]=38|0 [ 272.978051] cpu24 [0]=15|0 [1]=64|0 [2]=65|0 [3]=53|0 [4]=102|0 [5]=69|0 [ 272.984812] cpu25 [0]=94|0 [1]=108|0 [2]=67|0 [3]=125|0 [4]=107|0 [5]=8|0 [ 272.991663] cpu26 [0]=84|128 [1]=86|128 [2]=91|128 [3]=121|128 [4]=77|128 [5]=25|128 [ 272.999464] cpu27 [0]=60|0 [1]=105|0 [2]=61|0 [3]=91|0 [4]=79|0 [5]=6|0 [ 273.006141] cpu28 [0]=39|0 [1]=91|0 [2]=11|0 [3]=87|0 [4]=112|0 [5]=10|0 [ 273.012904] cpu29 [0]=88|0 [1]=43|0 [2]=0|0 [3]=77|0 [4]=10|0 [5]=79|0 [ 273.019492] cpu30 [0]=65|0 [1]=24|0 [2]=125|0 [3]=24|0 [4]=54|0 [5]=21|0 [ 273.026254] cpu31 [0]=26|0 [1]=90|0 [2]=42|0 [3]=17|0 [4]=73|0 [5]=35|0 [ 273.032929] cpu32 [0]=1|0 [1]=83|0 [2]=76|0 [3]=62|0 [4]=117|0 [5]=96|0 [ 273.039612] cpu33 [0]=50|0 [1]=55|0 [2]=63|0 [3]=79|0 [4]=86|0 [5]=15|0 [ 273.046293] cpu34 [0]=122|128 [1]=36|128 [2]=36|128 [3]=79|128 [4]=113|128 [5]=80|128 [ 273.054179] cpu35 [0]=101|0 [1]=18|0 [2]=7|0 [3]=10|0 [4]=7|0 [5]=112|0 [ 273.060854] cpu36 [0]=12|0 [1]=107|0 [2]=43|0 [3]=60|0 [4]=19|0 [5]=110|0 [ 273.067703] cpu37 [0]=90|0 [1]=34|0 [2]=66|0 [3]=91|0 [4]=85|0 [5]=31|0 [ 273.074378] cpu38 [0]=0|0 [1]=22|0 [2]=18|0 [3]=73|0 [4]=54|0 [5]=96|0 [ 273.080968] cpu39 [0]=109|0 [1]=54|0 [2]=124|0 [3]=21|0 [4]=88|0 [5]=61|0 [ 273.087816] cpu40 [0]=95|0 [1]=50|0 [2]=45|0 [3]=66|0 [4]=30|0 [5]=84|0 [ 273.094490] cpu41 [0]=99|0 [1]=47|0 [2]=8|0 [3]=81|0 [4]=0|0 [5]=95|0 [ 273.100992] cpu42 [0]=25|128 [1]=92|128 [2]=53|128 [3]=49|128 [4]=43|128 [5]=78|128 [ 273.108704] cpu43 [0]=88|0 [1]=42|0 [2]=10|0 [3]=124|0 [4]=4|0 [5]=105|0 [ 273.115464] cpu44 [0]=80|0 [1]=63|0 [2]=1|0 [3]=123|0 [4]=35|0 [5]=17|0 [ 273.122139] cpu45 [0]=31|0 [1]=92|0 [2]=8|0 [3]=60|0 [4]=74|0 [5]=92|0 [ 273.128727] cpu46 [0]=78|0 [1]=40|0 [2]=95|0 [3]=33|0 [4]=67|0 [5]=63|0 [ 273.135401] cpu47 [0]=112|0 [1]=93|0 [2]=96|0 [3]=24|0 [4]=93|0 [5]=15|0 [ 273.142162] cpu48 [0]=92|0 [1]=120|0 [2]=49|0 [3]=118|0 [4]=1|0 [5]=83|0 [ 273.148923] cpu49 [0]=101|0 [1]=7|0 [2]=108|0 [3]=15|0 [4]=69|0 [5]=116|0 [ 273.155771] cpu50 [0]=111|128 [1]=98|128 [2]=21|128 [3]=27|128 [4]=109|128 [5]=21|128 [ 273.163655] cpu51 [0]=31|0 [1]=33|0 [2]=82|0 [3]=117|0 [4]=98|0 [5]=1|0 [ 273.170329] cpu52 [0]=113|0 [1]=64|0 [2]=16|0 [3]=48|0 [4]=97|0 [5]=80|0 [ 273.177090] cpu53 [0]=95|0 [1]=39|0 [2]=26|0 [3]=107|0 [4]=2|0 [5]=18|0 [ 273.183764] cpu54 [0]=114|0 [1]=94|0 [2]=110|0 [3]=85|0 [4]=66|0 [5]=45|0 [ 273.190611] cpu55 [0]=52|0 [1]=89|0 [2]=43|0 [3]=117|0 [4]=115|0 [5]=91|0 [ 273.197460] cpu56 [0]=0|0 [1]=51|0 [2]=81|0 [3]=60|0 [4]=20|0 [5]=27|0 [ 273.204048] cpu57 [0]=124|0 [1]=121|0 [2]=56|0 [3]=0|0 [4]=77|0 [5]=59|0 [ 273.210807] cpu58 [0]=109|128 [1]=98|128 [2]=6|128 [3]=39|128 [4]=64|128 [5]=24|128 [ 273.218518] cpu59 [0]=57|0 [1]=62|0 [2]=66|0 [3]=55|0 [4]=95|0 [5]=47|0 [ 273.225192] cpu60 [0]=29|0 [1]=12|0 [2]=112|0 [3]=23|0 [4]=65|0 [5]=34|0 [ 273.231955] cpu61 [0]=92|0 [1]=5|0 [2]=19|0 [3]=91|0 [4]=101|0 [5]=97|0 [ 273.238629] cpu62 [0]=97|0 [1]=42|0 [2]=30|0 [3]=111|0 [4]=99|0 [5]=2|0 [ 273.245304] cpu63 [0]=77|0 [1]=79|0 [2]=62|0 [3]=56|0 [4]=17|0 [5]=76|0 [ 273.251982] cpu64 [0]=0|0 [1]=108|0 [2]=87|0 [3]=58|0 [4]=26|0 [5]=0|0 [ 273.258582] cpu65 [0]=0|0 [1]=114|0 [2]=99|0 [3]=73|0 [4]=33|0 [5]=0|0 [ 273.265174] cpu66 [0]=39|128 [1]=52|128 [2]=82|128 [3]=107|128 [4]=73|128 [5]=13|128 [ 273.272974] cpu67 [0]=0|0 [1]=123|0 [2]=87|0 [3]=49|0 [4]=27|0 [5]=98|0 [ 273.279651] cpu68 [0]=108|0 [1]=0|0 [2]=123|0 [3]=47|0 [4]=30|0 [5]=109|0 [ 273.286500] cpu69 [0]=92|0 [1]=110|0 [2]=92|0 [3]=59|0 [4]=42|0 [5]=104|0 [ 273.293351] cpu70 [0]=105|0 [1]=106|0 [2]=99|0 [3]=57|0 [4]=15|0 [5]=111|0 [ 273.300287] cpu71 [0]=106|0 [1]=108|0 [2]=108|0 [3]=58|0 [4]=35|0 [5]=98|0 [ 273.307222] cpu72 [0]=0|0 [1]=0|0 [2]=114|0 [3]=118|0 [4]=95|0 [5]=0|0 [ 273.313813] cpu73 [0]=0|0 [1]=0|0 [2]=0|0 [3]=98|0 [4]=95|0 [5]=124|0 [ 273.320318] cpu74 [0]=65|0 [1]=76|0 [2]=109|0 [3]=58|128 [4]=115|128 [5]=105|0 [ 273.327603] cpu75 [0]=0|0 [1]=0|0 [2]=123|0 [3]=102|0 [4]=109|0 [5]=0|0 [ 273.334279] cpu76 [0]=125|0 [1]=0|0 [2]=0|0 [3]=101|0 [4]=112|0 [5]=0|0 [ 273.340954] cpu77 [0]=124|0 [1]=124|0 [2]=0|0 [3]=103|0 [4]=100|0 [5]=0|0 [ 273.347803] cpu78 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=118|0 [5]=0|0 [ 273.354134] cpu79 [0]=124|0 [1]=0|0 [2]=116|0 [3]=104|0 [4]=120|0 [5]=0|0 [ 273.360981] cpu80 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.367138] cpu81 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.373292] cpu82 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.379447] cpu83 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.385601] cpu84 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.391755] cpu85 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.397910] cpu86 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.404065] cpu87 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.410220] cpu88 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.416376] cpu89 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.422532] cpu90 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.428687] cpu91 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.434841] cpu92 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.440997] cpu93 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.447152] cpu94 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.453311] cpu95 [0]=0|0 [1]=0|0 [2]=0|0 [3]=0|0 [4]=0|0 [5]=0|0 [ 273.459468] cpu96 [0]=78|0 [1]=95|0 [2]=117|0 [3]=120|0 [4]=37|0 [5]=31|0 [ 273.466317] cpu97 [0]=105|0 [1]=99|0 [2]=37|0 [3]=24|0 [4]=86|0 [5]=39|0 [ 273.473080] cpu98 [0]=37|128 [1]=13|128 [2]=102|128 [3]=124|128 [4]=60|128 [5]=36|128 [ 273.480967] cpu99 [0]=69|0 [1]=12|0 [2]=84|0 [3]=49|0 [4]=108|0 [5]=127|0 [ 273.487818] cpu100 [0]=92|0 [1]=104|0 [2]=22|0 [3]=31|0 [4]=2|0 [5]=72|0 [ 273.494581] cpu101 [0]=0|0 [1]=31|0 [2]=44|0 [3]=8|0 [4]=13|0 [5]=50|0 [ 273.501170] cpu102 [0]=79|0 [1]=14|0 [2]=88|0 [3]=53|0 [4]=7|0 [5]=44|0 [ 273.507846] cpu103 [0]=10|0 [1]=117|0 [2]=53|0 [3]=112|0 [4]=11|0 [5]=71|0 [ 273.514781] cpu104 [0]=123|0 [1]=30|0 [2]=9|0 [3]=2|0 [4]=30|0 [5]=24|0 [ 273.521462] cpu105 [0]=116|0 [1]=44|0 [2]=29|0 [3]=104|0 [4]=71|0 [5]=3|0 [ 273.528309] cpu106 [0]=38|128 [1]=127|128 [2]=21|128 [3]=83|128 [4]=59|128 [5]=81|128 [ 273.536195] cpu107 [0]=127|0 [1]=126|0 [2]=125|0 [3]=79|0 [4]=12|0 [5]=115|0 [ 273.543303] cpu108 [0]=0|0 [1]=43|0 [2]=111|0 [3]=10|0 [4]=51|0 [5]=68|0 [ 273.550065] cpu109 [0]=37|0 [1]=63|0 [2]=11|0 [3]=10|0 [4]=58|0 [5]=9|0 [ 273.556742] cpu110 [0]=86|0 [1]=44|0 [2]=83|0 [3]=85|0 [4]=82|0 [5]=119|0 [ 273.563590] cpu111 [0]=114|0 [1]=27|0 [2]=111|0 [3]=11|0 [4]=45|0 [5]=56|0 [ 273.570524] cpu112 [0]=74|0 [1]=86|0 [2]=93|0 [3]=122|0 [4]=126|0 [5]=31|0 [ 273.577458] cpu113 [0]=51|0 [1]=81|0 [2]=7|0 [3]=124|0 [4]=71|0 [5]=5|0 [ 273.584133] cpu114 [0]=58|128 [1]=95|128 [2]=14|128 [3]=119|128 [4]=85|128 [5]=23|128 [ 273.592018] cpu115 [0]=59|0 [1]=66|0 [2]=1|0 [3]=50|0 [4]=77|0 [5]=6|0 [ 273.598607] cpu116 [0]=65|0 [1]=81|0 [2]=22|0 [3]=18|0 [4]=96|0 [5]=2|0 [ 273.605282] cpu117 [0]=23|0 [1]=41|0 [2]=19|0 [3]=69|0 [4]=108|0 [5]=126|0 [ 273.612216] cpu118 [0]=74|0 [1]=40|0 [2]=0|0 [3]=100|0 [4]=109|0 [5]=7|0 [ 273.618979] cpu119 [0]=47|0 [1]=31|0 [2]=103|0 [3]=65|0 [4]=84|0 [5]=124|0 [ 273.625914] cpu120 [0]=74|0 [1]=18|0 [2]=87|0 [3]=117|0 [4]=17|0 [5]=62|0 [ 273.632762] cpu121 [0]=64|0 [1]=73|0 [2]=3|0 [3]=127|0 [4]=120|0 [5]=105|0 [ 273.639698] cpu122 [0]=46|128 [1]=89|128 [2]=22|128 [3]=78|128 [4]=100|128 [5]=106|128 [ 273.647671] cpu123 [0]=36|0 [1]=71|0 [2]=25|0 [3]=23|0 [4]=126|0 [5]=86|0 [ 273.654521] cpu124 [0]=26|0 [1]=93|0 [2]=101|0 [3]=103|0 [4]=54|0 [5]=100|0 [ 273.661542] cpu125 [0]=58|0 [1]=57|0 [2]=116|0 [3]=35|0 [4]=73|0 [5]=12|0 [ 273.668392] cpu126 [0]=11|0 [1]=54|0 [2]=41|0 [3]=0|0 [4]=27|0 [5]=54|0 [ 273.675067] cpu127 [0]=70|0 [1]=38|0 [2]=58|0 [3]=123|0 [4]=47|0 [5]=21|0 [ 273.681913] share cache: [0]=0 [1]=4 [2]=9 [3]=3 [4]=6 [5]=5
let's call this the depot
[ 273.687633] rb_total: 59443