On Thu, Dec 31, 2020 at 09:36:27AM +0000, liweihang wrote:

On 2020/12/31 15:59, Leon Romanovsky wrote:

...

On Tue, Dec 29, 2020 at 01:31:39PM +0000, liweihang wrote:

...

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0

Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?

Thanks

Hi Leon,

Thanks for your response. If we can achieve it in rdmatool, it may looks like:

1. We want to get some information of a ucontext (assuming that each

ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:

$ rdma res show ucontext uctx_id 1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096 uctx_id 2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.

2. We want to know which ucontext a QP belongs to:

$ rdma res show qp link hns_0/1 lqpn 1 type GSI ... uctx_id 1 link hns_0/1 lqpn 2 type RC ... uctx_id 1

So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.

We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.

[leonro@vm ~]$ ibv_rc_pingpong & [leonro@vm ~]$ rdma res show 0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1 [leonro@vm ~]$ rdma res show qp type RC link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong [leonro@vm ~]$ rdma res show pd pdn 3 ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong ^^^^^^

Thanks

Thanks Weihang

On Tue, Jan 05, 2021 at 03:15:58AM +0000, liweihang wrote:

On 2020/12/31 18:39, Leon Romanovsky wrote:

...

On Thu, Dec 31, 2020 at 09:36:27AM +0000, liweihang wrote:

...

On 2020/12/31 15:59, Leon Romanovsky wrote:

...

On Tue, Dec 29, 2020 at 01:31:39PM +0000, liweihang wrote:

...

Hi all,

We want to dump some hns driver-defined information that belongs to a process to keep track of current memory usage. For example, there is a ucontext-level(process-level) memory pool to store WQE which is shared by a lot of QPs, we want to record and query which QPs are using this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp QPN Total(kB) SQ(kB) SGE(kB) RQ(kB) 110 6400 256 2048 4096 118 6400 256 2048 0

Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP attribute? Can you please send "rdmatool" example?

Thanks

Hi Leon,

Thanks for your response. If we can achieve it in rdmatool, it may looks like:

1. We want to get some information of a ucontext (assuming that each

ucontext has an ID), for example, the size of a memory pool that belongs to a ucontext as I mentioned above:

$ rdma res show ucontext uctx_id 1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096 uctx_id 2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

I have no problems to add "rdma res show ucontext" command, we just need to find what should be printed.

...

2. We want to know which ucontext a QP belongs to:

$ rdma res show qp link hns_0/1 lqpn 1 type GSI ... uctx_id 1 link hns_0/1 lqpn 2 type RC ... uctx_id 1

So the question is, we don't have a ucontext-level restrack currently, and there in no 'id' for each ucontext.

We have IDs for every ucontext, it is called "ctxn" and because QP is not connected directly to ucontext, but through PDs, it is visible when you check PDs.

[leonro@vm ~]$ ibv_rc_pingpong & [leonro@vm ~]$ rdma res show 0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1 [leonro@vm ~]$ rdma res show qp type RC link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong [leonro@vm ~]$ rdma res show pd pdn 3 ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong ^^^^^^

Thanks

OK, thank you, we will try to use rdmatool to achieve our goal.

By the way, can we use debugfs to add some trace functions? Debugfs can quickly get a large amount of information at a time, it's more convenient than netlink in such situation, especially for some vendor-defined funtions.

If you are talking about connecting your driver to ftrace, so sure go for it. However for the data, please use rdmatool, we have raw option (-r) in the rdmatool that dumps in binary format your specific object.

Thanks

Thanks Weihang