On 2021/2/10 3:54, Jason Gunthorpe wrote:
On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote:
The HIP09 introduces the DCA(Dynamic Context Attachment) feature which supports many RC QPs to share the WQE buffer in a memory pool. If a QP enables DCA feature, the WQE's buffer will not be allocated when creating but when the users start to post WRs. This will reduce the memory consumption when there are too many QPs are inactive. One a WQE buffer is allocated it still acts as a normal WQE ring
buffer? So this DCA logic is to remap the send queue buffer based on demand for SQEs? How does it interact with the normal max send queue entries reported?
Not exactly. If DCA is enabled, we first allocate a memory pool with a default size when opening device. Each time we trying to post WR(s) to a QP, the driver will check if current QP has WQE buffer.
If not, the driver will check whether there is enough free memory in the DCA memory pool. If there is, the QP will get WQE buffer from the pool, including SQ buffer in size of max_send_wr, RQ buffer in size of max_recv_wr and extended sge buffer. If there is no enough space for the WQE buffer in the DCA pool, the driver will expand the size of pool and then assign the buffer to the QP.
And if the WQE buffer of a QP is not used, this buffer would be recycled and the DCA memory pool would be shrinked.
Would like to see proper man pages explaining how this all works for rdma-core.
Jason
OK, will add man pages for DCA in next version.
Thanks Weihang