April 2025 - High-performance-network

[PATCH] libhns: Bugfixes and one debug improvement
by Junxian Huang 28 Apr '25

28 Apr '25

From: Xinghai Cen <cenxinghai(a)h-partners.com> The last commit was found when I created a XRC SRQ in lock-free mode but failed to destroy it because of the refcnt check added in the previous commit. The failure was because the PAD was acquired through ibv_srq->pd in destroy_srq(), while ibv_srq->pd wasn't assigned when the SRQ was created by ibv_create_srq_ex(). So let's assign ibv_srq->pd in the common ibv_icmd_create_srq() , so that drivers can get the correct pd no matter which api the SRQ is created by. Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> --- ...hns-Add-debug-log-for-lock-free-mode.patch | 59 +++++++++++ ...s-Fix-ret-not-assigned-in-create-srq.patch | 58 +++++++++++ ...efcnt-leaking-in-error-flow-of-creat.patch | 99 +++++++++++++++++++ ...-freeing-pad-without-checking-refcnt.patch | 69 +++++++++++++ ...-Assign-ibv-srq-pd-when-creating-SRQ.patch | 43 ++++++++ rdma-core.spec | 13 ++- 6 files changed, 340 insertions(+), 1 deletion(-) create mode 100644 0058-libhns-Add-debug-log-for-lock-free-mode.patch create mode 100644 0059-libhns-Fix-ret-not-assigned-in-create-srq.patch create mode 100644 0060-libhns-Fix-pad-refcnt-leaking-in-error-flow-of-creat.patch create mode 100644 0061-libhns-Fix-freeing-pad-without-checking-refcnt.patch create mode 100644 0062-verbs-Assign-ibv-srq-pd-when-creating-SRQ.patch diff --git a/0058-libhns-Add-debug-log-for-lock-free-mode.patch b/0058-libhns-Add-debug-log-for-lock-free-mode.patch new file mode 100644 index 0000000..86bc0a5 --- /dev/null +++ b/0058-libhns-Add-debug-log-for-lock-free-mode.patch @@ -0,0 +1,59 @@ +From 20dc7f183603b936ba7a865fc8a6d115073b1e29 Mon Sep 17 00:00:00 2001 +From: Junxian Huang <huangjunxian6(a)hisilicon.com> +Date: Thu, 24 Apr 2025 20:32:12 +0800 +Subject: [PATCH 58/62] libhns: Add debug log for lock-free mode + +mainline inclusion +from mainline-v56.0-65 +commit fb96940fcf6f96185d407d57bcaf775ccf8f1762 +category: cheanup +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC3X57 +CVE: NA + +Reference: +https://github.com/linux-rdma/rdma-core/pull/1599/commits/fb96940fcf6f96185d407d57bcaf775ccf8f1762 + +--------------------------------------------------------------------- + +Currently there is no way to observe whether the lock-free mode is +configured from the driver's perspective. Add debug log for this. + +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u_verbs.c | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +diff --git a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c +index 5fe169e..3efc2f4 100644 +--- a/providers/hns/hns_roce_u_verbs.c ++++ b/providers/hns/hns_roce_u_verbs.c +@@ -182,6 +182,7 @@ err: + struct ibv_pd *hns_roce_u_alloc_pad(struct ibv_context *context, + struct ibv_parent_domain_init_attr *attr) + { ++ struct hns_roce_pd *protection_domain; + struct hns_roce_pad *pad; + + if (ibv_check_alloc_parent_domain(attr)) +@@ -198,12 +199,16 @@ struct ibv_pd *hns_roce_u_alloc_pad(struct ibv_context *context, + return NULL; + } + ++ protection_domain = to_hr_pd(attr->pd); + if (attr->td) { + pad->td = to_hr_td(attr->td); + atomic_fetch_add(&pad->td->refcount, 1); ++ verbs_debug(verbs_get_ctx(context), ++ "set PAD(0x%x) to lock-free mode.\n", ++ protection_domain->pdn); + } + +- pad->pd.protection_domain = to_hr_pd(attr->pd); ++ pad->pd.protection_domain = protection_domain; + atomic_fetch_add(&pad->pd.protection_domain->refcount, 1); + + atomic_init(&pad->pd.refcount, 1); +-- +2.33.0 + diff --git a/0059-libhns-Fix-ret-not-assigned-in-create-srq.patch b/0059-libhns-Fix-ret-not-assigned-in-create-srq.patch new file mode 100644 index 0000000..10443d6 --- /dev/null +++ b/0059-libhns-Fix-ret-not-assigned-in-create-srq.patch @@ -0,0 +1,58 @@ +From cf284feddf6bb98c600061a3fd1f0095e46b540e Mon Sep 17 00:00:00 2001 +From: Junxian Huang <huangjunxian6(a)hisilicon.com> +Date: Wed, 23 Apr 2025 16:55:14 +0800 +Subject: [PATCH 59/62] libhns: Fix ret not assigned in create srq() + +mainline inclusion +from mainline-v56.0-65 +commit 2034b1860c5a8b0cc3879315259462c04e53a98d +category: bugfix +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC3X57 +CVE: NA + +Reference: +https://github.com/linux-rdma/rdma-core/pull/1599/commits/2034b1860c5a8b0cc3879315259462c04e53a98d + +--------------------------------------------------------------------- + +Fix the problem that ret may not be assigned in the error flow +of create_srq(). + +Fixes: aa7bcf7f7e44 ("libhns: Add support for lock-free SRQ") +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u_verbs.c | 10 +++++++--- + 1 file changed, 7 insertions(+), 3 deletions(-) + +diff --git a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c +index 3efc2f4..b26ac29 100644 +--- a/providers/hns/hns_roce_u_verbs.c ++++ b/providers/hns/hns_roce_u_verbs.c +@@ -933,16 +933,20 @@ static struct ibv_srq *create_srq(struct ibv_context *context, + if (pad) + atomic_fetch_add(&pad->pd.refcount, 1); + +- if (hns_roce_srq_spinlock_init(context, srq, init_attr)) ++ ret = hns_roce_srq_spinlock_init(context, srq, init_attr) ++ if (ret) + goto err_free_srq; + + set_srq_param(context, srq, init_attr); +- if (alloc_srq_buf(srq)) ++ ret = alloc_srq_buf(srq) ++ if (ret) + goto err_destroy_lock; + + srq->rdb = hns_roce_alloc_db(hr_ctx, HNS_ROCE_SRQ_TYPE_DB); +- if (!srq->rdb) ++ if (!srq->rdb) { ++ ret = ENOMEM; + goto err_srq_buf; ++ } + + ret = exec_srq_create_cmd(context, srq, init_attr); + if (ret) +-- +2.33.0 + diff --git a/0060-libhns-Fix-pad-refcnt-leaking-in-error-flow-of-creat.patch b/0060-libhns-Fix-pad-refcnt-leaking-in-error-flow-of-creat.patch new file mode 100644 index 0000000..95a1e69 --- /dev/null +++ b/0060-libhns-Fix-pad-refcnt-leaking-in-error-flow-of-creat.patch @@ -0,0 +1,99 @@ +From 2bc3cafa227528e2893dadfff7cf54cfee427e1a Mon Sep 17 00:00:00 2001 +From: Junxian Huang <huangjunxian6(a)hisilicon.com> +Date: Wed, 23 Apr 2025 16:55:15 +0800 +Subject: [PATCH 60/62] libhns: Fix pad refcnt leaking in error flow of create + qp/cq/srq + +mainline inclusion +from mainline-v56.0-65 +commit f877d6e610e438515e1535c9ec7a3a3ef37c58e0 +category: bugfix +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC3X57 +CVE: NA + +Reference: +https://github.com/linux-rdma/rdma-core/pull/1599/commits/f877d6e610e438515e1535c9ec7a3a3ef37c58e0 + +--------------------------------------------------------------------- + +Decrease pad refcnt by 1 in error flow of create qp/cq/srq. + +Fixes: f8b4f622b1c5 ("libhns: Add support for lock-free QP") +Fixes: 95225025e24c ("libhns: Add support for lock-free CQ") +Fixes: aa7bcf7f7e44 ("libhns: Add support for lock-free SRQ") +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u_verbs.c | 20 +++++++++++++------- + 1 file changed, 13 insertions(+), 7 deletions(-) + +diff --git a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c +index b26ac29..5a62bb2 100644 +--- a/providers/hns/hns_roce_u_verbs.c ++++ b/providers/hns/hns_roce_u_verbs.c +@@ -445,12 +445,9 @@ static int verify_cq_create_attr(struct ibv_cq_init_attr_ex *attr, + return EOPNOTSUPP; + } + +- if (attr->comp_mask & IBV_CQ_INIT_ATTR_MASK_PD) { +- if (!pad) { +- verbs_err(&context->ibv_ctx, "failed to check the pad of cq.\n"); +- return EINVAL; +- } +- atomic_fetch_add(&pad->pd.refcount, 1); ++ if (attr->comp_mask & IBV_CQ_INIT_ATTR_MASK_PD && !pad) { ++ verbs_err(&context->ibv_ctx, "failed to check the pad of cq.\n"); ++ return EINVAL; + } + + attr->cqe = max_t(uint32_t, HNS_ROCE_MIN_CQE_NUM, +@@ -556,6 +553,7 @@ static void hns_roce_uninit_cq_swc(struct hns_roce_cq *cq) + static struct ibv_cq_ex *create_cq(struct ibv_context *context, + struct ibv_cq_init_attr_ex *attr) + { ++ struct hns_roce_pad *pad = to_hr_pad(attr->parent_domain); + struct hns_roce_context *hr_ctx = to_hr_ctx(context); + struct hns_roce_cq *cq; + int ret; +@@ -570,8 +568,10 @@ static struct ibv_cq_ex *create_cq(struct ibv_context *context, + goto err; + } + +- if (attr->comp_mask & IBV_CQ_INIT_ATTR_MASK_PD) ++ if (attr->comp_mask & IBV_CQ_INIT_ATTR_MASK_PD) { + cq->parent_domain = attr->parent_domain; ++ atomic_fetch_add(&pad->pd.refcount, 1); ++ } + + ret = hns_roce_cq_spinlock_init(context, cq, attr); + if (ret) +@@ -611,6 +611,8 @@ err_db: + err_buf: + hns_roce_spinlock_destroy(&cq->hr_lock); + err_lock: ++ if (attr->comp_mask & IBV_CQ_INIT_ATTR_MASK_PD) ++ atomic_fetch_sub(&pad->pd.refcount, 1); + free(cq); + err: + if (ret < 0) +@@ -977,6 +979,8 @@ err_destroy_lock: + hns_roce_spinlock_destroy(&srq->hr_lock); + + err_free_srq: ++ if (pad) ++ atomic_fetch_sub(&pad->pd.refcount, 1); + free(srq); + + err: +@@ -1872,6 +1876,8 @@ err_cmd: + err_buf: + hns_roce_qp_spinlock_destroy(qp); + err_spinlock: ++ if (pad) ++ atomic_fetch_sub(&pad->pd.refcount, 1); + free(qp); + err: + if (ret < 0) +-- +2.33.0 + diff --git a/0061-libhns-Fix-freeing-pad-without-checking-refcnt.patch b/0061-libhns-Fix-freeing-pad-without-checking-refcnt.patch new file mode 100644 index 0000000..d37dee6 --- /dev/null +++ b/0061-libhns-Fix-freeing-pad-without-checking-refcnt.patch @@ -0,0 +1,69 @@ +From 0db7ff07caa483da0fb2cfd7944d549a38b4c720 Mon Sep 17 00:00:00 2001 +From: Junxian Huang <huangjunxian6(a)hisilicon.com> +Date: Wed, 23 Apr 2025 16:55:16 +0800 +Subject: [PATCH 61/62] libhns: Fix freeing pad without checking refcnt + +mainline inclusion +from mainline-v56.0-65 +commit 234d135276ea8ef83633113e224e0cd735ebeca8 +category: bugfix +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC3X57 +CVE: NA + +Reference: +https://github.com/linux-rdma/rdma-core/pull/1599/commits/234d135276ea8ef83633113e224e0cd735ebeca8 + +--------------------------------------------------------------------- + +Currently pad refcnt will be added when creating qp/cq/srq, but it is +not checked when freeing pad. Add a check to prevent freeing pad when +it is still used by any qp/cq/srq. + +Fixes: 7b6b3dae328f ("libhns: Add support for thread domain and parent +domain") +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u_verbs.c | 12 +++++++----- + 1 file changed, 7 insertions(+), 5 deletions(-) + +diff --git a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c +index 5a62bb2..8c37496 100644 +--- a/providers/hns/hns_roce_u_verbs.c ++++ b/providers/hns/hns_roce_u_verbs.c +@@ -218,14 +218,18 @@ struct ibv_pd *hns_roce_u_alloc_pad(struct ibv_context *context, + return &pad->pd.ibv_pd; + } + +-static void hns_roce_free_pad(struct hns_roce_pad *pad) ++static int hns_roce_free_pad(struct hns_roce_pad *pad) + { ++ if (atomic_load(&pad->pd.refcount) > 1) ++ return EBUSY; ++ + atomic_fetch_sub(&pad->pd.protection_domain->refcount, 1); + + if (pad->td) + atomic_fetch_sub(&pad->td->refcount, 1); + + free(pad); ++ return 0; + } + + static int hns_roce_free_pd(struct hns_roce_pd *pd) +@@ -248,10 +252,8 @@ int hns_roce_u_dealloc_pd(struct ibv_pd *ibv_pd) + struct hns_roce_pad *pad = to_hr_pad(ibv_pd); + struct hns_roce_pd *pd = to_hr_pd(ibv_pd); + +- if (pad) { +- hns_roce_free_pad(pad); +- return 0; +- } ++ if (pad) ++ return hns_roce_free_pad(pad); + + return hns_roce_free_pd(pd); + } +-- +2.33.0 + diff --git a/0062-verbs-Assign-ibv-srq-pd-when-creating-SRQ.patch b/0062-verbs-Assign-ibv-srq-pd-when-creating-SRQ.patch new file mode 100644 index 0000000..e7d0395 --- /dev/null +++ b/0062-verbs-Assign-ibv-srq-pd-when-creating-SRQ.patch @@ -0,0 +1,43 @@ +From f8f9295695921fa796bb93c5ee7066e50221bbc3 Mon Sep 17 00:00:00 2001 +From: Junxian Huang <huangjunxian6(a)hisilicon.com> +Date: Wed, 23 Apr 2025 16:55:17 +0800 +Subject: [PATCH 62/62] verbs: Assign ibv srq->pd when creating SRQ + +mainline inclusion +from mainline-v56.0-65 +commit bf1e427141fde2651bab4860e77a432bb7e26094 +category: bugfix +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC3X57 +CVE: NA + +Reference: +https://github.com/linux-rdma/rdma-core/pull/1599/commits/bf1e427141fde2651bab4860e77a432bb7e26094 + +--------------------------------------------------------------------- + +Some providers need to access ibv_srq->pd during SRQ destruction, but +it may not be assigned currently when using ibv_create_srq_ex(). This +may lead to some SRQ-related resource leaks. Assign ibv_srq->pd when +creating SRQ to ensure pd can be obtained correctly. + +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + libibverbs/cmd_srq.c | 1 + + 1 file changed, 1 insertion(+) + +diff --git a/libibverbs/cmd_srq.c b/libibverbs/cmd_srq.c +index dfaaa6a..259ea0d 100644 +--- a/libibverbs/cmd_srq.c ++++ b/libibverbs/cmd_srq.c +@@ -63,6 +63,7 @@ static int ibv_icmd_create_srq(struct ibv_pd *pd, struct verbs_srq *vsrq, + struct verbs_xrcd *vxrcd = NULL; + enum ibv_srq_type srq_type; + ++ srq->pd = pd; + srq->context = pd->context; + pthread_mutex_init(&srq->mutex, NULL); + pthread_cond_init(&srq->cond, NULL); +-- +2.33.0 + diff --git a/rdma-core.spec b/rdma-core.spec index e760f40..928fc75 100644 --- a/rdma-core.spec +++ b/rdma-core.spec @@ -1,6 +1,6 @@ Name: rdma-core Version: 50.0 -Release: 27 +Release: 28 Summary: RDMA core userspace libraries and daemons License: GPL-2.0-only OR BSD-2-Clause AND BSD-3-Clause Url: https://github.com/linux-rdma/rdma-core @@ -63,6 +63,11 @@ patch54: 0054-libhns-Fix-wrong-max-inline-data-value.patch patch55: 0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch patch56: 0056-libhns-Add-initial-support-for-HNS-LTTng-tracing.patch patch57: 0057-libhns-Add-tracepoint-for-HNS-RoCE-I-O.patch +patch58: 0058-libhns-Add-debug-log-for-lock-free-mode.patch +patch59: 0059-libhns-Fix-ret-not-assigned-in-create-srq.patch +patch60: 0060-libhns-Fix-pad-refcnt-leaking-in-error-flow-of-creat.patch +patch61: 0061-libhns-Fix-freeing-pad-without-checking-refcnt.patch +patch62: 0062-verbs-Assign-ibv-srq-pd-when-creating-SRQ.patch BuildRequires: binutils cmake >= 2.8.11 gcc libudev-devel pkgconfig pkgconfig(libnl-3.0) BuildRequires: pkgconfig(libnl-route-3.0) systemd systemd-devel @@ -642,6 +647,12 @@ fi %doc %{_docdir}/%{name}-%{version}/70-persistent-ipoib.rules %changelog +* Fri Apr 25 2025 Xinghai Cen <cenxinghai(a)h-partners.com> - 50.0-28 +- Type: bugfix +- ID: NA +- SUG: NA +- DESC: Bugfixes and one debug improvement + * Wed Apr 23 2025 Xinghai Cen <cenxinghai(a)h-partners.com> - 50.0-27 - Type: feature - ID: NA -- 2.33.0

1 0

Perftest: Do not align SRQ recv length to MTU for hns
by Junxian Huang 27 Apr '25

27 Apr '25

From: Chengchang Tang <tangchengchang(a)huawei.com> HNS RoCE will check whether the length of the recv sge exceeds the range of the corresponding MR, which can cause hns RoCE to report a CQE error. Signed-off-by: Chengchang Tang <tangchengchang(a)huawei.com> Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> --- src/perftest_resources.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/perftest_resources.c b/src/perftest_resources.c index 9e9f41f..f8c0de2 100755 --- a/src/perftest_resources.c +++ b/src/perftest_resources.c @@ -3261,6 +3261,19 @@ void ctx_set_send_reg_wqes(struct pingpong_context *ctx, } } +static uint64_t set_recv_length(struct pingpong_context *ctx, + struct perftest_parameters *user_param) +{ + enum ctx_device current_dev = ib_dev_name(ctx->context); + int mtu = MTU_SIZE(user_param->curr_mtu); + uint64_t length = SIZE(user_param->connection_type, user_param->size, 1); + + if (current_dev != HNS && user_param->use_srq == ON) + length = ((length + mtu - 1 )/ mtu) * mtu; + + return length; +} + /****************************************************************************** * ******************************************************************************/ @@ -3270,8 +3283,7 @@ int ctx_set_recv_wqes(struct pingpong_context *ctx,struct perftest_parameters *u int num_of_qps = user_param->num_of_qps; struct ibv_recv_wr *bad_wr_recv; int size_per_qp = user_param->rx_depth / user_param->recv_post_list; - int mtu = MTU_SIZE(user_param->curr_mtu); - uint64_t length = user_param->use_srq == ON ? (((SIZE(user_param->connection_type ,user_param->size, 1) + mtu - 1 )/ mtu) * mtu) : SIZE(user_param->connection_type, user_param->size, 1); + uint64_t length = set_recv_length(ctx, user_param); /* Write w/imm completions have zero recieve buffer length */ if (user_param->verb == WRITE_IMM) -- 2.33.0

1 0

[PATCH] libhns: Add support for LTTng tracing
by Junxian Huang 25 Apr '25

25 Apr '25

From: Xinghai Cen <cenxinghai(a)h-partners.com> Add support for LTTng tracing. For now it is used for post_send, post_recv and poll_cq. Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> --- ...nitial-support-for-HNS-LTTng-tracing.patch | 112 +++++ ...bhns-Add-tracepoint-for-HNS-RoCE-I-O.patch | 382 ++++++++++++++++++ rdma-core.spec | 10 +- 3 files changed, 503 insertions(+), 1 deletion(-) create mode 100644 0056-libhns-Add-initial-support-for-HNS-LTTng-tracing.patch create mode 100644 0057-libhns-Add-tracepoint-for-HNS-RoCE-I-O.patch diff --git a/0056-libhns-Add-initial-support-for-HNS-LTTng-tracing.patch b/0056-libhns-Add-initial-support-for-HNS-LTTng-tracing.patch new file mode 100644 index 0000000..f84fde7 --- /dev/null +++ b/0056-libhns-Add-initial-support-for-HNS-LTTng-tracing.patch @@ -0,0 +1,112 @@ +From dfcef98e85b947dd38738436c769926f66438a7d Mon Sep 17 00:00:00 2001 +From: wenglianfa <wenglianfa(a)huawei.com> +Date: Tue, 22 Apr 2025 16:18:44 +0800 +Subject: [PATCH 56/57] libhns: Add initial support for HNS LTTng tracing + +mainline inclusion +from mainline-v56.0-65 +commit 5d96d96c822323a1c9b0a6b98ce58a17a8f165c1 +category: feature +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC3E67 +CVE: NA + +Reference: https://github.com/linux-rdma/rdma-core/pull/1587/commits/5d96d96c822323a1c… + +--------------------------------------------------------------------- + +Add initial support for HNS LTTng tracing. + +Signed-off-by: wenglianfa <wenglianfa(a)huawei.com> +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/CMakeLists.txt | 10 +++++++++ + providers/hns/hns_roce_u_trace.c | 9 ++++++++ + providers/hns/hns_roce_u_trace.h | 35 ++++++++++++++++++++++++++++++++ + 3 files changed, 54 insertions(+) + create mode 100644 providers/hns/hns_roce_u_trace.c + create mode 100644 providers/hns/hns_roce_u_trace.h + +diff --git a/providers/hns/CMakeLists.txt b/providers/hns/CMakeLists.txt +index 58139ae..36ebfac 100644 +--- a/providers/hns/CMakeLists.txt ++++ b/providers/hns/CMakeLists.txt +@@ -1,5 +1,10 @@ ++if (ENABLE_LTTNG AND LTTNGUST_FOUND) ++ set(TRACE_FILE hns_roce_u_trace.c) ++endif() ++ + rdma_shared_provider(hns libhns.map + 1 1.0.${PACKAGE_VERSION} ++ ${TRACE_FILE} + hns_roce_u.c + hns_roce_u_buf.c + hns_roce_u_db.c +@@ -12,3 +17,8 @@ publish_headers(infiniband + ) + + rdma_pkg_config("hns" "libibverbs" "${CMAKE_THREAD_LIBS_INIT}") ++ ++if (ENABLE_LTTNG AND LTTNGUST_FOUND) ++ target_include_directories(hns PUBLIC ".") ++ target_link_libraries(hns LINK_PRIVATE LTTng::UST) ++endif() +diff --git a/providers/hns/hns_roce_u_trace.c b/providers/hns/hns_roce_u_trace.c +new file mode 100644 +index 0000000..812f54c +--- /dev/null ++++ b/providers/hns/hns_roce_u_trace.c +@@ -0,0 +1,9 @@ ++// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause ++/* ++ * Copyright (c) 2025 Hisilicon Limited. ++ */ ++ ++#define LTTNG_UST_TRACEPOINT_CREATE_PROBES ++#define LTTNG_UST_TRACEPOINT_DEFINE ++ ++#include "hns_roce_u_trace.h" +diff --git a/providers/hns/hns_roce_u_trace.h b/providers/hns/hns_roce_u_trace.h +new file mode 100644 +index 0000000..9b9485c +--- /dev/null ++++ b/providers/hns/hns_roce_u_trace.h +@@ -0,0 +1,35 @@ ++/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ ++/* ++ * Copyright (c) 2025 Hisilicon Limited. ++ */ ++ ++#if defined(LTTNG_ENABLED) ++ ++#undef LTTNG_UST_TRACEPOINT_PROVIDER ++#define LTTNG_UST_TRACEPOINT_PROVIDER rdma_core_hns ++ ++#undef LTTNG_UST_TRACEPOINT_INCLUDE ++#define LTTNG_UST_TRACEPOINT_INCLUDE "hns_roce_u_trace.h" ++ ++#if !defined(__HNS_TRACE_H__) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ) ++#define __HNS_TRACE_H__ ++ ++#include <lttng/tracepoint.h> ++#include <infiniband/verbs.h> ++ ++#define rdma_tracepoint(arg...) lttng_ust_tracepoint(arg) ++ ++#endif /* __HNS_TRACE_H__*/ ++ ++#include <lttng/tracepoint-event.h> ++ ++#else ++ ++#ifndef __HNS_TRACE_H__ ++#define __HNS_TRACE_H__ ++ ++#define rdma_tracepoint(arg...) ++ ++#endif /* __HNS_TRACE_H__*/ ++ ++#endif /* defined(LTTNG_ENABLED) */ +-- +2.33.0 + diff --git a/0057-libhns-Add-tracepoint-for-HNS-RoCE-I-O.patch b/0057-libhns-Add-tracepoint-for-HNS-RoCE-I-O.patch new file mode 100644 index 0000000..39c6d6f --- /dev/null +++ b/0057-libhns-Add-tracepoint-for-HNS-RoCE-I-O.patch @@ -0,0 +1,382 @@ +From feec8deebf58cf6faaf9f70eda49b929eb674f72 Mon Sep 17 00:00:00 2001 +From: wenglianfa <wenglianfa(a)huawei.com> +Date: Tue, 22 Apr 2025 16:18:45 +0800 +Subject: [PATCH 57/57] libhns: Add tracepoint for HNS RoCE I/O + +mainline inclusion +from mainline-v56.0-65 +commit 19cb51c73029b593608f0c5d41a4ace8d1f1e334 +category: feature +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC3E67 +CVE: NA + +Reference: https://github.com/linux-rdma/rdma-core/pull/1587/commits/19cb51c73029b5936… + +--------------------------------------------------------------------- + +Add tracepoint for HNS RoCE I/O, including post_send, post_recv and +poll_cq. + +Signed-off-by: wenglianfa <wenglianfa(a)huawei.com> +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u_hw_v2.c | 153 +++++++++++++++++++++++++++++++ + providers/hns/hns_roce_u_trace.h | 98 ++++++++++++++++++++ + 2 files changed, 251 insertions(+) + +diff --git a/providers/hns/hns_roce_u_hw_v2.c b/providers/hns/hns_roce_u_hw_v2.c +index 3a1249f..b80c574 100644 +--- a/providers/hns/hns_roce_u_hw_v2.c ++++ b/providers/hns/hns_roce_u_hw_v2.c +@@ -38,6 +38,7 @@ + #include "hns_roce_u.h" + #include "hns_roce_u_db.h" + #include "hns_roce_u_hw_v2.h" ++#include "hns_roce_u_trace.h" + + #define HR_IBV_OPC_MAP(ib_key, hr_key) \ + [IBV_WR_ ## ib_key] = HNS_ROCE_WQE_OP_ ## hr_key +@@ -764,6 +765,80 @@ static int parse_cqe_for_cq(struct hns_roce_context *ctx, struct hns_roce_cq *cq + return 0; + } + ++#ifdef LTTNG_ENABLED ++static uint8_t read_wc_sl(struct hns_roce_qp *hr_qp, ++ struct hns_roce_v2_cqe *cqe, ++ struct ibv_wc *wc) ++{ ++ return hr_qp->verbs_qp.qp.qp_type == IBV_QPT_UD && ++ hr_reg_read(cqe, CQE_S_R) == CQE_FOR_RQ ? ++ wc->sl : UINT8_MAX; ++} ++ ++static uint32_t read_wc_rqpn(struct hns_roce_qp *hr_qp, ++ struct hns_roce_v2_cqe *cqe, ++ struct ibv_wc *wc) ++{ ++ return hr_qp->verbs_qp.qp.qp_type == IBV_QPT_UD && ++ hr_reg_read(cqe, CQE_S_R) == CQE_FOR_RQ ? ++ wc->src_qp : UINT32_MAX; ++} ++ ++static uint32_t read_wc_byte_len(struct hns_roce_v2_cqe *cqe, ++ struct ibv_wc *wc) ++{ ++ if (hr_reg_read(cqe, CQE_S_R) == CQE_FOR_RQ) ++ return wc->byte_len; ++ ++ switch (hr_reg_read(cqe, CQE_OPCODE)) { ++ case HNS_ROCE_SQ_OP_RDMA_READ: ++ case HNS_ROCE_SQ_OP_ATOMIC_COMP_AND_SWAP: ++ case HNS_ROCE_SQ_OP_ATOMIC_FETCH_AND_ADD: ++ case HNS_ROCE_SQ_OP_ATOMIC_MASK_COMP_AND_SWAP: ++ case HNS_ROCE_SQ_OP_ATOMIC_MASK_FETCH_AND_ADD: ++ return wc->byte_len; ++ default: ++ return UINT32_MAX; ++ } ++} ++ ++static uint8_t trace_wc_read_sl(struct ibv_cq_ex *cq_ex) ++{ ++ return cq_ex->read_sl ? cq_ex->read_sl(cq_ex) : UINT8_MAX; ++} ++ ++static uint32_t trace_wc_read_qp_num(struct ibv_cq_ex *cq_ex) ++{ ++ return cq_ex->read_qp_num ? ++ cq_ex->read_qp_num(cq_ex) : UINT32_MAX; ++} ++ ++static uint32_t trace_wc_read_src_qp(struct ibv_cq_ex *cq_ex) ++{ ++ return cq_ex->read_src_qp ? ++ cq_ex->read_src_qp(cq_ex) : UINT32_MAX; ++} ++ ++static uint32_t trace_wc_read_byte_len(struct ibv_cq_ex *cq_ex) ++{ ++ return cq_ex->read_byte_len ? ++ cq_ex->read_byte_len(cq_ex) : UINT32_MAX; ++} ++ ++static uint32_t get_send_wr_rqpn(struct ibv_send_wr *wr, ++ uint8_t qp_type) ++{ ++ return qp_type == IBV_QPT_UD ? wr->wr.ud.remote_qpn : UINT32_MAX; ++} ++ ++static uint8_t get_send_wr_tclass(struct ibv_send_wr *wr, ++ uint8_t qp_type) ++{ ++ return qp_type == IBV_QPT_UD ? ++ to_hr_ah(wr->wr.ud.ah)->av.tclass : UINT8_MAX; ++} ++#endif ++ + static int hns_roce_poll_one(struct hns_roce_context *ctx, + struct hns_roce_qp **cur_qp, struct hns_roce_cq *cq, + struct ibv_wc *wc) +@@ -800,8 +875,27 @@ static int hns_roce_poll_one(struct hns_roce_context *ctx, + wc->status = wc_status; + wc->vendor_err = hr_reg_read(cqe, CQE_SUB_STATUS); + wc->qp_num = qpn; ++ ++ rdma_tracepoint(rdma_core_hns, poll_cq, ++ cq->verbs_cq.cq.context->device->name, ++ wc->wr_id, wc_status, wc->opcode, ++ wc->wc_flags, wc->vendor_err, ++ read_wc_sl(*cur_qp, cqe, wc), ++ wc->qp_num, read_wc_rqpn(*cur_qp, cqe, wc), ++ read_wc_byte_len(cqe, wc)); + } else { + cq->verbs_cq.cq_ex.status = wc_status; ++ ++ rdma_tracepoint(rdma_core_hns, poll_cq, ++ cq->verbs_cq.cq.context->device->name, ++ cq->verbs_cq.cq_ex.wr_id, wc_status, ++ ibv_wc_read_opcode(&cq->verbs_cq.cq_ex), ++ ibv_wc_read_wc_flags(&cq->verbs_cq.cq_ex), ++ ibv_wc_read_vendor_err(&cq->verbs_cq.cq_ex), ++ trace_wc_read_sl(&cq->verbs_cq.cq_ex), ++ trace_wc_read_qp_num(&cq->verbs_cq.cq_ex), ++ trace_wc_read_src_qp(&cq->verbs_cq.cq_ex), ++ trace_wc_read_byte_len(&cq->verbs_cq.cq_ex)); + } + + if (status == HNS_ROCE_V2_CQE_SUCCESS || +@@ -1635,6 +1729,14 @@ int hns_roce_u_v2_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, + *bad_wr = wr; + goto out; + } ++ ++ rdma_tracepoint(rdma_core_hns, post_send, ++ ibvqp->context->device->name, wr->wr_id, ++ sge_info.valid_num, ibvqp->qp_num, ++ get_send_wr_rqpn(wr, ibvqp->qp_type), ++ wr->send_flags, sge_info.total_len, ++ wr->opcode, qp->sl, ++ get_send_wr_tclass(wr, ibvqp->qp_type)); + } + + out: +@@ -1785,6 +1887,10 @@ static int hns_roce_u_v2_post_recv(struct ibv_qp *ibvqp, struct ibv_recv_wr *wr, + wqe_idx = (qp->rq.head + nreq) & (qp->rq.wqe_cnt - 1); + fill_rq_wqe(qp, wr, wqe_idx, max_sge); + qp->rq.wrid[wqe_idx] = wr->wr_id; ++ ++ rdma_tracepoint(rdma_core_hns, post_recv, ++ ibvqp->context->device->name, wr->wr_id, ++ wr->num_sge, ibvqp->qp_num, 0); + } + + out: +@@ -2153,6 +2259,10 @@ static int hns_roce_u_v2_post_srq_recv(struct ibv_srq *ib_srq, + fill_wqe_idx(srq, wqe_idx); + + srq->wrid[wqe_idx] = wr->wr_id; ++ ++ rdma_tracepoint(rdma_core_hns, post_recv, ++ ib_srq->context->device->name, wr->wr_id, ++ wr->num_sge, srq->srqn, 1); + } + + if (nreq) { +@@ -2442,6 +2552,12 @@ static void wr_set_sge_rc(struct ibv_qp_ex *ibv_qp, uint32_t lkey, + wqe->msg_len = htole32(length); + hr_reg_write(wqe, RCWQE_LEN0, length); + hr_reg_write(wqe, RCWQE_SGE_NUM, !!length); ++ ++ rdma_tracepoint(rdma_core_hns, post_send, ++ ibv_qp->qp_base.context->device->name, ibv_qp->wr_id, ++ !!length, ibv_qp->qp_base.qp_num, UINT32_MAX, ++ ibv_qp->wr_flags, length, ++ hr_reg_read(wqe, RCWQE_OPCODE), qp->sl, UINT8_MAX); + } + + static void set_sgl_rc(struct hns_roce_v2_wqe_data_seg *dseg, +@@ -2506,6 +2622,12 @@ static void wr_set_sge_list_rc(struct ibv_qp_ex *ibv_qp, size_t num_sge, + + wqe->msg_len = htole32(qp->sge_info.total_len); + hr_reg_write(wqe, RCWQE_SGE_NUM, qp->sge_info.valid_num); ++ ++ rdma_tracepoint(rdma_core_hns, post_send, ++ ibv_qp->qp_base.context->device->name, ibv_qp->wr_id, ++ qp->sge_info.valid_num, ibv_qp->qp_base.qp_num, ++ UINT32_MAX, ibv_qp->wr_flags, qp->sge_info.total_len, ++ opcode, qp->sl, UINT8_MAX); + } + + static void wr_send_rc(struct ibv_qp_ex *ibv_qp) +@@ -2680,6 +2802,14 @@ static void set_inline_data_list_rc(struct hns_roce_qp *qp, + + hr_reg_write(wqe, RCWQE_SGE_NUM, qp->sge_info.valid_num); + } ++ ++ rdma_tracepoint(rdma_core_hns, post_send, ++ qp->verbs_qp.qp.context->device->name, ++ qp->verbs_qp.qp_ex.wr_id, ++ hr_reg_read(wqe, RCWQE_SGE_NUM), ++ qp->verbs_qp.qp.qp_num, UINT32_MAX, ++ qp->verbs_qp.qp_ex.wr_flags, msg_len, ++ hr_reg_read(wqe, RCWQE_OPCODE), qp->sl, UINT8_MAX); + } + + static void wr_set_inline_data_rc(struct ibv_qp_ex *ibv_qp, void *addr, +@@ -2812,6 +2942,13 @@ static void wr_set_sge_ud(struct ibv_qp_ex *ibv_qp, uint32_t lkey, + dseg->len = htole32(length); + + qp->sge_info.start_idx++; ++ ++ rdma_tracepoint(rdma_core_hns, post_send, ++ ibv_qp->qp_base.context->device->name, ibv_qp->wr_id, ++ 1, ibv_qp->qp_base.qp_num, ++ hr_reg_read(wqe, UDWQE_DQPN), ibv_qp->wr_flags, ++ length, hr_reg_read(wqe, UDWQE_OPCODE), ++ qp->sl, hr_reg_read(wqe, UDWQE_TCLASS)); + } + + static void wr_set_sge_list_ud(struct ibv_qp_ex *ibv_qp, size_t num_sge, +@@ -2850,6 +2987,13 @@ static void wr_set_sge_list_ud(struct ibv_qp_ex *ibv_qp, size_t num_sge, + hr_reg_write(wqe, UDWQE_SGE_NUM, cnt); + + qp->sge_info.start_idx += cnt; ++ ++ rdma_tracepoint(rdma_core_hns, post_send, ++ ibv_qp->qp_base.context->device->name, ibv_qp->wr_id, ++ cnt, ibv_qp->qp_base.qp_num, ++ hr_reg_read(wqe, UDWQE_DQPN), ibv_qp->wr_flags, ++ msg_len, hr_reg_read(wqe, UDWQE_OPCODE), ++ qp->sl, hr_reg_read(wqe, UDWQE_TCLASS)); + } + + static void set_inline_data_list_ud(struct hns_roce_qp *qp, +@@ -2898,6 +3042,15 @@ static void set_inline_data_list_ud(struct hns_roce_qp *qp, + + hr_reg_write(wqe, UDWQE_SGE_NUM, qp->sge_info.valid_num); + } ++ ++ rdma_tracepoint(rdma_core_hns, post_send, ++ qp->verbs_qp.qp.context->device->name, ++ qp->verbs_qp.qp_ex.wr_id, ++ hr_reg_read(wqe, UDWQE_SGE_NUM), ++ qp->verbs_qp.qp.qp_num, hr_reg_read(wqe, UDWQE_DQPN), ++ qp->verbs_qp.qp_ex.wr_flags, msg_len, ++ hr_reg_read(wqe, UDWQE_OPCODE), qp->sl, ++ hr_reg_read(wqe, UDWQE_TCLASS)); + } + + static void wr_set_inline_data_ud(struct ibv_qp_ex *ibv_qp, void *addr, +diff --git a/providers/hns/hns_roce_u_trace.h b/providers/hns/hns_roce_u_trace.h +index 9b9485c..4654985 100644 +--- a/providers/hns/hns_roce_u_trace.h ++++ b/providers/hns/hns_roce_u_trace.h +@@ -17,6 +17,104 @@ + #include <lttng/tracepoint.h> + #include <infiniband/verbs.h> + ++LTTNG_UST_TRACEPOINT_EVENT( ++ /* Tracepoint provider name */ ++ rdma_core_hns, ++ ++ /* Tracepoint name */ ++ post_send, ++ ++ /* Input arguments */ ++ LTTNG_UST_TP_ARGS( ++ char *, dev_name, ++ uint64_t, wr_id, ++ int32_t, num_sge, ++ uint32_t, lqpn, ++ uint32_t, rqpn, ++ uint32_t, send_flags, ++ uint32_t, msg_len, ++ uint8_t, opcode, ++ uint8_t, sl, ++ uint8_t, t_class ++ ), ++ ++ /* Output event fields */ ++ LTTNG_UST_TP_FIELDS( ++ lttng_ust_field_string(dev_name, dev_name) ++ lttng_ust_field_integer_hex(uint64_t, wr_id, wr_id) ++ lttng_ust_field_integer_hex(int32_t, num_sge, num_sge) ++ lttng_ust_field_integer_hex(uint32_t, lqpn, lqpn) ++ lttng_ust_field_integer_hex(uint32_t, rqpn, rqpn) ++ lttng_ust_field_integer_hex(uint32_t, send_flags, send_flags) ++ lttng_ust_field_integer_hex(uint32_t, msg_len, msg_len) ++ lttng_ust_field_integer_hex(uint8_t, opcode, opcode) ++ lttng_ust_field_integer_hex(uint8_t, sl, sl) ++ lttng_ust_field_integer_hex(uint8_t, t_class, t_class) ++ ) ++) ++ ++LTTNG_UST_TRACEPOINT_EVENT( ++ /* Tracepoint provider name */ ++ rdma_core_hns, ++ ++ /* Tracepoint name */ ++ post_recv, ++ ++ /* Input arguments */ ++ LTTNG_UST_TP_ARGS( ++ char *, dev_name, ++ uint64_t, wr_id, ++ int32_t, num_sge, ++ uint32_t, rqn, ++ uint8_t, is_srq ++ ), ++ ++ /* Output event fields */ ++ LTTNG_UST_TP_FIELDS( ++ lttng_ust_field_string(dev_name, dev_name) ++ lttng_ust_field_integer_hex(uint64_t, wr_id, wr_id) ++ lttng_ust_field_integer_hex(int32_t, num_sge, num_sge) ++ lttng_ust_field_integer_hex(uint32_t, rqn, rqn) ++ lttng_ust_field_integer_hex(uint8_t, is_srq, is_srq) ++ ) ++) ++ ++LTTNG_UST_TRACEPOINT_EVENT( ++ /* Tracepoint provider name */ ++ rdma_core_hns, ++ ++ /* Tracepoint name */ ++ poll_cq, ++ ++ /* Input arguments */ ++ LTTNG_UST_TP_ARGS( ++ char *, dev_name, ++ uint64_t, wr_id, ++ uint8_t, status, ++ uint8_t, opcode, ++ uint8_t, wc_flags, ++ uint8_t, vendor_err, ++ uint8_t, pktype, ++ uint32_t, lqpn, ++ uint32_t, rqpn, ++ uint32_t, byte_len ++ ), ++ ++ /* Output event fields */ ++ LTTNG_UST_TP_FIELDS( ++ lttng_ust_field_string(dev_name, dev_name) ++ lttng_ust_field_integer_hex(uint64_t, wr_id, wr_id) ++ lttng_ust_field_integer_hex(uint8_t, status, status) ++ lttng_ust_field_integer_hex(uint8_t, opcode, opcode) ++ lttng_ust_field_integer_hex(uint8_t, wc_flags, wc_flags) ++ lttng_ust_field_integer_hex(uint8_t, vendor_err, vendor_err) ++ lttng_ust_field_integer_hex(uint8_t, pktype, pktype) ++ lttng_ust_field_integer_hex(uint32_t, lqpn, lqpn) ++ lttng_ust_field_integer_hex(uint32_t, rqpn, rqpn) ++ lttng_ust_field_integer_hex(uint32_t, byte_len, byte_len) ++ ) ++) ++ + #define rdma_tracepoint(arg...) lttng_ust_tracepoint(arg) + + #endif /* __HNS_TRACE_H__*/ +-- +2.33.0 + diff --git a/rdma-core.spec b/rdma-core.spec index b49c6bf..e760f40 100644 --- a/rdma-core.spec +++ b/rdma-core.spec @@ -1,6 +1,6 @@ Name: rdma-core Version: 50.0 -Release: 26 +Release: 27 Summary: RDMA core userspace libraries and daemons License: GPL-2.0-only OR BSD-2-Clause AND BSD-3-Clause Url: https://github.com/linux-rdma/rdma-core @@ -61,6 +61,8 @@ patch52: 0052-libxscale-Match-dev-by-vid-and-did.patch patch53: 0053-libhns-Clean-up-data-type-issues.patch patch54: 0054-libhns-Fix-wrong-max-inline-data-value.patch patch55: 0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch +patch56: 0056-libhns-Add-initial-support-for-HNS-LTTng-tracing.patch +patch57: 0057-libhns-Add-tracepoint-for-HNS-RoCE-I-O.patch BuildRequires: binutils cmake >= 2.8.11 gcc libudev-devel pkgconfig pkgconfig(libnl-3.0) BuildRequires: pkgconfig(libnl-route-3.0) systemd systemd-devel @@ -640,6 +642,12 @@ fi %doc %{_docdir}/%{name}-%{version}/70-persistent-ipoib.rules %changelog +* Wed Apr 23 2025 Xinghai Cen <cenxinghai(a)h-partners.com> - 50.0-27 +- Type: feature +- ID: NA +- SUG: NA +- DESC: libhns: Add support for LTTng tracing + * Thu Apr 17 2025 Xinghai Cen <cenxinghai(a)h-partners.com> - 50.0-26 - Type: bugfix - ID: NA -- 2.33.0

1 0

[PATCH v2 rdma-core 1/5] libhns: Add debug log for lock-free mode
by Junxian Huang 24 Apr '25

24 Apr '25

Currently there is no way to observe whether the lock-free mode is configured from the driver's perspective. Add debug log for this. Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> --- providers/hns/hns_roce_u_verbs.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c index 6fc8ece32..ae60955ba 100644 --- a/providers/hns/hns_roce_u_verbs.c +++ b/providers/hns/hns_roce_u_verbs.c @@ -177,6 +177,7 @@ err: struct ibv_pd *hns_roce_u_alloc_pad(struct ibv_context *context, struct ibv_parent_domain_init_attr *attr) { + struct hns_roce_pd *protection_domain; struct hns_roce_pad *pad; if (ibv_check_alloc_parent_domain(attr)) @@ -193,12 +194,16 @@ struct ibv_pd *hns_roce_u_alloc_pad(struct ibv_context *context, return NULL; } + protection_domain = to_hr_pd(attr->pd); if (attr->td) { pad->td = to_hr_td(attr->td); atomic_fetch_add(&pad->td->refcount, 1); + verbs_debug(verbs_get_ctx(context), + "set PAD(0x%x) to lock-free mode.\n", + protection_domain->pdn); } - pad->pd.protection_domain = to_hr_pd(attr->pd); + pad->pd.protection_domain = protection_domain; atomic_fetch_add(&pad->pd.protection_domain->refcount, 1); atomic_init(&pad->pd.refcount, 1); -- 2.33.0

1 0

[PATCH v1.1 rdma-core 0/5] Bugfixes and one debug improvement
by Junxian Huang 23 Apr '25

23 Apr '25

The first commit adds some debug logs for lock-free mode so that we can intuitively observe whether the lock-free mode is configured. The following three commits fix some errors. The issue of the last commit was found during the testing of the previous ones, where I created a XRC SRQ in lock-free mode but failed to destroy it because of the refcnt check added in the second commit. The failure was because the PAD was acquired through ibv_srq->pd in destroy_srq(), while ibv_srq->pd wasn't assigned when the SRQ was created by ibv_create_srq_ex(). Junxian Huang (5): libhns: Add debug logs for lock-free mode libhns: Fix ret not assigned in create_srq() libhns: Fix pad refcnt leaking in error flow of create_qp/cq/srq libhns: Fix freeing pad without checking refcnt verbs: Assign ibv_srq->pd when creating SRQ libibverbs/cmd_srq.c | 1 + providers/hns/hns_roce_u_verbs.c | 51 +++++++++++++++++++++++--------- 2 files changed, 38 insertions(+), 14 deletions(-) -- 2.33.0

1 5

libhns: Add initial support for HNS LTTng tracing
by Junxian Huang 22 Apr '25

22 Apr '25

From: wenglianfa <wenglianfa(a)huawei.com> Add initial support for HNS LTTng tracing. Signed-off-by: wenglianfa <wenglianfa(a)huawei.com> Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> --- providers/hns/CMakeLists.txt | 10 +++++++++ providers/hns/hns_roce_u_trace.c | 9 ++++++++ providers/hns/hns_roce_u_trace.h | 35 ++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+) create mode 100644 providers/hns/hns_roce_u_trace.c create mode 100644 providers/hns/hns_roce_u_trace.h diff --git a/providers/hns/CMakeLists.txt b/providers/hns/CMakeLists.txt index 58139ae2b..36ebfacfb 100644 --- a/providers/hns/CMakeLists.txt +++ b/providers/hns/CMakeLists.txt @@ -1,5 +1,10 @@ +if (ENABLE_LTTNG AND LTTNGUST_FOUND) + set(TRACE_FILE hns_roce_u_trace.c) +endif() + rdma_shared_provider(hns libhns.map 1 1.0.${PACKAGE_VERSION} + ${TRACE_FILE} hns_roce_u.c hns_roce_u_buf.c hns_roce_u_db.c @@ -12,3 +17,8 @@ publish_headers(infiniband ) rdma_pkg_config("hns" "libibverbs" "${CMAKE_THREAD_LIBS_INIT}") + +if (ENABLE_LTTNG AND LTTNGUST_FOUND) + target_include_directories(hns PUBLIC ".") + target_link_libraries(hns LINK_PRIVATE LTTng::UST) +endif() diff --git a/providers/hns/hns_roce_u_trace.c b/providers/hns/hns_roce_u_trace.c new file mode 100644 index 000000000..812f54cfe --- /dev/null +++ b/providers/hns/hns_roce_u_trace.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +/* + * Copyright (c) 2025 Hisilicon Limited. + */ + +#define LTTNG_UST_TRACEPOINT_CREATE_PROBES +#define LTTNG_UST_TRACEPOINT_DEFINE + +#include "hns_roce_u_trace.h" diff --git a/providers/hns/hns_roce_u_trace.h b/providers/hns/hns_roce_u_trace.h new file mode 100644 index 000000000..9b9485c59 --- /dev/null +++ b/providers/hns/hns_roce_u_trace.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ +/* + * Copyright (c) 2025 Hisilicon Limited. + */ + +#if defined(LTTNG_ENABLED) + +#undef LTTNG_UST_TRACEPOINT_PROVIDER +#define LTTNG_UST_TRACEPOINT_PROVIDER rdma_core_hns + +#undef LTTNG_UST_TRACEPOINT_INCLUDE +#define LTTNG_UST_TRACEPOINT_INCLUDE "hns_roce_u_trace.h" + +#if !defined(__HNS_TRACE_H__) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ) +#define __HNS_TRACE_H__ + +#include <lttng/tracepoint.h> +#include <infiniband/verbs.h> + +#define rdma_tracepoint(arg...) lttng_ust_tracepoint(arg) + +#endif /* __HNS_TRACE_H__*/ + +#include <lttng/tracepoint-event.h> + +#else + +#ifndef __HNS_TRACE_H__ +#define __HNS_TRACE_H__ + +#define rdma_tracepoint(arg...) + +#endif /* __HNS_TRACE_H__*/ + +#endif /* defined(LTTNG_ENABLED) */ -- 2.33.0

1 1

[PATCH v3 rdma-core 1/2] libhns: Add initial support for HNS LTTng tracing
by Junxian Huang 22 Apr '25

22 Apr '25

From: wenglianfa <wenglianfa(a)huawei.com> Add initial support for HNS LTTng tracing. Signed-off-by: wenglianfa <wenglianfa(a)huawei.com> Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> --- providers/hns/CMakeLists.txt | 10 +++++++++ providers/hns/hns_roce_u_trace.c | 9 ++++++++ providers/hns/hns_roce_u_trace.h | 35 ++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+) create mode 100644 providers/hns/hns_roce_u_trace.c create mode 100644 providers/hns/hns_roce_u_trace.h diff --git a/providers/hns/CMakeLists.txt b/providers/hns/CMakeLists.txt index 58139ae2b..36ebfacfb 100644 --- a/providers/hns/CMakeLists.txt +++ b/providers/hns/CMakeLists.txt @@ -1,5 +1,10 @@ +if (ENABLE_LTTNG AND LTTNGUST_FOUND) + set(TRACE_FILE hns_roce_u_trace.c) +endif() + rdma_shared_provider(hns libhns.map 1 1.0.${PACKAGE_VERSION} + ${TRACE_FILE} hns_roce_u.c hns_roce_u_buf.c hns_roce_u_db.c @@ -12,3 +17,8 @@ publish_headers(infiniband ) rdma_pkg_config("hns" "libibverbs" "${CMAKE_THREAD_LIBS_INIT}") + +if (ENABLE_LTTNG AND LTTNGUST_FOUND) + target_include_directories(hns PUBLIC ".") + target_link_libraries(hns LINK_PRIVATE LTTng::UST) +endif() diff --git a/providers/hns/hns_roce_u_trace.c b/providers/hns/hns_roce_u_trace.c new file mode 100644 index 000000000..812f54cfe --- /dev/null +++ b/providers/hns/hns_roce_u_trace.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +/* + * Copyright (c) 2025 Hisilicon Limited. + */ + +#define LTTNG_UST_TRACEPOINT_CREATE_PROBES +#define LTTNG_UST_TRACEPOINT_DEFINE + +#include "hns_roce_u_trace.h" diff --git a/providers/hns/hns_roce_u_trace.h b/providers/hns/hns_roce_u_trace.h new file mode 100644 index 000000000..9b9485c59 --- /dev/null +++ b/providers/hns/hns_roce_u_trace.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ +/* + * Copyright (c) 2025 Hisilicon Limited. + */ + +#if defined(LTTNG_ENABLED) + +#undef LTTNG_UST_TRACEPOINT_PROVIDER +#define LTTNG_UST_TRACEPOINT_PROVIDER rdma_core_hns + +#undef LTTNG_UST_TRACEPOINT_INCLUDE +#define LTTNG_UST_TRACEPOINT_INCLUDE "hns_roce_u_trace.h" + +#if !defined(__HNS_TRACE_H__) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ) +#define __HNS_TRACE_H__ + +#include <lttng/tracepoint.h> +#include <infiniband/verbs.h> + +#define rdma_tracepoint(arg...) lttng_ust_tracepoint(arg) + +#endif /* __HNS_TRACE_H__*/ + +#include <lttng/tracepoint-event.h> + +#else + +#ifndef __HNS_TRACE_H__ +#define __HNS_TRACE_H__ + +#define rdma_tracepoint(arg...) + +#endif /* __HNS_TRACE_H__*/ + +#endif /* defined(LTTNG_ENABLED) */ -- 2.33.0

1 1

[PATCH] libhns: Cleanup and Bugfixes
by Junxian Huang 22 Apr '25

22 Apr '25

From: Xinghai Cen <cenxinghai(a)h-partners.com> Cleanup and Bugfixes: 0053-libhns-Clean-up-data-type-issues.patch 0054-libhns-Fix-wrong-max-inline-data-value.patch 0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> --- 0053-libhns-Clean-up-data-type-issues.patch | 152 ++++++++++++++++++ ...bhns-Fix-wrong-max-inline-data-value.patch | 63 ++++++++ ...ng-order-of-spin-unlock-in-modify-qp.patch | 42 +++++ rdma-core.spec | 11 +- 4 files changed, 267 insertions(+), 1 deletion(-) create mode 100644 0053-libhns-Clean-up-data-type-issues.patch create mode 100644 0054-libhns-Fix-wrong-max-inline-data-value.patch create mode 100644 0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch diff --git a/0053-libhns-Clean-up-data-type-issues.patch b/0053-libhns-Clean-up-data-type-issues.patch new file mode 100644 index 0000000..95a95cc --- /dev/null +++ b/0053-libhns-Clean-up-data-type-issues.patch @@ -0,0 +1,152 @@ +From 8f95635c359ca3c36f5b1b48889719b6840c07cc Mon Sep 17 00:00:00 2001 +From: Junxian Huang <huangjunxian6(a)hisilicon.com> +Date: Thu, 13 Mar 2025 17:26:50 +0800 +Subject: [PATCH 53/55] libhns: Clean up data type issues + +mainline inclusion +from mainline-v56.0-65 +commit fbe8827f270d0aff4a28bb645b826fa98fe00c9d +category: bugfix +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC1V44 +CVE: NA + +Reference: https://github.com/linux-rdma/rdma-core/pull/1579/commits/fbe8827f270d0aff4… + +--------------------------------------------------------------------- + +Clean up mixed signed/unsigned type issues. Fix a wrong format +character as well. + +Fixes: cf6d9149f8f5 ("libhns: Introduce hns direct verbs") +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u.h | 4 ++-- + providers/hns/hns_roce_u_hw_v2.c | 15 ++++++++------- + providers/hns/hns_roce_u_verbs.c | 6 +++--- + 3 files changed, 13 insertions(+), 12 deletions(-) + +diff --git a/providers/hns/hns_roce_u.h b/providers/hns/hns_roce_u.h +index 5eedb81..e7e3f01 100644 +--- a/providers/hns/hns_roce_u.h ++++ b/providers/hns/hns_roce_u.h +@@ -356,7 +356,7 @@ struct hns_roce_wq { + unsigned long *wrid; + struct hns_roce_spinlock hr_lock; + unsigned int wqe_cnt; +- int max_post; ++ unsigned int max_post; + unsigned int head; + unsigned int tail; + unsigned int max_gs; +@@ -392,7 +392,7 @@ struct hns_roce_qp { + struct verbs_qp verbs_qp; + struct hns_roce_buf buf; + struct hns_roce_dca_buf dca_wqe; +- int max_inline_data; ++ unsigned int max_inline_data; + unsigned int buf_size; + unsigned int sq_signal_bits; + struct hns_roce_wq sq; +diff --git a/providers/hns/hns_roce_u_hw_v2.c b/providers/hns/hns_roce_u_hw_v2.c +index 3137111..cea3043 100644 +--- a/providers/hns/hns_roce_u_hw_v2.c ++++ b/providers/hns/hns_roce_u_hw_v2.c +@@ -173,7 +173,7 @@ static enum ibv_wc_status get_wc_status(uint8_t status) + { HNS_ROCE_V2_CQE_XRC_VIOLATION_ERR, IBV_WC_REM_INV_RD_REQ_ERR }, + }; + +- for (int i = 0; i < ARRAY_SIZE(map); i++) { ++ for (unsigned int i = 0; i < ARRAY_SIZE(map); i++) { + if (status == map[i].cqe_status) + return map[i].wc_status; + } +@@ -1189,7 +1189,7 @@ static int fill_ext_sge_inl_data(struct hns_roce_qp *qp, + unsigned int sge_mask = qp->ex_sge.sge_cnt - 1; + void *dst_addr, *src_addr, *tail_bound_addr; + uint32_t src_len, tail_len; +- int i; ++ uint32_t i; + + if (sge_info->total_len > qp->sq.ext_sge_cnt * HNS_ROCE_SGE_SIZE) + return EINVAL; +@@ -1259,7 +1259,7 @@ static void fill_ud_inn_inl_data(const struct ibv_send_wr *wr, + + static bool check_inl_data_len(struct hns_roce_qp *qp, unsigned int len) + { +- int mtu = mtu_enum_to_int(qp->path_mtu); ++ unsigned int mtu = mtu_enum_to_int(qp->path_mtu); + + return (len <= qp->max_inline_data && len <= mtu); + } +@@ -1698,7 +1698,8 @@ static void fill_recv_sge_to_wqe(struct ibv_recv_wr *wr, void *wqe, + unsigned int max_sge, bool rsv) + { + struct hns_roce_v2_wqe_data_seg *dseg = wqe; +- unsigned int i, cnt; ++ unsigned int cnt; ++ int i; + + for (i = 0, cnt = 0; i < wr->num_sge; i++) { + /* Skip zero-length sge */ +@@ -1726,7 +1727,7 @@ static void fill_recv_inl_buf(struct hns_roce_rinl_buf *rinl_buf, + unsigned int wqe_idx, struct ibv_recv_wr *wr) + { + struct ibv_sge *sge_list; +- unsigned int i; ++ int i; + + if (!rinl_buf->wqe_cnt) + return; +@@ -2053,7 +2054,7 @@ static int check_post_srq_valid(struct hns_roce_srq *srq, + static int get_wqe_idx(struct hns_roce_srq *srq, unsigned int *wqe_idx) + { + struct hns_roce_idx_que *idx_que = &srq->idx_que; +- int bit_num; ++ unsigned int bit_num; + int i; + + /* bitmap[i] is set zero if all bits are allocated */ +@@ -2451,7 +2452,7 @@ static void set_sgl_rc(struct hns_roce_v2_wqe_data_seg *dseg, + unsigned int mask = qp->ex_sge.sge_cnt - 1; + unsigned int msg_len = 0; + unsigned int cnt = 0; +- int i; ++ unsigned int i; + + for (i = 0; i < num_sge; i++) { + if (!sge[i].length) +diff --git a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c +index 848f836..f0098ed 100644 +--- a/providers/hns/hns_roce_u_verbs.c ++++ b/providers/hns/hns_roce_u_verbs.c +@@ -422,7 +422,7 @@ static int verify_cq_create_attr(struct ibv_cq_init_attr_ex *attr, + { + struct hns_roce_pad *pad = to_hr_pad(attr->parent_domain); + +- if (!attr->cqe || attr->cqe > context->max_cqe) { ++ if (!attr->cqe || attr->cqe > (uint32_t)context->max_cqe) { + verbs_err(&context->ibv_ctx, "unsupported cq depth %u.\n", + attr->cqe); + return EINVAL; +@@ -1080,7 +1080,7 @@ static int check_hnsdv_qp_attr(struct hns_roce_context *ctx, + return 0; + + if (!check_comp_mask(hns_attr->comp_mask, HNSDV_QP_SUP_COMP_MASK)) { +- verbs_err(&ctx->ibv_ctx, "invalid hnsdv comp_mask 0x%x.\n", ++ verbs_err(&ctx->ibv_ctx, "invalid hnsdv comp_mask 0x%llx.\n", + hns_attr->comp_mask); + return EINVAL; + } +@@ -1257,7 +1257,7 @@ static int alloc_recv_rinl_buf(uint32_t max_sge, + struct hns_roce_rinl_buf *rinl_buf) + { + unsigned int cnt; +- int i; ++ unsigned int i; + + cnt = rinl_buf->wqe_cnt; + rinl_buf->wqe_list = calloc(cnt, sizeof(struct hns_roce_rinl_wqe)); +-- +2.33.0 + diff --git a/0054-libhns-Fix-wrong-max-inline-data-value.patch b/0054-libhns-Fix-wrong-max-inline-data-value.patch new file mode 100644 index 0000000..4389023 --- /dev/null +++ b/0054-libhns-Fix-wrong-max-inline-data-value.patch @@ -0,0 +1,63 @@ +From 10534f0ef2ca73e8e59a38e51969cae864f9fbbf Mon Sep 17 00:00:00 2001 +From: wenglianfa <wenglianfa(a)huawei.com> +Date: Thu, 13 Mar 2025 17:26:51 +0800 +Subject: [PATCH 54/55] libhns: Fix wrong max inline data value + +mainline inclusion +from mainline-v56.0-65 +commit 8307b7c54ed81c343ec874e2066de79260b666d2 +category: bugfix +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC1V44 +CVE: NA + +Reference: https://github.com/linux-rdma/rdma-core/pull/1579/commits/8307b7c54ed81c343… + +--------------------------------------------------------------------- + +When cap.max_inline_data is 0, it will be modified to 1 since +roundup_pow_of_two(0) == 1, which violates users' expectations. +Here fix it. + +Fixes: 2aff0d55098c ("libhns: Fix the problem of sge nums") +Signed-off-by: wenglianfa <wenglianfa(a)huawei.com> +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u_verbs.c | 14 +++++++++++--- + 1 file changed, 11 insertions(+), 3 deletions(-) + +diff --git a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c +index f0098ed..5fe169e 100644 +--- a/providers/hns/hns_roce_u_verbs.c ++++ b/providers/hns/hns_roce_u_verbs.c +@@ -1494,6 +1494,16 @@ static unsigned int get_sge_num_from_max_inl_data(bool is_ud, + return inline_sge; + } + ++static uint32_t get_max_inline_data(struct hns_roce_context *ctx, ++ struct ibv_qp_cap *cap) ++{ ++ if (cap->max_inline_data) ++ return min_t(uint32_t, roundup_pow_of_two(cap->max_inline_data), ++ ctx->max_inline_data); ++ ++ return 0; ++} ++ + static void set_ext_sge_param(struct hns_roce_context *ctx, + struct ibv_qp_init_attr_ex *attr, + struct hns_roce_qp *qp, unsigned int wr_cnt) +@@ -1510,9 +1520,7 @@ static void set_ext_sge_param(struct hns_roce_context *ctx, + attr->cap.max_send_sge); + + if (ctx->config & HNS_ROCE_RSP_EXSGE_FLAGS) { +- attr->cap.max_inline_data = min_t(uint32_t, roundup_pow_of_two( +- attr->cap.max_inline_data), +- ctx->max_inline_data); ++ attr->cap.max_inline_data = get_max_inline_data(ctx, &attr->cap); + + inline_ext_sge = max(ext_wqe_sge_cnt, + get_sge_num_from_max_inl_data(is_ud, +-- +2.33.0 + diff --git a/0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch b/0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch new file mode 100644 index 0000000..b16fe12 --- /dev/null +++ b/0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch @@ -0,0 +1,42 @@ +From d1409106e1323c54fbbb0618c071efb024f58130 Mon Sep 17 00:00:00 2001 +From: Junxian Huang <huangjunxian6(a)hisilicon.com> +Date: Thu, 13 Mar 2025 17:26:52 +0800 +Subject: [PATCH 55/55] libhns: Fix wrong order of spin unlock in modify qp + +mainline inclusion +from mainline-v56.0-65 +commit d2b41c86c49335b3c6ab638abb1c0e31f5ba0e8f +category: bugfix +bugzilla: https://gitee.com/src-openeuler/rdma-core/issues/IC1V44 +CVE: NA + +Reference: https://github.com/linux-rdma/rdma-core/pull/1579/commits/d2b41c86c49335b3c… + +--------------------------------------------------------------------- + +The spin_unlock order should be the reverse of spin_lock order. + +Fixes: 179f015e090d ("libhns: Add support for lock-free QP") +Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> +Signed-off-by: Xinghai Cen <cenxinghai(a)h-partners.com> +--- + providers/hns/hns_roce_u_hw_v2.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/providers/hns/hns_roce_u_hw_v2.c b/providers/hns/hns_roce_u_hw_v2.c +index cea3043..3a1249f 100644 +--- a/providers/hns/hns_roce_u_hw_v2.c ++++ b/providers/hns/hns_roce_u_hw_v2.c +@@ -1910,8 +1910,8 @@ static int hns_roce_u_v2_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, + if (flag) { + if (!ret) + qp->state = IBV_QPS_ERR; +- hns_roce_spin_unlock(&hr_qp->sq.hr_lock); + hns_roce_spin_unlock(&hr_qp->rq.hr_lock); ++ hns_roce_spin_unlock(&hr_qp->sq.hr_lock); + } + + if (ret) +-- +2.33.0 + diff --git a/rdma-core.spec b/rdma-core.spec index 286b8dd..b49c6bf 100644 --- a/rdma-core.spec +++ b/rdma-core.spec @@ -1,6 +1,6 @@ Name: rdma-core Version: 50.0 -Release: 25 +Release: 26 Summary: RDMA core userspace libraries and daemons License: GPL-2.0-only OR BSD-2-Clause AND BSD-3-Clause Url: https://github.com/linux-rdma/rdma-core @@ -58,6 +58,9 @@ patch49: 0049-libzrdma-Add-poll-cqe-error-to-Failed-status.patch patch50: 0050-libzrdma-Add-sq-rq-flush-cqe-and-log-optimization.patch patch51: 0051-libzrdma-Fix-capability-related-bugs.patch patch52: 0052-libxscale-Match-dev-by-vid-and-did.patch +patch53: 0053-libhns-Clean-up-data-type-issues.patch +patch54: 0054-libhns-Fix-wrong-max-inline-data-value.patch +patch55: 0055-libhns-Fix-wrong-order-of-spin-unlock-in-modify-qp.patch BuildRequires: binutils cmake >= 2.8.11 gcc libudev-devel pkgconfig pkgconfig(libnl-3.0) BuildRequires: pkgconfig(libnl-route-3.0) systemd systemd-devel @@ -637,6 +640,12 @@ fi %doc %{_docdir}/%{name}-%{version}/70-persistent-ipoib.rules %changelog +* Thu Apr 17 2025 Xinghai Cen <cenxinghai(a)h-partners.com> - 50.0-26 +- Type: bugfix +- ID: NA +- SUG: NA +- DESC: libhns: Cleanup and Bugfixes + * Thu Mar 20 2025 Xin Tian <tianx(a)yunsilicon.com> - 50.0-25 - Type: bugfix - ID: NA -- 2.33.0

1 0

Perftest: Fix TD lock-free mode not working for SRQ/XRC QP
by Junxian Huang 15 Apr '25

15 Apr '25

From: Guofeng Yue <yueguofeng(a)h-partners.com> When creating SRQ/XRC QP in TD lock-free mode, pass in ctx->pad instead of ctx->pd, otherwise the lock-free won't work. Besides, use ctx->pad directly when creating QP/SRQ since pad is designed to be interchangeable with the usual pd. When lock-free mode is disabled, pad is the exactly the usual pd. Fixes: b6f957f6bc6c ("Perftest: Add support for TD lock-free mode") Signed-off-by: Guofeng Yue <yueguofeng(a)h-partners.com> Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> --- src/perftest_resources.c | 21 ++++++++++++--------- src/perftest_resources.h | 2 +- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/src/perftest_resources.c b/src/perftest_resources.c index 2d2e9f2..6543bc7 100755 --- a/src/perftest_resources.c +++ b/src/perftest_resources.c @@ -737,7 +737,8 @@ static int ctx_xrc_srq_create(struct pingpong_context *ctx, else srq_init_attr.cq = ctx->send_cq; - srq_init_attr.pd = ctx->pd; + srq_init_attr.pd = ctx->pad; + ctx->srq = ibv_create_srq_ex(ctx->context, &srq_init_attr); if (ctx->srq == NULL) { fprintf(stderr, "Couldn't open XRC SRQ\n"); @@ -780,7 +781,8 @@ static struct ibv_qp *ctx_xrc_qp_create(struct pingpong_context *ctx, qp_init_attr.cap.max_send_wr = user_param->tx_depth; qp_init_attr.cap.max_send_sge = 1; qp_init_attr.comp_mask = IBV_QP_INIT_ATTR_PD; - qp_init_attr.pd = ctx->pd; + qp_init_attr.pd = ctx->pad; + #ifdef HAVE_IBV_WR_API if (!user_param->use_old_post_send) qp_init_attr.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS; @@ -2037,9 +2039,14 @@ int ctx_init(struct pingpong_context *ctx, struct perftest_parameters *user_para fprintf(stderr, "Couldn't allocate PAD\n"); goto td; } + } else { + #endif + ctx->pad = ctx->pd; + #ifdef HAVE_TD_API } #endif + #ifdef HAVE_AES_XTS if(user_param->aes_xts){ struct mlx5dv_dek_init_attr dek_attr = {}; @@ -2127,7 +2134,7 @@ int ctx_init(struct pingpong_context *ctx, struct perftest_parameters *user_para attr.comp_mask = IBV_SRQ_INIT_ATTR_TYPE | IBV_SRQ_INIT_ATTR_PD; attr.attr.max_wr = user_param->rx_depth; attr.attr.max_sge = 1; - attr.pd = ctx->pd; + attr.pd = ctx->pad; attr.srq_type = IBV_SRQT_BASIC; ctx->srq = ibv_create_srq_ex(ctx->context, &attr); @@ -2148,7 +2155,7 @@ int ctx_init(struct pingpong_context *ctx, struct perftest_parameters *user_para .max_sge = 1 } }; - ctx->srq = ibv_create_srq(ctx->pd, &attr); + ctx->srq = ibv_create_srq(ctx->pad, &attr); if (!ctx->srq) { fprintf(stderr, "Couldn't create SRQ\n"); goto xrcd; @@ -2406,11 +2413,7 @@ struct ibv_qp* ctx_qp_create(struct pingpong_context *ctx, attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_READ; } - #ifdef HAVE_TD_API - attr_ex.pd = user_param->no_lock ? ctx->pad : ctx->pd; - #else - attr_ex.pd = ctx->pd; - #endif + attr_ex.pd = ctx->pad; attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS | IBV_QP_INIT_ATTR_PD; attr_ex.send_cq = attr.send_cq; diff --git a/src/perftest_resources.h b/src/perftest_resources.h index b28e136..25a7438 100755 --- a/src/perftest_resources.h +++ b/src/perftest_resources.h @@ -183,8 +183,8 @@ struct pingpong_context { struct ibv_pd *pd; #ifdef HAVE_TD_API struct ibv_td *td; - struct ibv_pd *pad; #endif + struct ibv_pd *pad; struct ibv_mr **mr; struct ibv_mr *null_mr; struct ibv_cq *send_cq; -- 2.33.0

1 0

libhns: Add initial support for HNS LTTng tracing
by Junxian Huang 15 Apr '25

15 Apr '25

From: wenglianfa <wenglianfa(a)huawei.com> Add initial support for HNS LTTng tracing. Add a new libhns_trace so that the original libhns does not need to have a dependency on LTTng. Signed-off-by: wenglianfa <wenglianfa(a)huawei.com> Signed-off-by: Junxian Huang <huangjunxian6(a)hisilicon.com> --- providers/hns/CMakeLists.txt | 12 ++++++ providers/hns/hns_roce_u_hw_v2.c | 3 ++ providers/hns/hns_roce_u_trace.c | 36 ++++++++++++++++++ providers/hns/hns_roce_u_trace.h | 63 ++++++++++++++++++++++++++++++++ providers/hns/libhns_trace.map | 5 +++ 5 files changed, 119 insertions(+) create mode 100644 providers/hns/hns_roce_u_trace.c create mode 100644 providers/hns/hns_roce_u_trace.h create mode 100644 providers/hns/libhns_trace.map diff --git a/providers/hns/CMakeLists.txt b/providers/hns/CMakeLists.txt index 58139ae2b..84ddd6912 100644 --- a/providers/hns/CMakeLists.txt +++ b/providers/hns/CMakeLists.txt @@ -1,3 +1,10 @@ +if (ENABLE_LTTNG AND LTTNGUST_FOUND) + rdma_shared_provider(hns_trace libhns_trace.map + 1 1.0.${PACKAGE_VERSION} + hns_roce_u_trace.c +) +endif() + rdma_shared_provider(hns libhns.map 1 1.0.${PACKAGE_VERSION} hns_roce_u.c @@ -12,3 +19,8 @@ publish_headers(infiniband ) rdma_pkg_config("hns" "libibverbs" "${CMAKE_THREAD_LIBS_INIT}") + +if (ENABLE_LTTNG AND LTTNGUST_FOUND) + target_include_directories(hns_trace PUBLIC ".") + target_link_libraries(hns_trace LINK_PRIVATE LTTng::UST) +endif() diff --git a/providers/hns/hns_roce_u_hw_v2.c b/providers/hns/hns_roce_u_hw_v2.c index c02d2c1b8..b4b335fba 100644 --- a/providers/hns/hns_roce_u_hw_v2.c +++ b/providers/hns/hns_roce_u_hw_v2.c @@ -39,6 +39,9 @@ #include "hns_roce_u_db.h" #include "hns_roce_u_hw_v2.h" +#define LTTNG_UST_TRACEPOINT_DEFINE +#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE + #define HR_IBV_OPC_MAP(ib_key, hr_key) \ [IBV_WR_ ## ib_key] = HNS_ROCE_WQE_OP_ ## hr_key diff --git a/providers/hns/hns_roce_u_trace.c b/providers/hns/hns_roce_u_trace.c new file mode 100644 index 000000000..65aa45ca3 --- /dev/null +++ b/providers/hns/hns_roce_u_trace.c @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +/* + * Copyright (c) 2025 Hisilicon Limited. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#define LTTNG_UST_TRACEPOINT_CREATE_PROBES + +#include "hns_roce_u_trace.h" diff --git a/providers/hns/hns_roce_u_trace.h b/providers/hns/hns_roce_u_trace.h new file mode 100644 index 000000000..74d33344d --- /dev/null +++ b/providers/hns/hns_roce_u_trace.h @@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ +/* + * Copyright (c) 2025 Hisilicon Limited. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#if defined(LTTNG_ENABLED) + +#undef LTTNG_UST_TRACEPOINT_PROVIDER +#define LTTNG_UST_TRACEPOINT_PROVIDER rdma_core_hns + +#undef LTTNG_UST_TRACEPOINT_INCLUDE +#define LTTNG_UST_TRACEPOINT_INCLUDE "hns_roce_u_trace.h" + +#if !defined(__HNS_TRACE_H__) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ) +#define __HNS_TRACE_H__ + +#include <lttng/tracepoint.h> +#include <infiniband/verbs.h> + +#define rdma_tracepoint(arg...) lttng_ust_tracepoint(arg) + +#endif /* __HNS_TRACE_H__*/ + +#include <lttng/tracepoint-event.h> + +#else + +#ifndef __HNS_TRACE_H__ +#define __HNS_TRACE_H__ + +#define rdma_tracepoint(arg...) + +#endif /* __HNS_TRACE_H__*/ + +#endif /* defined(LTTNG_ENABLED) */ diff --git a/providers/hns/libhns_trace.map b/providers/hns/libhns_trace.map new file mode 100644 index 000000000..e74bc7479 --- /dev/null +++ b/providers/hns/libhns_trace.map @@ -0,0 +1,5 @@ +/* Export symbols should be added below according to + Documentation/versioning.md document. */ +HNS_TRACE_1.0 { + global:*; +}; \ No newline at end of file -- 2.33.0

1 0