From c273513401873230aa4fdc3e9d0af5211e09943b Mon Sep 17 00:00:00 2001 From: Denis Nagorny Date: Thu, 6 Oct 2022 17:18:33 +0300 Subject: [PATCH] nvme/rdma: Allows to use PCI Express Relaxed Ordering This fix allows to use relaxed ordering feature where it is supported. libibversb checks with the driver if relaxed ordering access flag is supported and ignores it if not. Experiments show that set by default it doesn't spoil performance but allows to reach desired one on AMD EPYC systems. For example fio read test (ConnectX-6, AMD EPYC 7763, two jobs, queue depth 32, block size 32K) can starve down to 6-7 GiB/s without it. Enabling this option allows to get bandwidth more than 21 GiB/s. Change-Id: I5983aed5d1f38ee7bec9c310597731c9a6a329da Signed-off-by: Denis Nagorny Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14885 Tested-by: SPDK CI Jenkins Community-CI: Mellanox Build Bot Reviewed-by: Jim Harris Reviewed-by: Ben Walker --- lib/rdma/common.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/rdma/common.c b/lib/rdma/common.c index 9e5604f68..ff8a63dc1 100644 --- a/lib/rdma/common.c +++ b/lib/rdma/common.c @@ -68,6 +68,9 @@ rdma_mem_notify(void *cb_ctx, struct spdk_mem_map *map, default: SPDK_UNREACHABLE(); } +#ifdef IBV_ACCESS_OPTIONAL_FIRST + access_flags |= IBV_ACCESS_RELAXED_ORDERING; +#endif mr = ibv_reg_mr(pd, vaddr, size, access_flags); if (mr == NULL) { SPDK_ERRLOG("ibv_reg_mr() failed\n");