nvme/rdma: Allows to use PCI Express Relaxed Ordering

This fix allows to use relaxed ordering feature where it is
supported. libibversb checks with the driver if relaxed ordering
access flag is supported and ignores it if not.

Experiments show that set by default it doesn't spoil performance but
allows to reach desired one on AMD EPYC systems. For example fio read
test (ConnectX-6, AMD EPYC 7763, two jobs, queue depth 32, block size
32K) can starve down to 6-7 GiB/s without it. Enabling this option
allows to get bandwidth more than 21 GiB/s.

Change-Id: I5983aed5d1f38ee7bec9c310597731c9a6a329da
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14885
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

lib/rdma/common.c

@@ -68,6 +68,9 @@ rdma_mem_notify(void *cb_ctx, struct spdk_mem_map *map,
 			default:
 				SPDK_UNREACHABLE();
 			}
+#ifdef IBV_ACCESS_OPTIONAL_FIRST
+			access_flags |= IBV_ACCESS_RELAXED_ORDERING;
+#endif
 			mr = ibv_reg_mr(pd, vaddr, size, access_flags);
 			if (mr == NULL) {
 				SPDK_ERRLOG("ibv_reg_mr() failed\n");