Commit Graph

274 Commits

Author SHA1 Message Date
Seth Howell
1039254319 nvme/rdma: add cq resizing.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I6350d76b8c1e778c18e693b2dfbb10dd36b3e3d8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1927
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-06-04 07:20:16 +00:00
Seth Howell
67b0dcfe29 nvme_rdma: add tracking for rdma objects in qpair.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I0b45aed21dc649888bb9d93c5937fb553f35eb27
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2568
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-06-04 07:20:16 +00:00
Seth Howell
8bef6f0bdf lib/nvme: rdma poll group with shared cq.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ifde29f633f09cccbebfdcde5ab2f96d9590449f1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1167
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-06-04 07:20:16 +00:00
Seth Howell
1a9c19a954 lib/nvme: remove spdk prefix from internal headers.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Iccde5860b83217163428ff504cba87a1cf209720
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2444
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2020-06-01 13:07:30 +00:00
Seth Howell
6d18ea425b lib/nvme: force qpair disconnect before aborting rdma requests.
This is needed for shared completion queues which can still give us
successful completions on aborted requests if the qpair hasn't been
disconnected.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I85cf1a81ef563d8c02d684b09d2f7ad5008e38cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1961
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-06-01 09:22:05 +00:00
Seth Howell
b4e060b560 lib/nvme: check that req is not null in RDMA.
When a request has been aborted, it's possible to get a completion
for an rdma request but the rdma_req->req object has already been
cleared to NULL.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I5f7b1b96ff4be8c436aae9a7e2a7c9927d04e627
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1960
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-06-01 09:22:05 +00:00
Shuhei Matsumoto
f21f51bd81 lib/nvme: Remove inclusion of SPDK event library
Remove inclusion of spdk/event.h and spdk_internal/event.h from
SPDK NVMe library. Their dependency had been removed before.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Ide3a0902b1cebb9c9033ade45d7488622e38696c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2688
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-06-01 09:20:41 +00:00
Seth Howell
63732d8880 lib/nvme: split cq completion processing to its own function.
This helps create a separation between processing a qpair and processing
a completion queue which can be shared across multiple qpairs.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I111dd16ec4327854f232988a96891a65813f00e1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1166
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-05-28 07:13:44 +00:00
zkhatami88
fe3fab26bf nvme/rdma: Using hooks in reg mr
Signed-off-by: zkhatami88 <z.khatami88@gmail.com>
Change-Id: I9493fe82b5b758c0092d20ef18b79d652fefed85
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1905
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-05-28 07:11:39 +00:00
Seth Howell
fadfef63d1 lib/nvme: provide mechanism for tracking request completions
Add wrappers around the request and response values and track
those using the wr_id value.
This will come in handy when we start doing poll group based
completion processing.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Iaff75b03e41d49f53e55e0ce65d384567988fc9d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1165
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-05-21 09:21:27 +00:00
Alexey Marchuk
9b86f31a38 nvme/rdma: Handle failed send/recv as a fatal error
Do not make attempt to resubmit failed send/recv WR, instead
report and error to the upper layer (in case of new request) or
fail a qpair (in case of active polling).

In the case of failed ibv_post_send and disabled `delay_cmd_submit`
nvme_rdma_qpair_submit_request returns an error to the caller.
The caller completes failed request but RDMA layer still keeps
it in a send queue. Later RDMA layer can send the corresponding
WR and notify the upper layer about the completion of the request
for the second time.

Change-Id: I1260f215b8523d39157a5cc3fda39cd4bd87c8ec
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1662
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-05-20 12:03:50 +00:00
Alexey Marchuk
8c6a345534 nvme/rdma: Use RDMA provider API to send WRs
Change-Id: I3dc87751d250da84d988b1c7a9c57112b5bd10b0
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1661
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-05-20 12:03:50 +00:00
Alexey Marchuk
daee62a05b rdma: Add mlx5_dv RDMA provider
The new RDMA provider can be enabled by passing
--with-rdma=mlx5_dv parameter to configure script
This provider uses "externally created qpair"
functionality of rdma cm - it must move a qpair
to RTS state manually

Change-Id: I72484f6edd1f4dad15430e2c8d36b65d1975e8a2
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1658
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2020-05-20 12:03:50 +00:00
Alexey Marchuk
63c8cea783 rdma: Add API function to disconnect qpair
This is a wrapper over RDMA CM rdma_disconnect function
The wrapper is needed since in Mellanox Direct Verbs
(aka DV) we must move qpair to error state manually
before calling rdma_disconnect

Change-Id: Ia8623c6989e7679591f2da56bafa7f4262eeebf9
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1975
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
2020-05-20 12:03:50 +00:00
Alexey Marchuk
b4a9d7d318 nvme/rdma: Use RDMA provider API to create/destroy qpair
This patch adds use of RDMA provider API to NVMEoF initiator.
Makefiles have been updated with new RDMA lib dependency

Change-Id: Ieaefeb12ee9681d3db2b618c5cf0c54dc52230af
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1657
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-05-20 12:03:50 +00:00
WANGHAILIANG
023e3624e7 lib/nvme: remove lkey and rkey's warnings in nvme_rdma.c
One of these warnings, such as:
/home/wanghailiang/spdk20200428/lib/nvme/nvme_rdma.c: In function ‘nvme_rdma_qpair_submit_request’:
/home/wanghailiang/spdk20200428/lib/nvme/nvme_rdma.c:1512:29: warning: ‘lkey’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  rdma_req->send_sgl[1].lkey = lkey;
                             ^
/home/wanghailiang/spdk20200428/lib/nvme/nvme_rdma.c:1480:11: note: ‘lkey’ was declared here
  uint32_t lkey;
           ^

Change-Id: I67b25cb62c7a0d5b298ebfe7d2673b73261040ef
Signed-off-by: WANGHAILIANG <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2197
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-05-07 10:44:02 +00:00
zkhatami88
58a8fe2eee nvme/rdma: When RDMA hooks exist, prefer spdk_zmalloc for internal
allocations

Signed-off-by: zkhatami88 <z.khatami88@gmail.com>
Change-Id: I7f810ee78fecca7eb8a4387f6d63e1a952966e57
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1593
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-05-05 08:03:39 +00:00
Seth Howell
bf0561f741 nvme/nvme_rdma: assign rctrlr in each qpair->ctrlr check
While in practice the qpair->ctrlr variable will not change within
the disconnect function, when the code is built without debug enabled,
gcc thinks that rctrlr may be uninitialized.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I355cd62f3a2baaba65d806e3746f615a0dc37f58
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2056
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-04-29 06:32:12 +00:00
Seth Howell
1b818a28b5 lib/nvme: add naive poll_group implementation for rdma.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I55bae6dddc887a95c3e37195fac821de5aa1ed89
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/631
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-04-24 16:36:03 +00:00
Seth Howell
fc86e792e4 lib/nvme: switch poll group to use connect/disconnect semantics.
This makes more sense within the context of the nvme driver and
helps us avoid the awkward situation of getting a failed_qp callback
on a qpair that simply hasn't been connected.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ibac83c87c514ddcf7bd360af10fab462ae011112
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1734
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-04-22 19:06:26 +00:00
Seth Howell
6189c0ceb7 lib/nvme: abort all requests when disconnecting a qpair.
By aborting all requests from every qpair when it is disconnected,
we can completely avoid having to abort requests when we enable the
qpair since nothing will be left enabled.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Iba3bd866405dd182b72285def0843c9809f6500e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1788
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-04-22 19:06:26 +00:00
Seth Howell
6338af34fc lib/nvme: handle qpair state in transport layer.
The state should be changed and checked by the transport
layer. All transports should follow the same list of steps
when disconnecting/reconnecting.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: If2647624345f2c70f78a20bba4e2206d2762f120
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1853
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-04-22 19:06:26 +00:00
Seth Howell
e1c9185005 lib/nvme: always call the transport disconnect function.
The qpair states should be maintained at the generic level.
Always going through the transport disconnect function is
one step in that direction.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I019b2b4a14fe192eff5293f918d633dde2c5400a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1851
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-04-22 19:06:26 +00:00
Seth Howell
9649ee09fa lib/nvme: rename NVME_QPAIR_DISABLED
This variable really indicates when a qpair is
no longer connected. So NVME_QPAIR_DISCONNECTED is
actually much more accurate.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ia480d94f795bb0d8f5b4eff9f2857d6fe8ea1b34
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1850
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-04-22 19:06:26 +00:00
Seth Howell
c3eac3435a nvme/rdma: send an rdma_disconnect during disconnect.
The rdma_disconnect call triggers an RDMA_CM_EVENT_DISCONNECTED
message on the target side. The hope is that the target side will
reply with the same message in a reasonable amount of time. If the
target doesn't have that mechanism implemented, print an error message
and continue with the process.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I164a3538714fa3adfc306ea0c88220ea710e7c39
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1879
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-04-20 07:40:31 +00:00
Alexey Marchuk
f11989385e nvme/rdma: Clean pointer to nvme_request
That is done to make sure that scenario described in github
issue #1292 won't happen

Change-Id: Ie2ad001da701e25ef984ae57da850fb84d51b734
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1771
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-04-14 11:33:39 +00:00
Alexey Marchuk
581e1bb576 nvme/rdma: Wait for completions of both RDMA RECV and SEND
In some situations we may get a completion of RDMA_RECV before
completion of RDMA_SEND and this can lead to a bug described in #1292
To avoid such situations we must complete nvme_request only when
we received both RMDA_RECV and RDMA_SEND completions.
Add a new field to spdk_nvme_rdma_req to store response idx -
it is used to complete nvme request when RDMA_RECV was completed
before RDMA_SEND
Repost RDMA_RECV when both RDMA_SEND and RDMA_RECV are completed
Side changes: change type of spdk_nvme_rdma_req::id to uint16_t,
repack struct nvme_rdma_qpair

Fixes #1292

Change-Id: Ie51fbbba425acf37c306c5af031479bc9de08955
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1770
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-04-14 11:33:39 +00:00
Seth Howell
c998c6c69e nvme: add API for qpair poll groups.
This API will allow us to simplify the polling mechanism for qpairs on a single
thread. It also will pave the way for doing transport specific aggregation of
qpair polling to increase performance.

The generic implementation is included. The transport specific calls
have yet to be implemented.

Change-Id: If07b4170b2be61e4690847c993ec3bde9560b0f0
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/579
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-04-07 08:38:40 +00:00
Alexey Marchuk
14425544a6 nvme/rdma: Factor out memory key translation
Add function nvme_rdma_get_key to get either lkey
or rkey, use it in request building functions

Change-Id: Ic9e3429e07a10b2dddc133b553e437359532401d
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1462
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-04-06 07:49:48 +00:00
Alexey Marchuk
d2510a56f3 nvme/rdma: Simplify nvme_rdma_req_init
Cache payload type and in-capsule data transfer support

Change-Id: Id40a6e86d1f29235ca3e0189d7fbcf19baa30ffe
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1461
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-04-06 07:49:48 +00:00
yidong0635
20564d423b nvme/nvme_rdma: Reduced the code lines.
Here destruct contrllers are in one function, and we can
remove the duplicated codes using goto.

It can save several lines of codes.

Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: Ibf3cb9fe2ea4bfc65d42603a7b13aaf575854580
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1638
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-04-03 06:31:52 +00:00
Shuhei Matsumoto
c3d0a83347 nvme/rdma: Move post WRs on send/recv queue after poll CQ
If nvme_rdma_qpair_submit_sends() returns -ENOMEM,
nvme_rdma_qpair_process_completions() returns immediately.
In this case, nvme_rdma_qpair_process_completions() does not
poll CQ.

However, nvme_rdma_qpair_process_completions() can poll CQ even
when there is no free slot in SQ.

Hence move nvme_rdma_qpair_submit_sends() and
nvme_rdma_qpair_submit_recvs() after the loop to poll CQ.

nvme_rdma_qpair_submit_sends() and nvme_rdma_qpair_submit_recvs()
output error log and so checking return code of them is not
necessary and is removed in this patch.

This fixes part of the github issue #1271.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Icf22879c69c3f84e6b1d91dc061b6f44237eedd1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1342
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-20 08:39:53 +00:00
Seth Howell
2248e52150 nvme/rdma: make sure we free resources in error path.
Not sure how we missed this.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: If920cb3a7708c33032e1da28c564d4c28ddafdf4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1122
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-17 08:23:50 +00:00
Seth Howell
3b99ee9929 lib/nvme: move connect directly into alloc_io_qpair.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Iadbada599764c7a2f4cdd4848a81a2fa39a89b46
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1120
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-17 08:23:50 +00:00
Seth Howell
7f82fb653d nvme/rdma: Move stale connection retries to connect call.
This gives us a more standard path in the create_io_qpair path. Eventually
this will allow us to bring the connection commands out to the generic layer
in alloc_io_qpair. Then we can split the calls to create and connect at the
generic level making it possible to add rdma qpairs to a poll group in a meaningful
way.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ib1b125f834c3c39a2b5050ff4a9bc4a053b95c99
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1119
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-17 08:23:50 +00:00
Seth Howell
1850842461 nvme_rdma: rearrange spdk_nvme_rdma_req.
This allows it to fit on three cachelines instead of four.

Change-Id: I2510b50ffcefb77fa570e738b2c6588749f30a00
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1143
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-17 08:23:50 +00:00
Jacek Kalwas
62e0342eac nvme: minor alignment in ctrlr construct for pcie and rdma
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I35db32e867f91269608c72dbb9290a7ed2e3f31d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1234
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-12 09:04:26 +00:00
Jacek Kalwas
daa8f941e4 nvme: extend ctrlr opts with admin queue size
Align rdma and tcp to respect opts. Reduce default number of entries
for admin queue so it becomes memory optimization.

Linux driver by default creates admin queue with 32 depth, there is no
good reason to enlarge that queue by default within SPDK NVMe driver.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I97ceea8f350c52313021a63190fb0980f604c48e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1110
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-03-12 09:04:18 +00:00
Seth Howell
f146bbe42d lib/nvme: move common connect code into transport shim
This gets rid of some duplicate lines of code.

Change-Id: I24d4864921f6030672f3640b33f88f37a9e8175a
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1136
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-06 10:29:21 +00:00
Alexey Marchuk
94966468ae nvme/rdma: Introduce transport_ack_timeout
Add transport_ack_timeout parameter to nvme controller opts.
This parameter allows to configure RDMA ACK timeout according
to the formula 4.096 * 2^(transport_ack_timeout) usec.
The parameter should be in range 0..31 where 0 means use
driver-specific default value.

Change-Id: I0c8a5a636aa9d816bda5c1ba58f56a00a585b060
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/502
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-02-27 10:16:00 +00:00
Changpeng Liu
8d6f48fbf8 nvme: set transport string before the probe based on transport type
Users may only set the transport type, but for the actual probe
process, the trstring field is mandatory, so set the trstring
based on transport type at first.  Also remove unnecessary
spdk_nvme_trid_populate_transport() call from each transport
module.

Fix #1228.

Change-Id: I2378065945cf725df4b1997293a737c101969e69
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1001
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-02-26 09:26:09 +00:00
Alexey Marchuk
c3ba9127d0 nvme: Store NVMEoF ioccsz and icdoff in ctrlr structure
This allows to avoid calculation of ioccsz bytes on each request
and removes access to "cold" ctrlr structures in data path.
Add UT to check validness of calculation

Change-Id: I55ceff99eb924156155e69a20f587a4f92b83f0b
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/519
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-02-17 10:06:30 +00:00
Ben Walker
5ac51a3214 nvme:Make ctrlr_alloc_cmb_io_buffer optional for transports
If the transport doesn't define one, don't call it.

Change-Id: I8b83132f9fc0accbd4faa8fa0fc17a6bd11e543e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/783
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-02-17 10:06:20 +00:00
Ben Walker
7dbe0e7c61 nvme: Remove nvme_transport_get_ctrlr_registers
Wasn't used.

Change-Id: I9812e24540f6d86f47d39091ea5fd9b7880b4413
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/735
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-02-12 12:07:16 +00:00
Ben Walker
54a022dda2 nvme: Don't DECLARE_TRANSPORT(rdma)
With the transport plugin system, this is no longer necessary.

Change-Id: Ia73878599658db84150603223ac811cb5a34ffba
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/713
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-02-12 12:07:16 +00:00
Alexey Marchuk
f1539c2820 nvme/rdma: Use transport_retry_count from controller opts
This allows to configure desired retry_count instead of using
hard coded value

Change-Id: I25c9601997ace916dfb735469a4b443c0cd2a96b
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482499
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-02-04 18:19:20 +00:00
Seth Howell
ca693eaba8 lib/nvme: fix cm event handling during rdma qpair shutdown.
In the event that we have more than one event outstanding for a qpair
at the time of destruction, we need to ack all of the events, Luckily
the synchronization is already there in the form of the ctrlr lock.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ib297598f2e28d9b9bd83e904f950795a61fa883a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479171
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-01-23 15:14:55 +00:00
Seth Howell
9436ab59ba nvme/rdma: inline buffers for all host to ctrlr ops
Not inlining all host to controller operations breaks the target within
the context of fused commands. This issue was discovered when enabling
the compare-and-write fused command. Only the write command buffer was
being inlined which caused the write to jump the compare in the
transport specific state machine on the target side before our fused
command checks in the generic code.

Change-Id: I9e52ae6160e01ffd36d20429ffc8459491c729ef
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482001
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-01-22 13:41:47 +00:00
Seth Howell
b2225ff593 lib/nvme: remove extra transport functions.
Now that we have a more flexible function table strategy for
transports, we can get rid of some of the wrapping we were doing
to match the macro definitions exactly.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I12c868babfa7bd27dc8ed5e86d35e179f8ec984f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478874
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-01-16 09:10:38 +00:00
Seth Howell
f6cf92a31f lib/nvme: make transport.c use fn tables.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ida58785784b4ed50393e1d43a9cd902de74a2eaa
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478873
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-01-16 09:10:38 +00:00