ivampiresp/Spdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nvme_rdma: Poll disconnect until completion if async mode is disabled

Browse Source 
nvme_rdma_ctrlr_disconnect_qpair() does not poll the qpair until it is
actually disconnected if it is in a poll group even if its async mode
is disabled. Hence, spdk_nvme_ctrlr_free_io_qpair() removes the qpair
from a poll group when it is being disconnected.

On the other hand, I/O qpair is destroyed after it is actually
disconnected.

When SRQ is enabled and used, a SRQ is destroyed if the corresponding
poller does not have any I/O qpair after an I/O qpair is removed from
the poller.

In particular, if we use spdk_nvme_ctrlr_free_io_qpair(), a SRQ is
destroyed before the corresponding I/O qpairs are destroyed.
Destroying a SRQ failed because it is still referenced by I/O qpairs.

This bug was found when running the SPDK NVMe perf tool with SRQ.

The reason was we had nvme_rdma_poll_group_process_completions() to call
disconnected_qpair_cb after the qpair is actually disconnected.
However, it is ensured that nvme_rdma_poll_group_process_completions()
calls disconnected_qpair_cb for any disconnected qpair.

Hence, remove a check if qpair->poll_group is not NULL from
nvme_rdma_ctrlr_disconnect_qpair() and update the comment.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I0fde0d827eec3280e1cc5a0fce34d163a6069bc4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14908
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
This commit is contained in:
Shuhei Matsumoto 2022-10-11 12:41:27 +09:00 committed by Tomasz Zawadzki
parent 3fcee8ddcc
commit 6a59daad2b
1 changed files with 5 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
lib/nvme/nvme_rdma.c
View File

@ -2018,14 +2018,12 @@ nvme_rdma_ctrlr_disconnect_qpair(struct spdk_nvme_ctrlr *ctrlr, struct spdk_nvme
_nvme_rdma_ctrlr_disconnect_qpair(ctrlr, qpair, nvme_rdma_qpair_disconnected);
/* If the qpair is in a poll group, disconnected_qpair_cb has to be called
* asynchronously after the qpair is actually disconnected. Hence let
* poll_group_process_completions() poll the qpair until then.
*
* If the qpair is not in a poll group, poll the qpair until it is actually
* disconnected here.
/* If the async mode is disabled, poll the qpair until it is actually disconnected.
* It is ensured that poll_group_process_completions() calls disconnected_qpair_cb
* for any disconnected qpair. Hence, we do not have to check if the qpair is in
* a poll group or not.
*/
if (qpair->async || qpair->poll_group != NULL) {
if (qpair->async) {
return;
}