RDMA READs can cause cmds to be submitted to the target
layer in a different order than they were received
from the host. Normally this is fine, but not for
fused commands.
So track fused commands as they reach
nvmf_rdma_request_process(). If we find a pair of
sequential commands that don't have valid FUSED settings
(i.e. NONE/SECOND, FIRST/NONE, FIRST/FIRST), we mark
the requests as "fused_failed" and will later fail them
just before they would be normally sent to the target
layer.
When we do find a pair of valid fused commands (FIRST
followed by SECOND), we will wait until both are
READY_TO_EXECUTE, and then submit them to the target
layer consecutively.
This fixes issue #2428 for RDMA transport.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I01ebf90e17761499fb6601456811f442dc2a2950
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12018
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The process of matching qpair to poll group is split into
two distinct parts that occur on different threads.
See spdk_nvmf_tgt_new_qpair().
This results in a race condition for TCP between spdk_sock_map_lookup()
and spdk_sock_map_insert(), which are called in spdk_nvmf_get_optimal_poll_group()
and spdk_nvmf_poll_group_add() respectively.
Fixes#2113
This patch picks a hint from nvmf_tcp for next poll group,
which is then passed down to spdk_sock_map_lookup().
When matching placement_id exists, but does not have
a poll group assigned - the hint will be used.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I4abde2bc9c39225c9f5dd7c3654fa2639bb0a27f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10271
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The rdma buffer for stripping DIF metadata is added. CPU strips the DIF
metadata and copies it to the rdma buffer, improving the rdma write
bandwith. The network bandwidth during 4KB random read test is increased
from 79 Gbps to 99 Gbps, the IOPS is increased from 2075K to 2637K.
Fixes issue #2418
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Change-Id: If1c31256f0390f31d396812fa33cd650bf52b336
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11861
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Extract the code for DIF from nvmf_rdma_fill_wr_sgl() into
nvmf_rdma_fill_wr_sgl_with_dif().
Then clean up nvmf_rdma_request_fill_iovs() and
nvmf_rdma_request_fill_iovs_multi_sgl().
Additionally, this patch has a bug fix. nvmf_rdma_fill_wr_sgl_with_dif()
returned false if spdk_rdma_get_translation() failed. However, the
type of return value of nvmf_rdma_fill_wr_sgl_with_dif() is not bool
but int. The boolean false is 0 in integer. Hence in this case,
nvmf_rdma_fill_wr_sgl_with_dif() returned 0 even if it failed.
Change nvmf_rdma_fill_wr_sgl_with_dif() to return rc as is if
spdk_rdma_get_translation() returns non-zero rc.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I71cc186458bfe8863964ab68e2d014c495312cd3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11965
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: fengchunsong <fengchunsong@huawei.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
To help grep, use a standard sqid:%d style format for identifying queue
IDs.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ib82c81939f85f9beb333a4db10d006524522a1d9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11822
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
As a general utility function, move it up with the others.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I32881c01afd9819c889730d7c09163c95fbb827e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11790
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
For each queue, track its doorbell location individually, rather than
needlessly recalculating it every time we look up the doorbell value.
This will also greatly simplify shadow doorbell support.
Co-authored-by: Andreas Economides <andreas.economides@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I6882d2f92ee2f2b2b90c54ee14e5f6b41ecca85d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11789
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
R2Ts can cause cmds to be submitted to the target
layer in a different order than they were received
from the host. Normally this is fine, but not for
fused commands.
So track fused commands as they reach
nvmf_tcp_req_process(). If we find a pair of sequential
commands that don't have valid FUSED settings (i.e.
NONE/SECOND, FIRST/NONE, FIRST/FIRST), we mark the
requests as "fused_failed" and will later fail them
just before they would be normally sent to the target
layer.
When we do find a pair of valid fused commands (FIRST
followed by SECOND), we will wait until both are
READY_TO_EXECUTE, and then submit them to the target
layer consecutively.
This fixes issue #2428 for TCP transport.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8a9e13690ecb16429df68ae41b16b439a0913e4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12017
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
In vfio-user transport, whenever one IO is completed, it will trigger
an interrupt to guest machine. This cost quite some overhead. This patch
adds an adaptive irq feature to reduce interrupt overhead and boost
performance.
Signed-off-by: Rui Chang <rui.chang@arm.com>
Change-Id: I585be072231a934fa2e4fdf2439405de95151381
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11840
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Rename ->doorbells to ->bar0_doorbells. This will help avoid confusion
later with shadow doorbells.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Id432938cfeb3033e79dc6e1b491dad964227687a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11788
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
NVMe over PCIe Transport Spec 3.1.2:
The host should not read the doorbell registers.
Explicitly refuse these reads.
Co-authored-by: John Levon <john.levon@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ie64fd5ce7988ee86c612b3ef6046a57af467e266
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11787
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Refactor controller reset a little bit for cleaner code.
Co-authored-by: John Levon <john.levon@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I2b3323005d4e788ffe980d41c349702828886981
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11786
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
If we're in interrupt mode and live migrating a guest, there is a window
where the I/O queues haven't been set up but the device is in running
state, during which the guest might write to a doorbell. This doorbell
write will go unnoticed. This patch ensures that we re-check the
doorbells after an I/O queue has been set up.
Fixes#2410
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I161d2a0e7ab3065022b2bccbe17f019640cceeba
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11809
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Make sure cq->group is set even in interrupt mode.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I9f722917a8e3aebbd5d66648a3909f795897ec1a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11997
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We were accidentally reading the spdk_nvme_status from the CPL; on one
test, this was contributing to 2% of cache misses.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I7c967690458a183799f8d835360800d3094c3131
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11849
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Trivial chore, cleanup multiple branches for subsystem->flags.ana_reporting
that can be collapsed wtihin spdk_nvmf_ctrlr_identify_ctrlr, which makes the
code easier to read.
Signed-off-by: Jon Kohler <jon@nutanix.com>
Change-Id: Icb2064f078d730d1f28f299582dc46254c7f6df3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11951
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When rq is used, rnr_retry_count is not set. As a result, the initiator
cannot connect to the target after some error. Set the rnr_retry_count
to avoid infinite retries.
Fix issue #2429
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Change-Id: Iab03ea5ff10f5a9759c41f26589731aa92ef2f56
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11946
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
When generating a discovery log page, we will add entries
for the discovery subsystem for all listeners except the
one associated with the controller that generated the
log page command. We do this comparison using
spdk_nvme_transport_id_compare().
But this function compares the subnqn of the trid, and
the subnqn is not set in either of the trids that we
are comparing.
The listener's trid always has an empty subnqn, but
the source trid has an uninitialized subnqn when
we do the comparison. This means that sometimes the
subnqn may be empty (which always happens in debug
builds) but sometimes may contain garbage. This
means that sometimes an entry would be added to the
log, even for the trid of the discovery controller
that generated the command (meaning the discovery
controller would end up referring to itself which
is not allowed).
There is an even more subtle issue with this. If the
host reads just the log page header, the nvmf target
generates the entire log page, and just returns the
header contents. Let's say in this case, the source
trid has an empty subnqn, so we don't generate an entry
for it, and report numrec = X and genctr = Y. Then
the host reads the X log page entries. But now the
source trid is garbage, so a discovery log page entry
is returned, replacing one of the "real" log page
entries. And since genctr didn't change, the host
thinks the data is all valid, meaning there's a log
page entry for an NVM subsystem that ends up getting
dropped.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I96cfc566ddaf17153aec089bf3d9b3480bec3e4b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11933
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
If 0 - UINT32_MAX or UINT32_MAX - 0 is substituted into a int variable,
we cannot get any expected result.
Fix the bug and add unit test case to verify the fix.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Iad2ea681ad8ad234e70c7310b58785a999612156
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11837
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
To speed up qpair lookup, an idea to use the DPDK hash library was
proposed. But we have some concerns for the idea, we may see performance
regression for small configuration, lib/nvmf is depending on the DPDK
feature, and so on.
On the other hand, RB tree is simple to use and is not likely to have
negative side effect.
To use RB tree easily, cache QP num into struct spdk_nvmf_rdma_qpair.
Change-Id: Id4ea4e75692f86335b3e984c3fc128e187f22c72
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11798
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
nvmf_rdma_poll_group_add() added a rqpair to the poll_group->qpairs list
before its initialization.
If we change the poll_group->qpairs list from linked list to RB tree,
the qpair can be added to the poll_group_qpairs list only after its
initialization completes because QP number is necessary.
As a preparation, change nvmf_rdma_poll_group_add() to add a rqpair to
the poll_group->qpairs list after its initialization succeeds.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I42525f68c5efbd445f96da8f4f63d31f06f739e3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11800
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
When vfu_run_ctx() returns EBUSY, due to an ongoing quiesce, we did no
work, so should return SPDK_POLLER_IDLE.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I5953ab652f0adf22df81c94c4bece507833cece4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11810
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We will use nvmf library CSTS.CFS instead so that the client can
get this error status.
Change-Id: I42c248a7333d1f9c940bb29135c887a61c906bd4
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11676
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
No logic code changes, we will define a nvmf controller migration
data structure in public header file, this is a preparation patch.
Change-Id: I0a98581b320ac9b12b7e4bf838222c5075168e64
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11493
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
To fully support interrupt mode for the vfio-user transport, we need to
arrange to wake up in one of two conditions:
- we receive a vfio-user message on the socket
- a client writes to one of our BARs
In response, we can process any pending vfio-user messages, as well as
poll the actual queue pairs.
As there is no way for a client-mapped BAR write to cause us to wake up,
interrupt mode can only work when mappable BAR0 is disabled. In that
case, each BAR write becomes a vfio-user message, and can thus be
handled by registering the libvfio-user socket fd with SPDK.
For the poll group poller, we enable interrupt mode for it during the
vfio-user ->poll_group_create() callback; this only works in the case
that no other transports without interrupt mode support are sharing that
poll group.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ic784d01078b2006edb27892fc0613f934eca930b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10722
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Community-CI: Mellanox Build Bot
When generating the discovery log page, add entries for
the discovery subsystem, skipping the listener associated
with the command generating the log page.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id6a14a7d5cdce483f8f3c2eff1b4ededd40bc029
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11542
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Currently we accept connections to the discovery
subsystem on any listener that has been added to
any subsystem (not just the discovery subsystem).
This is not proper behavior, especially for TCP. TCP
defines port 8009 (not 4420) as the discovery port,
so the current behavior means that if NVM subsystems
are listening on port 4420, then the discovery
subsystem by default is listening there too.
For now, continue to allow connections, but print
a warning message when someone connects to the
discovery subsystem on a listener trid that wasn't
previously added.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I734bc49d1a21b2edfb675aef4b8551e2d0ccd4d7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11539
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
One simple fix for nvmf_vfio_user_qpair_abort_request().
Current implementation mixed up request of abort cmd and the request
to abort, which cause problems.
Signed-off-by: Rui Chang <rui.chang@arm.com>
Change-Id: Ia0db9aa738e372789fc502ef877fd1c841c0a2e3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11711
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We do the null check for 'fc_req', but already dereferenced it
before the check. Swap their position to avoid null dereference.
Fixes issue #2395.
Signed-off-by: Weiguo Li <liwg06@foxmail.com>
Change-Id: I33b9e6b51b54f6ada9c072cf7ab0acda2622472f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11721
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
A common pattern is:
if (foo->thread == spdk_get_thread())
cb(arg);
else
spdk_thread_send_msg(foo->thread, cb, arg);
for cases where it's important the callback runs on a particular thread,
but it doesn't matter if it's synchronous or asynchronous.
Add a new API to support this pattern, and convert over the current
instances.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Idfbf77c02c9321c52e07181ffd8b0c437e1ab335
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11503
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
We can ask libvfio-user for the listening socket fd, and register that
for SPDK interrupt handling.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I8d0ba7a86403f2d0170b9359480f1fefc1036557
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10721
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
The accept poller only needs to run when vfu_attach_ctx() makes sense:
in other words, when we don't have a controller created.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Icef4e6184c9ae6d7951d015530a05132c4ba6994
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10720
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Now vfio-user owns its the accept poller itself, there's no reason to
loop across all endpoints: instead, the lifetime of the accept poller is
better matched by creating it in the ->listen() transport callback.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ia92e29b1cee263f1461f640cfdd27cdb674848fb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10719
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
For the benefit of forthcoming vfio-user changes, register the poll
group poller prior to calling the transport create callback, and pass in
a pointer to the poll group itself.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Idbc24126c9d46f8162e4ded07c5a0ecf074fc7dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10718
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The above values were added with shared CQ feature, and they
are left when restore the CQs in destination VM.
Change-Id: Ib1f28dad833da31e571eb2e2f0b5c81f0bf05a3b
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11419
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The CSTS register in source VM was left to migrate.
Fix#2362.
Change-Id: Ieef028578d2897b27a6ff594b16801462eb1b75e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11418
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The dif_enable is not cleared in _nvmf_rdma_request_free. When the
rdma_req is used again, the dif_enable is true while
dif.dif_ctx->block_size is zero. As a result, an infinite loop occurs in
nvmf_rdma_fill_wr_sgl.
Fix issue: #2380
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Change-Id: Ic179855f7b257e39ed4a5f6705fbc9dea64210ae
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11646
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
This was a parameter on the nvmf_create_transport
RPC, and was replaced with max_io_qpairs_per_ctrlr to
reduce confusion on whether this number included the
admin queue or not.
nvmf_vhost test was using this deprecated parameter.
Change it to use -m (max_io_qpairs_per_ctrlr)
instead. '-p 4' would have been evaluated as 1 admin
queue + 3 I/O queues, but it's likely the intent
was for 4 I/O queues. This is a perfect example of
why this parameter was deprecated.
For reference, this was deprecated in June 2020,
commit 1551197db.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4364fc0a76c9993b376932b6eea243d7cefca9cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11543
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
The controller data structure may be freed before subsystem resume done
callback, we can take endpoint as the input parameter to avoid this issue.
AddressSanitizer: heap-use-after-free on address 0x625000046100 at pc 0x00000082818f bp 0x7fff7b09bd10 sp 0x7fff7b09bd00
READ of size 8 at 0x625000046100 thread T0 (reactor_0)
#0 0x82818e in vfio_user_dev_quiesce_resume_done /spdk/lib/nvmf/vfio_user.c:2147
#1 0x782cc0 in subsystem_state_change_done /spdk/lib/nvmf/subsystem.c:634
#2 0xad047b in _call_completion /spdk/lib/thread/thread.c:2344
#3 0xabc48d in msg_queue_run_batch /spdk/lib/thread/thread.c:710
#4 0xac0670 in thread_poll /spdk/lib/thread/thread.c:926
#5 0xac0ead in spdk_thread_poll /spdk/lib/thread/thread.c:986
#6 0x9a5b4f in _reactor_run /spdk/lib/event/reactor.c:920
#7 0x9a6442 in reactor_run /spdk/lib/event/reactor.c:958
#8 0x9a717c in spdk_reactors_start /spdk/lib/event/reactor.c:1060
#9 0x99884a in spdk_app_start /spdk/lib/event/app.c:643
#10 0x407e82 in main /spdk/app/nvmf_tgt/nvmf_main.c:75
#11 0x7f822095ff42 in __libc_start_main (/lib64/libc.so.6+0x23f42)
#12 0x407abd in _start (/spdk/build/bin/nvmf_tgt+0x407abd)
0x625000046100 is located 0 bytes inside of 8320-byte region [0x625000046100,0x625000048180)
freed by thread T0 (reactor_0) here:
#0 0x7f82219ff91f in __interceptor_free (/lib64/libasan.so.5+0x10d91f)
#1 0x837059 in _free_ctrlr /spdk/lib/nvmf/vfio_user.c:2976
#2 0x837327 in free_ctrlr /spdk/lib/nvmf/vfio_user.c:2996
#3 0x843541 in nvmf_vfio_user_close_qpair /spdk/lib/nvmf/vfio_user.c:3742
#4 0x7d1d91 in nvmf_transport_qpair_fini /spdk/lib/nvmf/transport.c:604
#5 0x7ad922 in _nvmf_qpair_destroy /spdk/lib/nvmf/nvmf.c:1055
#6 0x761362 in nvmf_qpair_request_cleanup /spdk/lib/nvmf/ctrlr.c:4026
#7 0x761906 in spdk_nvmf_request_free /spdk/lib/nvmf/ctrlr.c:4041
#8 0x75a931 in nvmf_qpair_free_aer /spdk/lib/nvmf/ctrlr.c:3576
#9 0x7ae626 in spdk_nvmf_qpair_disconnect /spdk/lib/nvmf/nvmf.c:1127
#10 0x83db36 in _vfio_user_qpair_disconnect /spdk/lib/nvmf/vfio_user.c:3433
#11 0xabc48d in msg_queue_run_batch /spdk/lib/thread/thread.c:710
#12 0xac0670 in thread_poll /spdk/lib/thread/thread.c:926
#13 0xac0ead in spdk_thread_poll /spdk/lib/thread/thread.c:986
#14 0x9a5b4f in _reactor_run /spdk/lib/event/reactor.c:920
#15 0x9a6442 in reactor_run /spdk/lib/event/reactor.c:958
#16 0x9a717c in spdk_reactors_start /spdk/lib/event/reactor.c:1060
#17 0x99884a in spdk_app_start /spdk/lib/event/app.c:643
#18 0x407e82 in main /spdk/app/nvmf_tgt/nvmf_main.c:75
#19 0x7f822095ff42 in __libc_start_main (/lib64/libc.so.6+0x23f42)
previously allocated by thread T0 (reactor_0) here:
#0 0x7f82219fff16 in __interceptor_calloc (/lib64/libasan.so.5+0x10df16)
#1 0x837413 in nvmf_vfio_user_create_ctrlr /spdk/lib/nvmf/vfio_user.c:3010
#2 0x83bc68 in nvmf_vfio_user_accept /spdk/lib/nvmf/vfio_user.c:3313
#3 0xabfbd8 in thread_execute_timed_poller /spdk/lib/thread/thread.c:872
#4 0xac0c75 in thread_poll /spdk/lib/thread/thread.c:960
#5 0xac0ead in spdk_thread_poll /spdk/lib/thread/thread.c:986
#6 0x9a5b4f in _reactor_run /spdk/lib/event/reactor.c:920
#7 0x9a6442 in reactor_run /spdk/lib/event/reactor.c:958
#8 0x9a717c in spdk_reactors_start /spdk/lib/event/reactor.c:1060
#9 0x99884a in spdk_app_start /spdk/lib/event/app.c:643
#10 0x407e82 in main /spdk/app/nvmf_tgt/nvmf_main.c:75
#11 0x7f822095ff42 in __libc_start_main (/lib64/libc.so.6+0x23f42)
SUMMARY: AddressSanitizer: heap-use-after-free /spdk/lib/nvmf/vfio_user.c:2147 in vfio_user_dev_quiesce_resume_done
Change-Id: Icf5e5b360b9107a3c5eb960ae59b7fe10ace1c66
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11420
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
if ctrlr->listener was NULL, nvmf_ctrlr_get_ana_state() returned
inaccessible even if ana_reporting was disabled. Then the corresponding
initiator received unexpected ANA error and could not process it
appropriately.
Change nvmf_ctrlr_get_ana_state() to return optimized always if
ana_reporting is disabled.
Additionally, check if ctrlr->listener is not NULL before calling
SPDK_DTRACE_PROBE3().
Fixes#2335
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ib2376694cf89d85ec5687fba7e87439f494f30b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11402
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
it shall be executed on ctrlr's thread not subsystem's
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I58c60525191085d3d6a583862ba5d71ea90940c7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11105
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
nvmf_vfio_user_sq->size and ->qsize both hold the number of entries in
the queue; merge them.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I6c7c2984cbdf90079eec9222e1acbedb92207308
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11297
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Since we already allocate ->sg on a per-request basis now, drop the
->reqs_internal allocation in favour of allocating individual requests
and placing them on the ->free_reqs queue, which simplifies the need to
track the array. For request abort, we'll use the ->outstanding request
list, now we have it, to find the victim request.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I25227ccd2ab7e00c8a2e7b2e2af2dc3b073584cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11427
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
