The CSTS.SHN is changed only in shutting down the controller,
nvmf library already ensure that all the outstanding IOs will
be flushed before that, so we can remove this check here.
Change-Id: Ib93a256e986b7b2ec1da0fc7992feb3a02c1d657
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11674
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
After finishing migration in source VM, the subsystem is in
PAUSED state, the controller is dead for the source VM, we will
destroy the controller when disconnecting socket, but after that,
we should RESUME the subsystem so that it can be ready for the
next new client.
Fix issue #2363.
Change-Id: Icf0999b9085cebe8be4c8783e1a43bb13d4f7987
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11422
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The completion callback of `spdk_nvmf_subsystem_resume`
and `spdk_nvmf_subsystem_pause` can run in different
core other than the `vfu_ctx` core, this may lead to
race condition when changing controller's state. Here
we use a thread message to change it in the same thread
context.
Change-Id: I53d139adcca6ff72a3b91a2a931f1239f3271fa9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12558
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: I3b75eea83bd7d700d20a6189e8fb6d1f066dc9b4
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12603
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Clarify via a variable name that we're dealing with the admin CQ
specifically.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I032f6b27e2d75bffb9d95481f177ce0c3655550c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12556
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Constantly polling the socket degrades performance significantly.
Polling the socket at a much lower frequency, every 1ms, is good enough
for now.
fixes#2494
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
Co-authored-by: John Levon <john.levon@nutanix.com>
Change-Id: I4a7d35c45ece863b9df756324c23f41736df49f8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12494
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If we don't set the cid before failing the misordered
command, we use some other random cid, causing the
initiator to think the wrong command was completed.
Fixes#2481.
For this issue, the target was completing a
previously submitted AER, not the fuzzed fused
command. The initiator would then submit another
AER to replace the completed one, but the target
complained that the initiator sent too many AERs
since the target didn't really know it had completed
an AER so hadn't adjusted its num_aer count.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4bd66f147086b262d0e48b8399d237e5ed3c2651
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12452
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
spdk_bdev_get_acwu() is a 1-based number, so we need
to subtract 1 from it before assigning the value to
nsdata->nacwu.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I32708b28a35670cba6013a48b79389fa48226285
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12399
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
ACWU is a 0's based value, and our intent is to
report that our target's ACWU is 1 block. This means
we should report ACWU as 0, not 1.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6ad0606be07fd38bc6c2e3a8e4bb78225b3dfadc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12385
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
The spdk_rmb() in nvmf_vfio_user_poll_group_poll() is unnecessary: we
already have a read barrier for SQ tail updates at the per-SQ level, so
this doesn't add anything.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I88cddd968f4a949640754526e19cb869d9fb31af
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12381
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
There's no need to spdk_rmb() in nvmf_vfio_user_sq_poll() unless we
actually found the tail has advanced.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I778835c527409764c3db78459b2aa76420cc0105
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12378
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
We need to keep track of the shadow doorbell buffer locations, and make
sure to re-initialize on resume.
Co-authored-by: Thanos Makatos <thanos.makatos@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If3ba456fb35f6f6199e4ff14cec1aad96775f71a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12237
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
There were a couple of places not using the standard formatting for qid
still.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If96c3f6d762128b0f274e2c4e9eebf4e80e35139
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12346
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This patch adds an extra spdk_thread_send_msg() call to destroy a qpair
to make sure that it isn't freed from the context of a socket write
callback. Otherwise, spdk_sock_close() won't abort pending requests,
causing their completions to be exected after the qpair is freed.
Fixes#2471
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia510d5d754baccca1e444afdb10696ab9b58e28b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12332
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
As per the NVMe specification, a host can identify two areas of guest
memory: one of which is used for the host-written doorbells, and one of
which contains event indexes. The host writes to the shadow doorbell
area, but also writes to the controller's BAR0 doorbell area if the
corresponding event index is crossed by the update. This avoids many
mmio exits in interrupt mode, where BAR0 doorbells are not directly
mapped into the guest VM, with greatly improved performance.
This isn't a useful feature in BAR0 doorbells are mapped into the VM, so
we explicitly disable support in that case.
NB: the Windows NVMe driver doesn't yet support this feature.
Although the specification says that the admin queues should also engage
in this behaviour, in practice, no VM does, so have to include some
hacks to account for this.
Co-authored-by: John Levon <john.levon@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I0646b234d31fbbf9a6b85572042c6cdaf8366659
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11492
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This lets us use it more widely.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I9c67be19020677fab3eafe05c1e0f91c3d04611d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12307
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
When a qpair is disconnected, any outstanding zero-copy requests are
freed to release their buffers before the qpair gets destroyed.
However, if there is a PDU being sent to the host as part of this
request (e.g. C2HData/R2T), we need to wait until that write is done
before freeing the request to avoid freeing it twice.
Fixes#2445
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I2a6e82f26a4f011dfd18c55c821e9039de7e584a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12255
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This makes the flag indicate whether there's an outstanding PDU write
for a given request. Additionally, it reduces the number of places we
need to update this flag.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id7e587f84955b096c46bfbf88d4dd222214d4a6a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12254
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
This will make it possible to have some common handling in request's PDU
write completion.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Icaff38da0e47dd93327e3d8f09edd9fdba8f532e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12253
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
When an request using zcopy is completed, it might have an unreleased
zcopy_bdev_io attached in three cases:
1) the request was a read,
2) the request was a failed write,
3) the qpair is being disconnected.
The last case was missing from the assertion.
Fixes#2425
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I5cbeaa198a1fd878c98caf148a0bc47060e35bca
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12263
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In a similar manner for what we do for other per IO data-structures of cmds,
cpls and bufs, use the conventional huge-pages based spdk allocation scheme
for RDMA requests and receives.
Change-Id: I4c2e86e928106e78c053f24915e2a9ce1a200c78
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12273
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
To maximize cache locality, use lifo and not fifo when managing objects
which are used per IO such as the RDMA receive elements queue.
Change-Id: Id8917558acc1bec29943fcbae6afe6b072bde6ac
Reported-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12272
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Only FDs are used for passing them to another process,
we can unlink them after creation.
Here we only unlink the files created in vfio-user,
and there is still one file created via libvfio-user,
it will be fixed via
https://github.com/nutanix/libvfio-user/issues/660.
Partly fix issue #2449.
Change-Id: Ie27640e0cb85f44596e9d0ad5a2b67adf0419f5c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12195
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
No functional change; this just makes the poll code a little easier to
read.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If6d1dcd940ed5b461856b535b1bf01c4efa8612a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12076
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
RDMA READs can cause cmds to be submitted to the target
layer in a different order than they were received
from the host. Normally this is fine, but not for
fused commands.
So track fused commands as they reach
nvmf_rdma_request_process(). If we find a pair of
sequential commands that don't have valid FUSED settings
(i.e. NONE/SECOND, FIRST/NONE, FIRST/FIRST), we mark
the requests as "fused_failed" and will later fail them
just before they would be normally sent to the target
layer.
When we do find a pair of valid fused commands (FIRST
followed by SECOND), we will wait until both are
READY_TO_EXECUTE, and then submit them to the target
layer consecutively.
This fixes issue #2428 for RDMA transport.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I01ebf90e17761499fb6601456811f442dc2a2950
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12018
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The process of matching qpair to poll group is split into
two distinct parts that occur on different threads.
See spdk_nvmf_tgt_new_qpair().
This results in a race condition for TCP between spdk_sock_map_lookup()
and spdk_sock_map_insert(), which are called in spdk_nvmf_get_optimal_poll_group()
and spdk_nvmf_poll_group_add() respectively.
Fixes#2113
This patch picks a hint from nvmf_tcp for next poll group,
which is then passed down to spdk_sock_map_lookup().
When matching placement_id exists, but does not have
a poll group assigned - the hint will be used.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I4abde2bc9c39225c9f5dd7c3654fa2639bb0a27f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10271
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The rdma buffer for stripping DIF metadata is added. CPU strips the DIF
metadata and copies it to the rdma buffer, improving the rdma write
bandwith. The network bandwidth during 4KB random read test is increased
from 79 Gbps to 99 Gbps, the IOPS is increased from 2075K to 2637K.
Fixes issue #2418
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Change-Id: If1c31256f0390f31d396812fa33cd650bf52b336
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11861
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Extract the code for DIF from nvmf_rdma_fill_wr_sgl() into
nvmf_rdma_fill_wr_sgl_with_dif().
Then clean up nvmf_rdma_request_fill_iovs() and
nvmf_rdma_request_fill_iovs_multi_sgl().
Additionally, this patch has a bug fix. nvmf_rdma_fill_wr_sgl_with_dif()
returned false if spdk_rdma_get_translation() failed. However, the
type of return value of nvmf_rdma_fill_wr_sgl_with_dif() is not bool
but int. The boolean false is 0 in integer. Hence in this case,
nvmf_rdma_fill_wr_sgl_with_dif() returned 0 even if it failed.
Change nvmf_rdma_fill_wr_sgl_with_dif() to return rc as is if
spdk_rdma_get_translation() returns non-zero rc.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I71cc186458bfe8863964ab68e2d014c495312cd3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11965
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: fengchunsong <fengchunsong@huawei.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
To help grep, use a standard sqid:%d style format for identifying queue
IDs.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ib82c81939f85f9beb333a4db10d006524522a1d9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11822
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
As a general utility function, move it up with the others.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I32881c01afd9819c889730d7c09163c95fbb827e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11790
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
For each queue, track its doorbell location individually, rather than
needlessly recalculating it every time we look up the doorbell value.
This will also greatly simplify shadow doorbell support.
Co-authored-by: Andreas Economides <andreas.economides@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I6882d2f92ee2f2b2b90c54ee14e5f6b41ecca85d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11789
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
R2Ts can cause cmds to be submitted to the target
layer in a different order than they were received
from the host. Normally this is fine, but not for
fused commands.
So track fused commands as they reach
nvmf_tcp_req_process(). If we find a pair of sequential
commands that don't have valid FUSED settings (i.e.
NONE/SECOND, FIRST/NONE, FIRST/FIRST), we mark the
requests as "fused_failed" and will later fail them
just before they would be normally sent to the target
layer.
When we do find a pair of valid fused commands (FIRST
followed by SECOND), we will wait until both are
READY_TO_EXECUTE, and then submit them to the target
layer consecutively.
This fixes issue #2428 for TCP transport.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8a9e13690ecb16429df68ae41b16b439a0913e4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12017
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
In vfio-user transport, whenever one IO is completed, it will trigger
an interrupt to guest machine. This cost quite some overhead. This patch
adds an adaptive irq feature to reduce interrupt overhead and boost
performance.
Signed-off-by: Rui Chang <rui.chang@arm.com>
Change-Id: I585be072231a934fa2e4fdf2439405de95151381
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11840
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Rename ->doorbells to ->bar0_doorbells. This will help avoid confusion
later with shadow doorbells.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Id432938cfeb3033e79dc6e1b491dad964227687a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11788
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
NVMe over PCIe Transport Spec 3.1.2:
The host should not read the doorbell registers.
Explicitly refuse these reads.
Co-authored-by: John Levon <john.levon@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ie64fd5ce7988ee86c612b3ef6046a57af467e266
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11787
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Refactor controller reset a little bit for cleaner code.
Co-authored-by: John Levon <john.levon@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I2b3323005d4e788ffe980d41c349702828886981
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11786
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
If we're in interrupt mode and live migrating a guest, there is a window
where the I/O queues haven't been set up but the device is in running
state, during which the guest might write to a doorbell. This doorbell
write will go unnoticed. This patch ensures that we re-check the
doorbells after an I/O queue has been set up.
Fixes#2410
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I161d2a0e7ab3065022b2bccbe17f019640cceeba
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11809
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Make sure cq->group is set even in interrupt mode.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I9f722917a8e3aebbd5d66648a3909f795897ec1a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11997
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We were accidentally reading the spdk_nvme_status from the CPL; on one
test, this was contributing to 2% of cache misses.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I7c967690458a183799f8d835360800d3094c3131
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11849
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Trivial chore, cleanup multiple branches for subsystem->flags.ana_reporting
that can be collapsed wtihin spdk_nvmf_ctrlr_identify_ctrlr, which makes the
code easier to read.
Signed-off-by: Jon Kohler <jon@nutanix.com>
Change-Id: Icb2064f078d730d1f28f299582dc46254c7f6df3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11951
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When rq is used, rnr_retry_count is not set. As a result, the initiator
cannot connect to the target after some error. Set the rnr_retry_count
to avoid infinite retries.
Fix issue #2429
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Change-Id: Iab03ea5ff10f5a9759c41f26589731aa92ef2f56
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11946
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
When generating a discovery log page, we will add entries
for the discovery subsystem for all listeners except the
one associated with the controller that generated the
log page command. We do this comparison using
spdk_nvme_transport_id_compare().
But this function compares the subnqn of the trid, and
the subnqn is not set in either of the trids that we
are comparing.
The listener's trid always has an empty subnqn, but
the source trid has an uninitialized subnqn when
we do the comparison. This means that sometimes the
subnqn may be empty (which always happens in debug
builds) but sometimes may contain garbage. This
means that sometimes an entry would be added to the
log, even for the trid of the discovery controller
that generated the command (meaning the discovery
controller would end up referring to itself which
is not allowed).
There is an even more subtle issue with this. If the
host reads just the log page header, the nvmf target
generates the entire log page, and just returns the
header contents. Let's say in this case, the source
trid has an empty subnqn, so we don't generate an entry
for it, and report numrec = X and genctr = Y. Then
the host reads the X log page entries. But now the
source trid is garbage, so a discovery log page entry
is returned, replacing one of the "real" log page
entries. And since genctr didn't change, the host
thinks the data is all valid, meaning there's a log
page entry for an NVM subsystem that ends up getting
dropped.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I96cfc566ddaf17153aec089bf3d9b3480bec3e4b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11933
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
If 0 - UINT32_MAX or UINT32_MAX - 0 is substituted into a int variable,
we cannot get any expected result.
Fix the bug and add unit test case to verify the fix.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Iad2ea681ad8ad234e70c7310b58785a999612156
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11837
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
To speed up qpair lookup, an idea to use the DPDK hash library was
proposed. But we have some concerns for the idea, we may see performance
regression for small configuration, lib/nvmf is depending on the DPDK
feature, and so on.
On the other hand, RB tree is simple to use and is not likely to have
negative side effect.
To use RB tree easily, cache QP num into struct spdk_nvmf_rdma_qpair.
Change-Id: Id4ea4e75692f86335b3e984c3fc128e187f22c72
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11798
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
nvmf_rdma_poll_group_add() added a rqpair to the poll_group->qpairs list
before its initialization.
If we change the poll_group->qpairs list from linked list to RB tree,
the qpair can be added to the poll_group_qpairs list only after its
initialization completes because QP number is necessary.
As a preparation, change nvmf_rdma_poll_group_add() to add a rqpair to
the poll_group->qpairs list after its initialization succeeds.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I42525f68c5efbd445f96da8f4f63d31f06f739e3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11800
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
When vfu_run_ctx() returns EBUSY, due to an ongoing quiesce, we did no
work, so should return SPDK_POLLER_IDLE.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I5953ab652f0adf22df81c94c4bece507833cece4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11810
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We will use nvmf library CSTS.CFS instead so that the client can
get this error status.
Change-Id: I42c248a7333d1f9c940bb29135c887a61c906bd4
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11676
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
No logic code changes, we will define a nvmf controller migration
data structure in public header file, this is a preparation patch.
Change-Id: I0a98581b320ac9b12b7e4bf838222c5075168e64
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11493
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
To fully support interrupt mode for the vfio-user transport, we need to
arrange to wake up in one of two conditions:
- we receive a vfio-user message on the socket
- a client writes to one of our BARs
In response, we can process any pending vfio-user messages, as well as
poll the actual queue pairs.
As there is no way for a client-mapped BAR write to cause us to wake up,
interrupt mode can only work when mappable BAR0 is disabled. In that
case, each BAR write becomes a vfio-user message, and can thus be
handled by registering the libvfio-user socket fd with SPDK.
For the poll group poller, we enable interrupt mode for it during the
vfio-user ->poll_group_create() callback; this only works in the case
that no other transports without interrupt mode support are sharing that
poll group.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ic784d01078b2006edb27892fc0613f934eca930b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10722
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Community-CI: Mellanox Build Bot
When generating the discovery log page, add entries for
the discovery subsystem, skipping the listener associated
with the command generating the log page.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id6a14a7d5cdce483f8f3c2eff1b4ededd40bc029
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11542
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Currently we accept connections to the discovery
subsystem on any listener that has been added to
any subsystem (not just the discovery subsystem).
This is not proper behavior, especially for TCP. TCP
defines port 8009 (not 4420) as the discovery port,
so the current behavior means that if NVM subsystems
are listening on port 4420, then the discovery
subsystem by default is listening there too.
For now, continue to allow connections, but print
a warning message when someone connects to the
discovery subsystem on a listener trid that wasn't
previously added.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I734bc49d1a21b2edfb675aef4b8551e2d0ccd4d7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11539
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
One simple fix for nvmf_vfio_user_qpair_abort_request().
Current implementation mixed up request of abort cmd and the request
to abort, which cause problems.
Signed-off-by: Rui Chang <rui.chang@arm.com>
Change-Id: Ia0db9aa738e372789fc502ef877fd1c841c0a2e3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11711
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We do the null check for 'fc_req', but already dereferenced it
before the check. Swap their position to avoid null dereference.
Fixes issue #2395.
Signed-off-by: Weiguo Li <liwg06@foxmail.com>
Change-Id: I33b9e6b51b54f6ada9c072cf7ab0acda2622472f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11721
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
A common pattern is:
if (foo->thread == spdk_get_thread())
cb(arg);
else
spdk_thread_send_msg(foo->thread, cb, arg);
for cases where it's important the callback runs on a particular thread,
but it doesn't matter if it's synchronous or asynchronous.
Add a new API to support this pattern, and convert over the current
instances.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Idfbf77c02c9321c52e07181ffd8b0c437e1ab335
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11503
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
We can ask libvfio-user for the listening socket fd, and register that
for SPDK interrupt handling.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I8d0ba7a86403f2d0170b9359480f1fefc1036557
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10721
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
The accept poller only needs to run when vfu_attach_ctx() makes sense:
in other words, when we don't have a controller created.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Icef4e6184c9ae6d7951d015530a05132c4ba6994
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10720
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Now vfio-user owns its the accept poller itself, there's no reason to
loop across all endpoints: instead, the lifetime of the accept poller is
better matched by creating it in the ->listen() transport callback.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ia92e29b1cee263f1461f640cfdd27cdb674848fb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10719
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
For the benefit of forthcoming vfio-user changes, register the poll
group poller prior to calling the transport create callback, and pass in
a pointer to the poll group itself.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Idbc24126c9d46f8162e4ded07c5a0ecf074fc7dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10718
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The above values were added with shared CQ feature, and they
are left when restore the CQs in destination VM.
Change-Id: Ib1f28dad833da31e571eb2e2f0b5c81f0bf05a3b
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11419
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The CSTS register in source VM was left to migrate.
Fix#2362.
Change-Id: Ieef028578d2897b27a6ff594b16801462eb1b75e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11418
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The dif_enable is not cleared in _nvmf_rdma_request_free. When the
rdma_req is used again, the dif_enable is true while
dif.dif_ctx->block_size is zero. As a result, an infinite loop occurs in
nvmf_rdma_fill_wr_sgl.
Fix issue: #2380
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Change-Id: Ic179855f7b257e39ed4a5f6705fbc9dea64210ae
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11646
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
This was a parameter on the nvmf_create_transport
RPC, and was replaced with max_io_qpairs_per_ctrlr to
reduce confusion on whether this number included the
admin queue or not.
nvmf_vhost test was using this deprecated parameter.
Change it to use -m (max_io_qpairs_per_ctrlr)
instead. '-p 4' would have been evaluated as 1 admin
queue + 3 I/O queues, but it's likely the intent
was for 4 I/O queues. This is a perfect example of
why this parameter was deprecated.
For reference, this was deprecated in June 2020,
commit 1551197db.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4364fc0a76c9993b376932b6eea243d7cefca9cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11543
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
The controller data structure may be freed before subsystem resume done
callback, we can take endpoint as the input parameter to avoid this issue.
AddressSanitizer: heap-use-after-free on address 0x625000046100 at pc 0x00000082818f bp 0x7fff7b09bd10 sp 0x7fff7b09bd00
READ of size 8 at 0x625000046100 thread T0 (reactor_0)
#0 0x82818e in vfio_user_dev_quiesce_resume_done /spdk/lib/nvmf/vfio_user.c:2147
#1 0x782cc0 in subsystem_state_change_done /spdk/lib/nvmf/subsystem.c:634
#2 0xad047b in _call_completion /spdk/lib/thread/thread.c:2344
#3 0xabc48d in msg_queue_run_batch /spdk/lib/thread/thread.c:710
#4 0xac0670 in thread_poll /spdk/lib/thread/thread.c:926
#5 0xac0ead in spdk_thread_poll /spdk/lib/thread/thread.c:986
#6 0x9a5b4f in _reactor_run /spdk/lib/event/reactor.c:920
#7 0x9a6442 in reactor_run /spdk/lib/event/reactor.c:958
#8 0x9a717c in spdk_reactors_start /spdk/lib/event/reactor.c:1060
#9 0x99884a in spdk_app_start /spdk/lib/event/app.c:643
#10 0x407e82 in main /spdk/app/nvmf_tgt/nvmf_main.c:75
#11 0x7f822095ff42 in __libc_start_main (/lib64/libc.so.6+0x23f42)
#12 0x407abd in _start (/spdk/build/bin/nvmf_tgt+0x407abd)
0x625000046100 is located 0 bytes inside of 8320-byte region [0x625000046100,0x625000048180)
freed by thread T0 (reactor_0) here:
#0 0x7f82219ff91f in __interceptor_free (/lib64/libasan.so.5+0x10d91f)
#1 0x837059 in _free_ctrlr /spdk/lib/nvmf/vfio_user.c:2976
#2 0x837327 in free_ctrlr /spdk/lib/nvmf/vfio_user.c:2996
#3 0x843541 in nvmf_vfio_user_close_qpair /spdk/lib/nvmf/vfio_user.c:3742
#4 0x7d1d91 in nvmf_transport_qpair_fini /spdk/lib/nvmf/transport.c:604
#5 0x7ad922 in _nvmf_qpair_destroy /spdk/lib/nvmf/nvmf.c:1055
#6 0x761362 in nvmf_qpair_request_cleanup /spdk/lib/nvmf/ctrlr.c:4026
#7 0x761906 in spdk_nvmf_request_free /spdk/lib/nvmf/ctrlr.c:4041
#8 0x75a931 in nvmf_qpair_free_aer /spdk/lib/nvmf/ctrlr.c:3576
#9 0x7ae626 in spdk_nvmf_qpair_disconnect /spdk/lib/nvmf/nvmf.c:1127
#10 0x83db36 in _vfio_user_qpair_disconnect /spdk/lib/nvmf/vfio_user.c:3433
#11 0xabc48d in msg_queue_run_batch /spdk/lib/thread/thread.c:710
#12 0xac0670 in thread_poll /spdk/lib/thread/thread.c:926
#13 0xac0ead in spdk_thread_poll /spdk/lib/thread/thread.c:986
#14 0x9a5b4f in _reactor_run /spdk/lib/event/reactor.c:920
#15 0x9a6442 in reactor_run /spdk/lib/event/reactor.c:958
#16 0x9a717c in spdk_reactors_start /spdk/lib/event/reactor.c:1060
#17 0x99884a in spdk_app_start /spdk/lib/event/app.c:643
#18 0x407e82 in main /spdk/app/nvmf_tgt/nvmf_main.c:75
#19 0x7f822095ff42 in __libc_start_main (/lib64/libc.so.6+0x23f42)
previously allocated by thread T0 (reactor_0) here:
#0 0x7f82219fff16 in __interceptor_calloc (/lib64/libasan.so.5+0x10df16)
#1 0x837413 in nvmf_vfio_user_create_ctrlr /spdk/lib/nvmf/vfio_user.c:3010
#2 0x83bc68 in nvmf_vfio_user_accept /spdk/lib/nvmf/vfio_user.c:3313
#3 0xabfbd8 in thread_execute_timed_poller /spdk/lib/thread/thread.c:872
#4 0xac0c75 in thread_poll /spdk/lib/thread/thread.c:960
#5 0xac0ead in spdk_thread_poll /spdk/lib/thread/thread.c:986
#6 0x9a5b4f in _reactor_run /spdk/lib/event/reactor.c:920
#7 0x9a6442 in reactor_run /spdk/lib/event/reactor.c:958
#8 0x9a717c in spdk_reactors_start /spdk/lib/event/reactor.c:1060
#9 0x99884a in spdk_app_start /spdk/lib/event/app.c:643
#10 0x407e82 in main /spdk/app/nvmf_tgt/nvmf_main.c:75
#11 0x7f822095ff42 in __libc_start_main (/lib64/libc.so.6+0x23f42)
SUMMARY: AddressSanitizer: heap-use-after-free /spdk/lib/nvmf/vfio_user.c:2147 in vfio_user_dev_quiesce_resume_done
Change-Id: Icf5e5b360b9107a3c5eb960ae59b7fe10ace1c66
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11420
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
if ctrlr->listener was NULL, nvmf_ctrlr_get_ana_state() returned
inaccessible even if ana_reporting was disabled. Then the corresponding
initiator received unexpected ANA error and could not process it
appropriately.
Change nvmf_ctrlr_get_ana_state() to return optimized always if
ana_reporting is disabled.
Additionally, check if ctrlr->listener is not NULL before calling
SPDK_DTRACE_PROBE3().
Fixes#2335
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ib2376694cf89d85ec5687fba7e87439f494f30b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11402
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
it shall be executed on ctrlr's thread not subsystem's
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I58c60525191085d3d6a583862ba5d71ea90940c7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11105
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
nvmf_vfio_user_sq->size and ->qsize both hold the number of entries in
the queue; merge them.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I6c7c2984cbdf90079eec9222e1acbedb92207308
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11297
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Since we already allocate ->sg on a per-request basis now, drop the
->reqs_internal allocation in favour of allocating individual requests
and placing them on the ->free_reqs queue, which simplifies the need to
track the array. For request abort, we'll use the ->outstanding request
list, now we have it, to find the victim request.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I25227ccd2ab7e00c8a2e7b2e2af2dc3b073584cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11427
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
On failure, we weren't cleaning up the poll group data properly, and in
one place, we were trying to remove ourselves from the tgt-> list prior
to being on it.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I9bbe5847b3703eba1ee1d762392ad3159a74ac8b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10717
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
There's no need to forward-declare this, when we can just place it
before its consumers, and this will also help follow-up fixes.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I201bd966371db76a3b789473041799bf55b13c95
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11437
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Use of %p in logging simplifies this code a little bit.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I8e5daa59a614b8bcde7d67d1e5cc6196923031a8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11244
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Avoid using the modulus operator in the hot-path cq_is_full(),
by aping how cq_tail_advance() is written.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Idbdf1715ab30d08233b38aa7691f0212ae93a542
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11445
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Avoid using the modulus operator in the hot-path sq_head_advance(),
by aping how cq_tail_advance() is written.
Suggested-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Id1e9d63a08e470344fdeb549d78ea505088b1a62
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11436
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
There is very little now shared between submission and completion
queues, so drop usage of this struct, folding its remaining members
into the relevant owning types.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I02195d1944ca9905ef03ddf2c099ddb806df70dc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11296
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Consistently wrap access to queue heads/tails, so it's easier to make
further changes. Adjust sqhd_advance() to match the head/tail naming of
the accessor functions, and move the definitions to be close to each
other.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I197e230ecc4e67fe0207f29281d7e4ca946c22e1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11295
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add a struct defining the local mapping of a queue.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Id3bbdf72bfc082f4496748571bd2617bdafe4309
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11294
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We did not have any practical limitation to support concurrent
execution of multiple abort commands.
NVMe specification recommends that implementations support a minimum
of four abort commands.
Let's follow the NVMe specification.
As stated in the head, we do not have any limitation, and we do not
have to check if abort commands exceeds ACL or not.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I31e066fadcb5d619d0c50c895c4cb64520b33513
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11232
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where two versions exists with matching
versions, but conflicting ABI.
Ex. Next SPDK release adds an API call increasing the minor version,
then LTS needs just a subset of those additions.
Increasing major so version after LTS, allows the future releases
to update versions as needed. Yet allowing LTS to increase minor
version separately.
Disabled test for increasing SO version without ABI change, as
that is goal of this patch. This check shall be removed with SPDK 22.05
release.
This patch:
- increases SO_VER by 1 for all components
- resets SO_MINOR to 0 for all components
- removes suppressions for ABI tests
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id1a5358882dc496faa5b0b5c9a63b326c378c551
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11361
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
With the new added property access API, we can send a internal
property access request to NVMf library, and we can use
it to reset controller.
Change-Id: Iee8b1146d9eb31bc98a9b297e5c635e43e6fdb12
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10952
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
VFIO in QEMU uses region 9 as the PCI passthrough devices' migration channel.
The format of the region 9 migration region is as follows:
------------------------------------------------------------------
|vfio_device_migration_info| data section |
------------------------------------------------------------------
QEMU will access vfio_device_migration_info to controll the migration
process.
For SPDK vfio-user target, we also implement the BAR9 via libvfio-user,
and we also define the NVMe device specific migration data stored in
data section of BAR9. QEMU doesn't care about the format in data section,
it will help us to gather the NVMe specific migration data in source VM and
then restore the migration date to data section of BAR9 in destination VM.
The core idea to implement live migration will following the device state
change which is controlled by QEMU. First QEMU will try to STOP the device
in the source VM, and set the destination VM to RESUME state, SPDK will save
NVMe devic state data structure to BAR9 in the source VM once the subsystem
is paused, then QEMU will read BAR9 in source VM and restore the content of
BAR9 in destination VM, finally in the destination VM, we will restore the
NVMe device state include BARs/PCI CFG/queue pairs in the destination VM.
Change-Id: I42e38f28c3ff59831be63290038b50d199d06658
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7617
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When subsystem is destroyed, it removes its listeners,
however transport level listeners remain active.
This patch removes all transport listerners when
the transport is being destroyed.
Fixes issue 2353/
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: Ica7bcb0052b626aa62d0da9049bb8f216027dc49
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11307
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
When doing live migration the migration BAR region is bytes stream
data, so here we use the helper function to save current controller
state into the stream in source VM and load it as internl data
structure from steam in destination VM.
We will remove the `unused` attrubute in next patch.
Change-Id: Ib44adb351c697b50b9220ce6943cc017137a6064
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10336
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When doing live migration, the destination VM will construct
ADMIN queue pair at the beginning, but the controller isn't
in READY state, we should not poll the ADMIN queue pair right
now. This is fine for normal controllers, normal controllers
will set ADMIN queue pair state in CC callback.
Change-Id: I0db36f75a463fb7476ee62323f9ed0c74c2451dc
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10621
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When doing live migration, there are some spdk_nvmf_ctrlr internal
data structures need to be saved/restored.
Change-Id: Ie39482e8c49765c36fc3700fbac4ce47ef306f29
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10058
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When doing live migration we need to restore the AER commands
in the destination VM, so here to provide an API to save
these CIDs and the transport layer can save the value.
After migration in destination VM, we should allocate
new AER requests based on CIDs in vfio-user.
Change-Id: I5881f833bbfacb0f030a2b135b4dd47726240378
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10040
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
There is a race condition between controller destruction and
subsystem state change, e.g. admin qpair may already be freed
when a namespace is added or removed. As result in function
poll_group_update_subsystem we may get heap-use-after-free error
Another problem is that some qpair's live time may exceed controller's
life time. To avoid it, start controller destruction process when the last
qpair finished the disconnect process (previously controller started
the descruction process before the last qpair starts to disconnect
and it could lead to raise conditions)
Fixes#2055
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: Ibc99b1d840e4796e1588cc217d65834bb556b909
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9995
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Previously we didn't post the response for CREATE IO SQ command
until the queue pair is connected finally, but for coming live
migration support, we will connect IO queue pairs in the destination
VM, and this function will also be called for this case, so here
we add a flag to indicate the CREATE IO SQ case.
Change-Id: Iab4c64a7ebb72bcffbfff712dc729c40eead7c7d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9464
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The miration region data structure is from `vfio_device_migration_info`
defined in `linux/vfio.h`, `vfio_device_migration_info` is in the 0th
offset of the VFIO_REGION_SUBTYPE_MIGRATION region, and in vfio-user,
we reserve first one page of BAR9 for this MMIO accesses.
libvfio-user already helps us to hide some implementation details
based on vfio migration specification, here we just use the two
fields to help the migration process.
Change-Id: I8917ba892bbfdfdf4f135f5d6b4923ab0e4a6250
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7628
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We will report the live migration region to VM via sparse
mmap, offset after 0x1000 is the NVMe device state data
structure, and offset start from 0 is the structure
vfio_device_migration_info defined by the VFIO driver.
All accesses between 0x0-0x1000 will use the MMIO callbacks,
and accesses to NVMe device state will use shared memory map
way.
Change-Id: Ib456fc61f587c1bffa8b38506b4480a6066abe87
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7627
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We will use the NVMe device state data structure to save/restore
a NVMe controller in source/destination VM.
NVMe device migration region is defined as below:
----------------------------------------------------------------------
| nvme_migr_device_state | private controller data | queue pairs | BARs |
----------------------------------------------------------------------
Change-Id: Idc73976e1de7f6da2da58e71db86df8cbb0d314d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7626
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
For CREATE IO SQ command, we will defer to post completion
until the SQ was connected, we may call post_completion()
in different threads, so here we will send a message
to CQ thread when necessary.
Change-Id: I87a0f8982811c76ce8eb49db6a136f4cbe6e0a93
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11078
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
