The completion callback of `spdk_nvmf_subsystem_resume`
and `spdk_nvmf_subsystem_pause` can run in different
core other than the `vfu_ctx` core, this may lead to
race condition when changing controller's state. Here
we use a thread message to change it in the same thread
context.
Change-Id: I53d139adcca6ff72a3b91a2a931f1239f3271fa9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12558
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The following patches swaps the ordering of destrloying I/O qpairs
and disconnecting a controller for PCIe transport.
prepare_for_reset is a flag for PCIe transport.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I3009de9fea089fc93ecf87adba42e85c9a77e715
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12582
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
As described in the previous patches, we need to delete all I/O
SQ/CQs before aborting trackers when disconnecting a controller.
The following patches reorder the operations. This patch changes
adminq disconnection to initiate a Controller Level Reset and
adminq completion processes it if ctrlr->is_disconnecting is true.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I64f06bae2ce8a9127124029fd042db0028198e3c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12560
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Make this a transport-level decision instead. TCP and RDMA do want to
abort, but PCIe cannot because these commands may still be receiving DMA
operations from the device.
Change-Id: I305acddc3819c903eb3217e8f710d4216d0b3931
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11509
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Previously, we did not do any Controller Level Reset when disconnecting
the admin qpair.
However, for PCIe transport, we need to stop any activity of the
controller, i.e., delete all I/O SQ and CQs before
nvme_transport_ctrlr_disconnect_qpair_done() calls
nvme_transport_qpair_abort_reqs() (i.e., nvme_pcie_qpair_abort_trackers()).
Otherwise, some corruption may occur because completed I/Os may still be
in progress on the NVMe device.
Not to change any public API, nvme_pcie_ctrlr_disconnect_qpair() is a
convenient place to initiate a Controller Level Reset because it is
called from spdk_nvme_ctrlr_disconnect(). Then
nvme_pcie_qpair_process_completions() can process it until completion.
However, necessary functions are not accessible from PCIe transport.
This patch adds two helper functions and guards us from some undesirable
behaviors because it was not assumed that nvme_ctrlr_process_init() is
called from the completion context and ends in the middle of transition.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I3d986e94ba71b83beeff7e75cf92033b5fa6f075
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12559
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When a new cluster is added to a thin provisioned blob,
md_page is allocated to update extents in base dev
This memory allocation reduces perfromance, it can
take 250usec - 1 msec on ARM platform.
Since we may have only 1 outstainding cluster
allocation per io_channel, we can preallcoate md_page
on each channel and remove dynamic memory allocation.
With this change blob_write_extent_page() expects
that md_page is given by the caller. Sicne this function
is also used during snapshot deletion, this patch also
updates this process. Now we allocate a single page
and reuse it for each extent in the snapshot.
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I815a4c8c69bd38d8eff4f45c088e5d05215b9e57
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12129
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
To fix issue: #2484
When unregistering the bdev, will send out the message
to each thread to abort all the IOs including IOs from
nomem_io queue, need_buf_small queue and need_buf_large queue.
The new SPDK_BDEV_STATUS_UNREGISTERING state is newly
added to indicate this unregister operation.
In this case, the bdev unregister operation becomes the
async operation as each thread will be sent the message
to abort the IOs and as the last step, it will unregister
the required bdev and associted io device.
On the other hand, the queued_resets will be handled
separately and not aborted in the bdev unregister.
New unit test cases are also added:
enomem_multi_bdev_unregister: to abort the IO from
nomem_io queue during the unregister operation
bdev_open_ext_unregister: to handle the events and
async operations from the unregister operation
Change-Id: Ib1663c0f71ffe87144869cb3a684e18eb956046b
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12573
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Instead of releasing the batch memory when the batch generates a
completion, instead do it via refcnt. This will allow us to later hold
onto batch memory longer if vectored transactions end up spanning a
batch.
Change-Id: I942d6aa5052029eb0951e51a046dd98943108b94
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12259
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
If nbytes is not set, then the desination iovec sent to the underlying
driver has a length of 0.
Change-Id: Ia55f5ece942bd70f32bfdb3bcf02134ba98fca96
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12612
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
It simplifies code and removes cast of nvme_qpair
to rdma_qpair
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I363246cf9d8c9cbafd48b26facdb5cc37fdd8e67
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12701
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When qpair is attached to a poll group, disconnect
process is async - we are waiting for the DISCONNECTED
event from rdmacm to destroy rdma resources. However
the user (nvme_perf) can destroy qpair immediatelly,
so memory allocated for qpair is freed but rdma
resouces are still allocated. That means that we may
receive rdmacm event (DISCONNECTED) for the destroyed qpair,
that leads to use-after-free.
To fix this problem, add a check for internal qpair state
when qpair is destroyed, if disconnect is not finished, then
we forcefully destroy rdma resources.
Fixes issue #2515
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reported-by: Or Gerlitz <ogerlitz@nvidia.com>
Change-Id: I7bfa53c9f6fe6ed787323a8941f1f2db17ea0c20
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12700
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Each spdk_vhost_dev_backend is local to either
SCSI or BLK backends, so its not possible to gauge which
backend is used by the vdev on generic vhost layer.
Added a `type` field with matching enums to differentiate
between the two. Later patches will check that field
in vhost.c.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I2a95961b9f9b5f070db7b22d44cf5114a24b1067
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12675
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Without this, it does not think there is a driver available.
Change-Id: I0b8b42374e0ed82abb22bf27e0b8907bb03c61f6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12641
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Currently, when you run vhost user target, no matter if the reactor is
busy or not, spdk_top always shows 100% busy. Fix this by real load
status.
Signed-off-by: Rui Chang <rui.chang@arm.com>
Change-Id: I610a8c2f4e74f46bd56955d31284372c775507ed
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12647
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
These function accept optional spdk_blob_ext_io_opts
structure. If this structure is provided by the user
then readv/writev_ext ops of base dev will be used
in data path
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I370dd43f8c56f5752f7a52d0780bcfe3e3ae2d9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11371
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
When snapshot is created, the new blob is loaded and
examined for BLOB_SNAPSHOT xattr in blob_load_backing_dev
function. At this step there is no such xattr, so zeroes
back_bs_dev is created. Later snapshot inherits back_bs_dev
from original blob, so previously created back_bs_dev can
be lost.
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I90cc9b02f56598d8c5c7fe00409f571fba0aa91a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11384
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Introduce spdk_blob_ext_io_opts structure which
is used in the new *_ext functions.
Zeroes dev is updated with implementation of
readv_ext which uses memory domains memzero
or regular memset().
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: Id94542196eff999827bf00591fd43804256fccb4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11369
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
In the following patches, nvme_ctrlr_process_init() will be used to
disable the controller when disconnecting the admin qpair for PCIe
transport. In this case, we will have to exit nvme_ctrlr_process_init()
after CSTS.RDY is 0. However, spdk_nvme_ctrlr_reset() and
spdk_nvme_ctrlr_reconnect_poll_async() have to continue
nvme_ctrlr_process_init() until the controller becomes ready.
To differentiate stop and continue clearly, add a new state
NVME_CTRLR_STATE_DISABLED to enum nvme_ctrlr_state.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ic0a5fb7114d4eeb1cefec28bc404184768fb0a96
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12613
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: I3b75eea83bd7d700d20a6189e8fb6d1f066dc9b4
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12603
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: I32dd9960bc397244d8e3d0a384fc8b67e907bf68
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12601
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Move check if opts needs copy to its own function.
Move check if opts is valid into its own function.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ie006ddd0b642eb97aaa3ab13890800322dee7a42
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12571
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
vfio_user/host is the PCI abstraction over vfio-user transport, it's
client library. We will add a target library to emulate PCI devices
in next patch, so new lib/vfio_user contains two libraries, one
is for host, the other one is for target.
Change-Id: I9bb40043105525654360691d6db62e4958384e7f
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12314
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Clarify via a variable name that we're dealing with the admin CQ
specifically.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I032f6b27e2d75bffb9d95481f177ce0c3655550c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12556
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: I6931e80c836b568dec8989dad2a7be4e112c42b4
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12577
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Fix ocf test script that was still using the
deprecated get_bdevs RPC name - change it to
bdev_get_bdevs.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7f8caedc250b80503671a0236694181613f63860
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12553
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2c9918ed0296f644b0728c5106c47d93e3c7ec30
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12552
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Constantly polling the socket degrades performance significantly.
Polling the socket at a much lower frequency, every 1ms, is good enough
for now.
fixes#2494
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
Co-authored-by: John Levon <john.levon@nutanix.com>
Change-Id: I4a7d35c45ece863b9df756324c23f41736df49f8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12494
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When using the SPDK nvme driver in multi-process mode,
and multiple processes detach from their devices
(i.e. at application exit) very close to each other,
there are cases where a process can have started its
own detach, and then get two remove notifications - one
from its own detach, and another from another process'
detach.
So we need to remove the assertion in pci_device_fini(),
if the removed flag has already been set.
Fixes#2456.
Tested using a modified version of the
nvme_multi_secondary() function in test/nvme/nvme.sh
that starts several additional perf applications.
The test consistently failed without this patch,
and passes every time with this patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I91e16985cdc4a463aaac2c45096bb967aab85560
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12454
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
We enable multiple engines by:
* getting rid of the globals that point to the one available HW
and one available SW engine
* adding a submit_tasks() entry point for the SW engine so that
it is treated like any other engine allowing us to just call
submit_tasks() to the assigned engine for the opcode instead of
checking what is supported
* changing the definition of engine capabilities from
"HW accelerated" to simply "supported"
* during init, use a global (g_engines_opc) that contains engines
and is indexed by opcode so we know what the best engine is for each
op code
* future patches will add RPC's to override engine priorities or
specifically assign an opcode(s) to an engine.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I9b9f3d5a2e499124aa7ccf71f0da83c8ee3dd9f9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11870
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
If we don't set the cid before failing the misordered
command, we use some other random cid, causing the
initiator to think the wrong command was completed.
Fixes#2481.
For this issue, the target was completing a
previously submitted AER, not the fuzzed fused
command. The initiator would then submit another
AER to replace the completed one, but the target
complained that the initiator sent too many AERs
since the target didn't really know it had completed
an AER so hadn't adjusted its num_aer count.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4bd66f147086b262d0e48b8399d237e5ed3c2651
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12452
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We see reports that Huawei SSDs can't handle hardware
SGL properly, it requires additional alignment, so add
a quirk here to force Huawei SSDs use PRP instead.
Fix#2489.
Change-Id: I20a57e754bc6ff8666d681191994818f2192decc
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12405
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: wanghailiang <hailiangx.e.wang@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
spdk_bdev_get_acwu() is a 1-based number, so we need
to subtract 1 from it before assigning the value to
nsdata->nacwu.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I32708b28a35670cba6013a48b79389fa48226285
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12399
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This does not actually need a doubly linked list. Single is enough. That
frees up 4 more bytes in the op for other uses.
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I8c2a30de175b42815afd0a3ba3c694aef2f35882
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12258
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
If we don't find a completion, break out of the loop at the top. This
removes a level of indentation but most importantly makes it very clear
that we only remove elements from ops_outstanding from the front of the
list.
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I5e8784a5af5449c14ff7015bc8e6062e6aee6b4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12257
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
The partition bdev checks if the underlying device supports
the io type and sends the bdev_io directly down to the bdev.
This patch adds missing compare and compare&write io types
to the partition bdev.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ice7e5c0332ce7e564bad2bb8d7f4bb1d535388c1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12390
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The bdevio test app has some test cases verifying
that write zeroes commands are handled correctly,
but using knowledge of the ZERO_BUFFER_SIZE that
the bdev library uses for splitting larger write
zeroes commands. Instead of hardcoding that 1MB
value in bdevio.c, have bdevio.c use ZERO_BUFFER_SIZE
directly instead. But this requires moving
ZERO_BUFFER_SIZE into bdev_internal.h and having
bdevio.c include that file.
We do this instead of putting ZERO_BUFFER_SIZE in
the public API because we don't want users to
make any kind of dependencies on this value.
While here, also rename the tests that are using this
value, so that the test names don't include any reference
to the specific size of this bdev-internal zero buffer
size.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia29d92a706cb1f86b4c29374dc2a9beccf679208
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12383
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
This patch adds process_virtio_blk_request() that will be called
by virtio_blk transports to process incoming requests.
Meanwhile vhost_user_blk_request_finish() will be replaced with
a callback to the virtio_blk transport to notify of the result.
blk_request_finish() should only be called as direct result of
process_virtio_blk_request(), usually from it.
Some error paths now call vhost_user_blk_request_finish() directly,
if vhost_user_process_blk_request() was not called yet.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I0cce22f15b922fe45f30fb659c384b6e836def4c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9537
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
struct spdk_vhost_blk_task shall contain only fields
that are relevant to the generic virtio blk layer.
Moved to vhost_internal.h header to allow for broader
use.
In contrast to spdk_vhost_user_blk_task that is
specific to vhost_user transport.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I8438d3e6dbca816855f55ee998a632f50acde045
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12282
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Later in the series, the spdk_vhost_blk_task will not contain
vhost_user specific fields. As such the generic part of processing
I/O will not be able to refer to those.
References to req_idx are replaced with pointer to the task,
meanwhile bvsession reference when queueing I/O is removed.
Note that vhost_user fields are accessible in the final callback
for the I/O, and will be printed. See blk_request_finish().
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I100d60968146da778bd6bf4fbcf2a2694d3be6e6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12335
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Previously the blk_request_queue_io() and blk_request_resubmit()
relied on vdev and channel contained in task or bvsession structures.
In an effor to make the I/O processing for virtio blk not reliant
on vhost_user, this patch caches the vbdev and io channel submited
in process_blk_request().
Later in the series, vhost_user structures will be separated out from
the spdk_vhost_blk_task.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If1ea38a77af8fcfee12054f5857a6db2db2093c6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12334
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
ACWU is a 0's based value, and our intent is to
report that our target's ACWU is 1 block. This means
we should report ACWU as 0, not 1.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6ad0606be07fd38bc6c2e3a8e4bb78225b3dfadc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12385
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
The spdk_rmb() in nvmf_vfio_user_poll_group_poll() is unnecessary: we
already have a read barrier for SQ tail updates at the per-SQ level, so
this doesn't add anything.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I88cddd968f4a949640754526e19cb869d9fb31af
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12381
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
There's no need to spdk_rmb() in nvmf_vfio_user_sq_poll() unless we
actually found the tail has advanced.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I778835c527409764c3db78459b2aa76420cc0105
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12378
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
When sending the first part of a fuse command, we set the
first_fused_submitted flag so that we don't ring the doorbell
immediately. When the second part is sent, we ring the doorbell for
both commands.
However, this doesn't work well when we use the option to delay ringing
the doorbell. We send both parts, then later when we try to ring the
doorbell, we don't because of the first_fused_submitted flag from the
first command.
Replace this mechanism by keeping track of the last submitted fuse.
Change-Id: Ia4ac9b3ce9c319ee4c7e42f86eadda93dac85fca
Signed-off-by: Alex Michon <amichon@kalrayinc.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12182
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
We need to keep track of the shadow doorbell buffer locations, and make
sure to re-initialize on resume.
Co-authored-by: Thanos Makatos <thanos.makatos@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If3ba456fb35f6f6199e4ff14cec1aad96775f71a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12237
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
process_blk_request() accepted spdk_vhost_blk_session argument
which is specific for vhost_user. In preparation to making
this function usable outside of vhost_user, replace this field
with struct spdk_io_channel and spdk_vhost_dev.
For this purpose vhost_user_process_blk_request() was created to
translate from spdk_vhost_blk_task to the generic arguments.
to_blk_dev() was moved further up so it can be more commonly used
throughout the file.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ie61a1ae2a615c4f1a95601e533b9eec51998cd07
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12333
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In the qemu virtualization environment, when the virtio driver of the
windows image is upgraded to virtio-win 0.1.185, and virtio uses
legacy mode. In the negotiation stage of vhost, qemu's host_features
will enable VIRTIO_F_ANY_LAYOUT, and guest_features will also enable
this feature, because qemu's feature is a superset, so the feature
will be passed to the spdk side through the set feature process,
which leads to "VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES
failed. VHOST_CONFIG: vhost message handling failed.", so we enable
this bit for spdk vhost.
Signed-off-by: suhua <suhua1@kingsoft.com>
Signed-off-by: lizhaoxin <lizhaoxin1@kingsoft.com>
Change-Id: I27323bf5a03dce774c8a74cfb070ddd43be05534
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12300
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
