For all the vhost-user messages processed in SPDK except
VHOST_USER_GET_VRING_BASE, DPDK rte_vhost "vhost-events"
thread already holds all VQ's access lock, before return
response to "vhost-events" thread, SPDK should not call
`rte_vhost_vring_call`, here we set a flag to TRUE for
these vhost-user messages, and avoid to kick VM. The
deferred IRQs will be posted in next round poll or
after restarting the device.
Fix issue #2518.
Change-Id: I82f14b97d0b0ce602a93fd66d5fdeef64f07d179
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14402
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Currently we will call `rte_vhost_vring_call` in the DPDK
"vhost-events" thread context when starting the device, and
DPDK vhost library already holds all VQ's access lock when
starting device, with new DPDK/dpdk@c573699 commit, it will
cause deadlock to call `rte_vhost_vring_call` in "vhost-events"
context, so here we increase 1 to `used_req_cnt` to make sure
one more `rte_vhost_vring_call` will be executed later in SPDK
thread context.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: Iab53941942335744bf25ab6e9b8747bd08b0c698
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14328
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
`rte_vhost_vring_call` may return error, then we can
try to call it in next poll.
Change-Id: I8f6a591837225079e004c6f57f2d7b01063f87a1
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14342
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
We do not support dynamic memory allocation with the virtio-user
library - it results in SET_MEM_TABLE vhost messages for every
change which is not supported by the vhost target.
Add '-s 256' to vhost fuzz script, to ensure it does not
violate the new restriction.
This is a follow-on patch for issue #2596.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If851f53d7d670ac8443f0d9c8f4e3cbe82e0df7c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14249
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Before this change, a negative `--shm-id` value is rejected by
`spdk_app_parse_args()` and this function simply errors out after
detecting it. However, `build_eal_cmdline()` has a dedicated branch
checking for a negative `opts->shm_id` and passes `--no-shconf` down
to DPDK as a parameter, so we cannot disable the shared config support
in DPDK.
After this change, a negative value `--shm-id` is accepted, but if it
cannot be parsed as an integer, `spdk_app_parse_args()` errors
out as before. In result we can disable shared config support in DPDK
by passing `--shm-id=-1` to SPDK application.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Change-Id: Ibe089f13638eefa9ac28c5c99e303bcc3102f307
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14097
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The generic bdev layer has a public API spdk_bdev_get_qd() but its
value is the most recently measured value and it requires qd sampling
to be enabled. We will have bdev modules to want to wait until
all bdev_ios are aborted by a reset. Unfortunately, spdk_bdev_get_qd()
is not suitable for the custom bdev module. Furthermore,
spdk_bdev_channel::io_outstanding is not accessible from bdev modules.
Hence, add a new public API spdk_bdev_get_current_qd().
This function should be used only from the bdev module and it should
be ensured that the bdev is not unregistered during execution.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ica30a8d8fe3264e28f0772a39bdf5f9ba72933e1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12791
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Some use cases want to abort every bdev_io submitted to the bdev by
traversing the bdev channels.
However, struct spdk_bdev_channel is private in lib/bdev/bdev.c.
Hence, add a helper function spdk_bdev_for_each_bdev_io() to execute
the function on the appropriate thread for every bdev_io submitted
to the bdev.
This function should be used only from the bdev module and it should
be ensured that the bdev is not unregistered during execution.
We keep this function as generic as possible because we may have
other use cases in future.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ic0209361bd1228ea8d4cb3241d0df07106be58d9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12751
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Former code, there're many repeated defines. And some add
asserts checking valid event and some don't add. To get the right
reports from debugging mode and catch the errors, so encapsulate
a common function to do these. And add assert in this function.
This will help get the right failure point.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I23d71eac6652c4104ceff80419f39634ac5ce395
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14335
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We already define a convenient variable for the admin CQ: use it.
Suggested-by: Alexis Lescouet <alexis.lescouet@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If6570f30844a52113633bdb5f3543eec700f05d7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14391
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In order to fit inside the maximum memory usage limit, recovery needs
to be split into multiple parts. During each iteration, part of L2P needs
to be read, modified as necessary and saved back to the cache.
This patch introduces the load/save steps, initialization of seq_id array
and valid map recovery. The actual L2P recovery is done in the followup patch.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I8ceadc5ef280542a173d83b932a983d5d86604a1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13371
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds recovery of open bands from P2L metadata region. Recovers the
commited P2Ls and write pointers for them.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I943c53f55e653dd075035cef7ddba448c990be87
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13370
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Recovers the open/close/free state of bands after shutdown, initializing
necessary lists.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4a6bd4ed1013ce8d04f44d1772dcd1f0e4e365bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13368
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Since base device doesn't require VSS, FTL introduces a mechanism that
will allow for recovering both the P2L and write pointer of open bands
after a dirty shutdown. After writing 1MiB of data to a band, a 4KiB
block describing the P2L will be persisted to cache device, effectively
emulating VSS for the base device.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ic6be52dc09b237297a5cda3e752d6c038e98b70e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13367
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Track the relative sequence of opening and closing bands and chunks.
Necessary for detecting the most recent user data during dirty shutdown recovery.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I682030e58284d7b090667e4e5a9f4bbc7615708a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13366
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
hexlify and unhexlify utils from vbdev_crypto.h have been moved so that
they could be included and reused outside of vbdev_crypto module.
Signed-off-by: Blachut, Bartosz <bartosz.blachut@intel.com>
Change-Id: Ia074250176907f4803b84024239ecd4e9d8a5fc1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14191
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The word engine was both used (interchangeably with module) to refer to
the things that plug into the framework and to the framework itself.
This patch eliminates all use of the word engine that meant the
framework. It leaves uses of the word that meant "module".
Change-Id: I6b9b50e2f045ac39f2a74d0152ee8d6269be4bd1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13918
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
spdk_accel_module_finish
Also move it into the internal header that defines the interface used by
modules.
Change-Id: I3aeb41e643f27a69556099cb8d166f64c9e5d67f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13917
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
For non-fabric controllers, the corresponding I/O qpairs are simply
re-enabled at controller reset.
This had a issue when I/O qpairs span multiple threads and poll group
is used.
spdk_nvme_ctrlr_reconnect_poll_async() calls
nvme_transport_ctrlr_connect_qpair() with qpair->async being false.
Then nvme_transport_ctrlr_connect_qpair() calls
spdk_nvme_poll_group_process_completions() until the qpair is connected.
spdk_nvme_poll_group_process_completions() may poll other qpairs.
This may cause I/O to complete on a wrong thread.
For PCIe controller, spdk_nvme_poll_group_process_completions() calls
spdk_nvme_qpair_process_completions() simply for each qpair.
Hence change nvme_transport_ctrlr_connect_qpair() to call
spdk_nvme_qpair_process_completions() if the controller is non-fabrics.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ieb270c2fb154124021ef6d25577b817d05e5ca9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14295
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Writing to unallocated cluster triggers copy-on-write sequence. If
this cluster is backed by zeroes device we can skip the copy part. For
a simple thin provisioned volume copy this shortcut is already
implemented because `blob->parent_id == SPDK_BLOBID_INVALID`. But this
will not work for thin provisioned volumes created from snapshot. In
this case we need to traverse the whole stack of underlying
`spdk_bs_dev` devices for specific cluster to check if it is zeroes
backed.
This patch adds `is_zeroes` operation to `spdk_bs_dev`. For zeroes
device it always returns 'true', for real bdev (`blob_bs_dev`) always
returns false, for another layer of `blob_bs_dev` does lba conversion
and forwards to backing device.
In blobstore's cluster copy flow we check if cluster is backed by
zeroes device and skip copy part if it is.
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I640773ac78f8f466b96e96a34c3a6c3c91f87dab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13446
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
There's no reason for this parameter to be non-const and it makes this
functions pain to use when you want to hardcode a specific sock
implementation.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifed4426a02ab54cbd51c8a2051b1eac010f86db9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14303
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
If initiator and target run on the same application, and initiator
uses SRQ, target may get async events for initiator, e.g.,
IBV_EVENT_QP_LAST_WQE_REACHED unexpectedly.
The reason is initiator and target may use the same device
simultaneously and only target polls async events.
Target sets attr.qp_context to rqpair when creating QP, but initiator
sets attr.qp_context to NULL when creating QP.
Hence one simple fix is to ignore async events whose qp_context is
NULL.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Id9ead1934f0b2ad1e18b174d2df2f1bf9853f7e1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14297
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
By the previous patches, a qpair is destroyed after it is actually
disconnected.
But after the qpair is destroyed, it is checked if drained by using
rqpair->current_num_sends and rqpair->current_num_recvs.
However, if the qpair is the last of a poller of a poll group,
CQ is destroyed before checking if the qpair is drained.
If CQ is destroyed, at least rqpair->current_num_recvs is not updated,
and we may get one second timeout.
This should be avoided.
Hence, destroy the qpair after it is disconnected and drained.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ibd6c83e8a3e7b6e11e9b45cee42669da6d42a621
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14278
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
If the being disconnected qpair is the last of a poller of a poll group,
CQ is destroyed and the poller is released before the qpair is actually
disconnected.
This patch destroy CQ and release the poller after the qpair is actually
disconnected.
One exception is when spdk_nvme_ctrlr_free_io_qpair() is called to a
connected qpair. In this case, the qpair is removed from a poll group
before the qpair is actually disconnected. In this case, destroy CQ and
release the poller when the qpair is removed from the poll group.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Idf266bbb6dbb40f04ae6313db724fabf80865763
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14253
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We have two cases to call nvme_rdma_poll_group_put_poller().
For consistency, make the two cases the same sequence.
This will make the next patch easier. The next patch will release
poller from poll group when qpair is actually disconnected as
possible as we can.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I4178113d5277240e287e83a57e97cf32fd0f7457
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14252
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds ability for FTL to startup after clean shutdown.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I2f1b83bb3eb1487b6665c95e76c48881e8899b16
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13364
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds additional debugging functionality - ability to check the validity
of all L2P entries and valid map to check for inconsistencies after FTL
startup. Since this is a very time consuming process, it's controlled by
an environment variable and not executed during normal operations.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4766a1576c058f69fa047f45d2d8be6d0ad0b3cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13363
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds necessary functions for setting up the state of FTL components
based on loaded in metadata.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I3a4c05230c877850e61d4f31d495d38121d27b3f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13362
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds the ability to persist only the most important metadata. The rest
is stored in shared memory.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4084c04ba09115a7a08ff66fd33552a2ec60d801
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13360
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add an extra step during FTL shutdown to save all metadata.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Idc2f77e15bbd02028548cc88355cd450175830e8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13359
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Flat L2P (all L2P in memory) needs to be specifically built against, due
to large memory consumption for big devices.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ib8906e10868455f88725b69b2b033b70a9f7256c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13358
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adds eviction of least recently used pages from the L2P cache - dirty
pages will be persisted.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ic646f7e9da777d077b5cb9b409c3f03ef05b1273
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13357
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds paging in from the cache device to memory.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I250009d12e9ed5ad52ee861ec5157cf983cf8cfc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13356
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
There is a set amount of pinned pages available. If exceeded they will be deferred
and processed in the future, using eviction logic.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ic642a5870db009ccf57152dd8a4178a6b2098ee1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13355
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This commit also introduces ranking pages, based on usage for determining the least used
page to be evicted.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Iaf3812177b61376bb38aa209e4ba8576d784ffb5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13354
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
L2P cache allows for partial storing of L2P in memory, paging in and out
as necessary, lowering the total memory consumption.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I727fec9d2f0ade4ca73e872d62a2ec10cfdb0a88
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13353
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
vhost_blk_start and vhost_scsi_start are now just
a single vhost_user_session_send_event() call, so
make this more generic by adding a top-level
start_session function.
Now this function will do the
vhost_user_session_send_event(), using the
user_dev_backend's start_session function pointer.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia89ba15011e231f0474405fb7225e713dcc920bf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14327
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Currently scsi sets it's svdev from the vhost
thread context, while blk does it from the
spdk thread context.
Make scsi match what blk does, to make the code
more consistent. This also will allow for an
upcoming simplification.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I609513bc8e05b49dd9455f2f61ba0cedc35236e6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14326
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Add parsing json as invalid cases:
1.json content that not enclosed in {}, it should be parsed as invalid, e.g.
"abc":"not encloesed in {}"
2.json content that 'subsystems' not associate with array, it will report error and return failure, e.g.
{"subsystems":"123"}
3.handle other invalid json formats, report and return failure, e.g. duplicate keys.
Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}.
json config with content:
1."not enclosed in {}"
2."'subsystems' not be an array"
3."duplicate key in json"
and some other invaild cases will be regarded as invalid json config, and will fail to start app.
Fixes#2599
Signed-off-by: tongkunkun <tongkunkun_yewu@cmss.chinamobile.com>
Change-Id: I02574c9acd7671e336d4c589ebbff8ed21eb3681
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13754
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Added a new RPC, vmd_rescan, that forces the VMD driver to do a rescan
of all devices behind the VMD. A device that was previously removed via
spdk_vmd_remove_device() will be found again during vmd_rescan.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ide87eb44c1d6d524234820dc07c78ba5b8bcd3ad
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13958
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Added new RPC, vmd_remove_device, that allows users to remove a PCI
device managed by the VMD library simulating a hot-remove.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifb84818ce8d147d1d586b52590527e85fe9c10de
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13957
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
It'll make it possible to remove a PCI device from within the callback.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I4cea2207a29bb145aee968715e873076a8c0993c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13956
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
This doesn't work anyway and can cause creating duplicate bus objects if
vmd_scan_single_bus() is called on a parent bus with previously
allocated child buses. Also, while here, removed a few unused functions
and flags in struct vmd_adapter.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ic757070188157d9851f648acd074ca4943a14c39
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13955
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
This is done in order to avoid having to decrement this counter in case
of a failure. Also, it makes the result valid for the few error cases
when we didn't decrement it.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia944fb8b810ce69caa8db5bc7c941e0905c9d3bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13954
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
This makes it possible to notify other PCI device providers (VMD) that a
PCI device is no longer used. The VMD will driver will unhook that
device and free any resources tied to it.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I42752afbb371a1d33972dac50fd679f68d05b597
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13887
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Idc9c7d0e5d0ebce8278e089bcfe5b7f76b86c270
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13953
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This patch implements the callback for attaching devices behind the VMD
with a given PCI address.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I07cf92c94cc7e6d3c8e31af7a8615e9a4ca641bf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13886
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
It makes it possible to call this function even if the VMD library
wasn't initialized.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I3d0f4677c4a1189f9d8acf07baee50a4e2050459
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14260
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Now that we have a attach_device() callback, the devices can be hooked
during spdk_pci_device_attach(). With DPDK, driver->cb_fn() is called
in pci_device_init(), so we need to do the same in
spdk_pci_hook_device().
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iada8b83ce7592aa62561530192072a50ec3a904b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13884
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This allows to free resources tied to a vmd_pci_device that isn't on the
dev_list or wasn't hooked to the PCI driver. Also, use that function
whenever a vmd_pci_device is freed instead of regular free().
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifca177a7eb6d8180d6f2ee2a9d9e36d58810e8ad
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14259
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
That way, we don't have to do TAILQ_REMOVE if vmd_assign_base_addrs()
fails.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id7a5df2093e4f9dfc95ee1fe415eb644c61bc971
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14258
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I79c35600fc9a758bbd9d58393b7eb98c8ac82acc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14257
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
It'll make it easier to reuse this part of the code.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id26f3f00abeeea6205df4f44689ffab1d367d777
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13885
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
The primary motivation for this patch is to allow the VMD driver to be
notified of when users wants to attach a device under a given BDF and to
make it more similar to the regular PCI path. Currently, the way the
VMD driver scans for the devices is a little bit different. The initial
scan is done during initialization and there's a separate poller for
checking hotplugs. Also, there's no device_attach() interface, so with
hotplug poller disabled, it isn't possible to attach to a device not
present in the initial scan, even if the BDF is known.
This causes a few issues. First of all, the VMD library isn't notified
when a device is stopped being used (i.e. user calls
spdk_pci_device_detach()), so when such a device is hotremoved, it never
gets unhooked. But we cannot simply add a spdk_pci_device.detach()
callback, as this would break cases when user detaches a device (without
hotremove) and then tries to reattach it again (via
spdk_pci_device_attach()), as the VMD doesn't get notified about the
device_attach() call.
So, in order to resolve this, a device_attach() callback is added, which
will notify the VMD library that the user wants to attach a device under
a specific PCI address. Then, in subsequent patches, a
spdk_pci_device_provider.detach_cb() callback is added to make sure that
devices are unhooked once they're no longer used.
Once that is done, it'll be also possible to get rid of the VMD hotplug
poller by adding something like scan_cb() to spdk_pci_device_provider and
call it from spdk_pci_enumerate().
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I084a27dcd12455f0f841440b7692375e80d07e84
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13883
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Hyper-V NVMe SSD controllers require admin queue
size to be even multiples of a page. Add quirk to
adjust the admin queue size if user overrides the
default value to something other than an even
multiple.
As part of this change, set the quirks earlier
when constructing a pcie controller, so that the
quirks value can be used in the generic
nvme_ctrlr_construct() function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I417cd3cdc7e3ba512ec412f4876b0e0b7432341c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14220
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The last line doesn't need the line break, otherwise
it will wrongly include the next line.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I06257b18d25c060b7c6bb00853fa44963fe5b439
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14241
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Better not to cache a value especially for there's an error return.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I3b243a66f4db9af34bc2ea01bafdac33004be128
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13650
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This was correct back when we only supported PCIe, but doesn't
in the newfangled world of fabrics and vfio-user.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I565edd2dab1eff862844585df8c25da508e4816d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14136
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Adds a debug function, that scans the whole P2L of band, when
it's getting closed. The P2L is compared against both L2P and
valid map to check for any discrepancies.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ia4d7be65415e6af3752d676de69b6fdcb73effb4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13352
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Utilize the valid map when picking physical blocks to compact/relocate,
speeding up the process.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I860e3cf25a5907591e4f3043def67156fec8b0df
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13351
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adds P2L validity map tracking - a bitmap marking all physical LBAs
as containing valid (current) user data or not. A clear bit denotes
the location has no valid data and may be skipped during relocation
or compaction. A set bit means it may have valid data (it's still
necessary to do the necessary comparision against L2P).
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I6a831a97b3080eb7c880d9c4feab41b523467885
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13350
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adding API for the bringup part of fast shutdown/startup. Adds
shared memory utilization for necessary functions during initialization.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Iab2da102fd0ccaa56fbdb9b3c765be5eeefff145
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13349
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adds API for fast shutdown - the ability for FTL to skip most
of the metadata persists made during clean shutdown, and relying
on their representation in shared memory instead. This allows for
faster update of SPDK (or just FTL, assuming no metadata changes),
with downtime reduction from 2-5 seconds to 500-1000 ms (for
14TiB+800GiB base and cache drives).
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5999d31698a81512db8d5893eabee7b505c80d06
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13348
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Helper functions which determine which md regions will be stored
in shm.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I94cbfca66dfb56457a350874dbd1de63a2e07661
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14159
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Stores P2L map of open bands in shared memory, allowing for
faster recovery times from application crash.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I519441af05e4d0f57768835bf01c800556873c58
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13347
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Allows for using shared memory in memory pools. Adds API for
accessing such pools after dirty shutdown (claiming them, ie.
marking an entry as actively used; calling the
ftl_mempool_initialize_ext will reclaim all unused entries back
to the pool). Also introduces API for accessing objects, since
using direct pointers is not possible (as addresses may change
inbetween application startups).
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5325b39d68aef7e231945cee9d92c925cab2fb2a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13346
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Main use case is to allow for keeping it in shared memory, to
speed up the recovery time after application crash.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I36b6b8331cd6483c5bd202e5f9103c351d705da8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13345
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Relocation will
1. Read LBA map of a given band
2. Pin the LBAs
3. Issue writes of valid LBAsto the new location
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ie753a790e56a86bfa1e451b5eda78b88eeacd3cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13344
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Multiprocess is only supported by a few libraries (e.g. NVMe driver).
Other libraries that don't support it will often fail on mempool
initialization when running as a secondary process, as the mempools are
already created by the primary process. But the error messages are
vague and don't indicate why this happened. So, this patch adds a check
to see if a mempool exists after spdk_mempool_create() fails and prints
an error message informing users that multiprocess is unsupported.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I6f915a94266e64dda380e3b269424cc579372a10
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14234
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
When we specify source address for admin and I/O qpairs,
rdma_resolve_addr() succeeded only for admin qpair and failed for
following all I/O qpairs because rdma_resolve_addr() returned
-EADDRINUSE.
To reuse source address among multiple qpairs, set the REUSEADDR option
for each CM ID before executing rdma_resolve_addr() if source address
is specified.
We may miss something. Even if rdma_set_option() fails, execute
rdma_resolve_addr().
Fixes issue #2604
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: If03f82d4499cf83c0e428a62e91c9d9e6aad28e0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14229
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Add indication of support for compare, compare & write and abort
in json bdev info dump.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ifc8dc1a1b180f08fcd9e9d58684eab1fd50356ff
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14137
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Commit a119799b ("test/nvme/aer: remove duplicated changed NS list log")
changed the nvme driver to read the CHANGED_NS_LIST log page before
calling the application's AER callback (previously it would read it
after).
Commit b801af090 ("nvme: add disable_read_changed_ns_list_log_page")
added a new ctrlr_opts member to allow the application to tell the
driver to not read this log page, and will read the log page itself
instead to clear the AEN. But we cannot add this option to the 22.01
LTS branch since it breaks the ABI. So adding this API here, which
can then be backported manually to the 22.01 branch for LTS users
that require it.
Restoring the old behavior is not correct for applications that
want to consume the CHANGED_NS_LIST log page contents itself to
know which namespaces have changed. Even if the driver reads the
log page after the application, that read could happen during a
small window between when a namespace change event has occurred and
the AEN has been sent to the host. The only safe way for the
application to consume ChANGED_NS_LIST log page contents itself
is to make sure the driver never issues such a log page request
itself.
Fixes issue #2647.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaeffe23dc7817c0c94441a36ed4d6f64a1f15a4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14134
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Fixes #issue 2636.
The existing allocation method (nvmf_rdma_get_optimal_poll_group())
is traversal and unperceived link disconnection. A more fair method
considering the number of real-time connections to allocate a poll
group is implemented.
Signed-off-by: liuqinfei <18138800392@163.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
Change-Id: Ic1e6283e386dbb0dd6655bedebe26aeedb16c333
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14002
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
If the bdev does not natively support compare we use
the fall-back which performs a read instead of a compare
operation. We then compare the results of the read with
the buffer provided by the user. In case the bdev has
metadata, there are two options:
1) md is interleaved -> the md will be part of the data
buffer allocated for the read and compared accordingly
2) md is separate -> currently we do not compare
the metadata but just ignore it.
This patch fixes 2) by comparing the md buffer after
the read is done.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: I1018b8c02540bffcba69408eb283bdc8f06bb747
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14132
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
bdev_io is allocated from a memory pool and is not zeroed on reuse.
So set bdev_io->u.bdev.ext_opts = NULL for io ops where it is not
supported (yet) so we can test against it.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ia579ea6b0787cf62572ea3a6bf2251867602e952
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14056
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Adds functions for reading end metadata and initializing band reloc state.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I3d12c4a7edd36f0437bf10316114c83efe449f0f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13343
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
After chunk is compacted it can be moved to the free state, able to be used
for new user IO again.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I7f9c341169b171ee246c5aa161d74903b91bdc2f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13338
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
During compaction FTL moves valid user data from the nv cache drive to
the bottom device.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ia200af39cec80014fac3a10f20d2859b10a81088
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13337
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add writer - tracks and manages band state transitions and write pointer as IO
is issued to it.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I5f878dc15bc1c1ac84835f75fe440672fad541d5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13335
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This allows mapping an nvme_request back to the
nvme_bdev_io.
This requires bumping up the max number of arguments per
tracepoint. 5 was previously chosen as max since it
exactly fit in 64 bytes (1 cacheline) when all
arguments were stored as uint64_t, but now that we
support uint32_t arguments we can afford extra
arguments when some of them are uint32_t. I've
bumped it to 8 so we can avoid having to touch
this value multiple times if we find some cases
where we need 7 or 8 args.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie2ef5e59d10549860b47542e68c1c34efa63047f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13995
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
This will allow us to map spdk_bdev_io events
to nvme_request events coming in a future patch.
Since we pass the nvme_bdev_io to the nvme driver
(not the spdk_bdev_io), we need to add tracepoints
for the nvme_bdev_io so that spdk_trace can
do the spdk_bdev_io->nvme_bdev_io->nvme_request
mapping.
An alternative would have been to pass the spdk_bdev_io
as the cb_arg to the nvme driver, but that change
seemed to invasive, and I think we will find other
uses for the nvme_bdev_io events anyways.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id7519e689b01875093359f41a1ca2af912061a8b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13994
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Calculates general priorities and trigger points for writers
(gc and compaction) dependent on number of free bands. GC will
be started at SPDK_FTL_LIMIT_START level, while at SPDK_FTL_LIMIT_CRIT
compaction needs to be stopped and only GC is allowed to work.
This is done to make sure FTL doesn't run out of free bands and deadlock
itself.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I1aab98503c2e79e97f8e4e9fb1257530fa9770e2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13326
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We'll likely need this eventually to address silicon version
specific workarounds.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ie6957674113cf0c7b7d695b468c694668ebbf2bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13571
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is targeted to fix the following error seen under clang:
ftl_nv_cache.c:54:1: error: unused function 'nvc_data_blocks'
[-Werror,-Wunused-function]
nvc_data_blocks(struct ftl_nv_cache *nv_cache)
^
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I11d52e76df5872819770d9468b6fa4ae54d8927c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14055
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
In multi-process, we need to make sure we don't
complete a register_operation in the wrong process. So
save the pid in the nvme_register_completion structure
when it is inserted into the STAILQ, then only complete
operations where the pid matches.
Fixes issue #2630.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I58c995237db486fecdd89d95e9e7a64379d0b0e5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13940
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
To prevent the processing of outstanding commands from starving the rest of the system
Fixes: #2586
Signed-off-by: Chen, You <you.chen@intel.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: I392db2359408cdef32cc1f46b76ecd94f0c3332c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13685
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This ensures that when fields are added, that the
size of the structure will change, ensuring different
versions of the structure can be detected using
sizeof.
Adding -Wno-address-of-packed-member to Makefiles
here, although we should consider disabling this
warning globally in SPDK just like DPDK.
Suppress abidiff errors around spdk_app_opts -
structure size and offsets of all existing members
were unchanged, so there is no ABI breakage here.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2249eddb604d7b44180cadb92ba30edcd946b9bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14091
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Similar to the disable_read_ana_log_page ctrlr_opt,
this enables the application to tell the NVMe
driver to *not* read the CHANGED_NS_LIST log
page in response to a NS_ATTR_CHANGED AEN, and
will do the read itself.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie447734187d4a4cb95ceef6e0131b640b8ba5984
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14088
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Various opts structures in SPDK have a size member, to enable
ABI compatibility should fields be added in the future.
But this requires the strucures to be packed, otherwise for
example a structure may be padded at the end, and a new
field added may just consume some of that padding.
So add STATIC_ASSERTS for the current sizes in this
patch. Upcoming patches will make the structures packed
and add in reserved fields to fill in holes.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9107d01d7b533f8542385a3538894bcd9f8c465d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14086
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Mellanox Build Bot
Wait for all bdevs to finish examination before
proceeding with the spdk_bdev_finish shutdown
logic. This ensures the bdev layer and its
modules are not trying to examine bdevs after
the bdev layer has reported it has shut down.
Theoretically, bdev modules could all defer their
fini callbacks until any outstanding examinations
are complete, but it is WAY simpler to just use
the existing spdK_bdev_wait_for_examine API
instead.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If90cc2a786281d348b82de8beb17ac37ba269c64
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13850
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
If the queue was on another poll group, we need to send a message back
to the admin CQ's thread to post the completion from the correct
context.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I997987d5d6b822a1a5124f54fc29ce5d7f03190d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14057
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Swapnil Ingle <swapnil.ingle@nutanix.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
spdk_nvme_qpair_process_completions() had called
always _nvme_qpair_complete_abort_queued_reqs() at its end.
However, the call was accidentally removed by a commit
59c8bb527b
to fix an issue.
By this removal, aborting request was not completed for some error
cases.
Fix the degradation by restoring the call.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I0099eb7a008f823e1282576504423cdc248911d7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14045
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
glibc 2.36 added arc4random(), which breaks
the SPDK iSCSI build since it always implements its
own arc4random() implementation on non-FreeBSD OS
(meaning always on Linux).
So instead add a CONFIG_HAVE_ARC4RANDOM and remove
the explicit FreeBSD dependency - this will work on
FreeBSD as well as Linux with >= glibc 2.36.
Also fix check_format.sh, so that it does not
enforce spdk/stdinc.h checks on code snippets in
the configure file.
Fixes issue #2637.
Reported-by: Karl Bonde Torp <k.torp@samsung.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iab9da8ae30d62a56869530846372ffddf7138eed
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14028
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Avoid putting a new req on the outstanding_reqs
TAILQ until we know it can be initialized
successfully. This avoids adding to the TAILQ
only to remove it just after.
This allow simplifies the outstanding_reqs TAILQ
handling, since reqs are now only inserted and
removed in one place each.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5ccc41c14abd541ffcf2a602246e0671386840c7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13991
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We were using "TR" for "tracker" previously, but
we are tracing the nvme_requests, not nvme_trackers,
so use the right names for the trace object to avoid
confusion.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia3886d74b162138c2cdbe0017224d9494f74966c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13990
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This allows us to pack more arguments into the same
amount of shared memory, for cases where those arguments
don't need a full 8 bytes.
1- and 2-byte sizes not supported for now, variadic args
do automatic promotion of types smaller than int, so
support for those may need more work.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iec56cfa851b408a77d7995126d2111b0bf3d7f95
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13999
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The public interface of lib/accel is now include/spdk/accel.h
Change-Id: Id94f623a494eb1b524b060f4413f633073ea7466
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13916
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
They no longer need to see the definition of this structure.
Change-Id: I3e3bb5942a50da22e0bf34aa8c10b9d812f42d2f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13915
Community-CI: Mellanox Build Bot
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This expects the full size of the task for each module. This only worked
because the software module returned the right size.
Change-Id: I481cfad8b4bb9c3748301bdacd90e7f44fd2d878
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13913
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
This will help keep the mixing of this code with the framework code to a
minimum.
Change-Id: I5937ebd84f32068456cdf2b9e03d3e194c760a87
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13912
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
spdk_internal/accel_engine.h will become the API for accel modules. Move
anything in there that a module doesn't need to see into
lib/accel/accel_internal.h
Some of the software fallback definitions didn't even need to be in a
header and were moved to accel_engine.c
Change-Id: Idb8b12b1c0c1de3d462b906e3df3ba9ee8f830b8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13911
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
These are 1:1 - they do not need to be separate objects.
Change-Id: I74ab52863f911d9be59ce98e1525302b5bd40846
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13910
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
There is a race condition if we call this function in the
polling mode when running with multi-cores, same as other
places where the function is called, we only kick controller
in interrupt mode, also in `vfio_user_ctrlr_intr`,
`ctrlr->sqs[0]` may be set to NULL after the controller
poll call, so return earlier for this case.
Change-Id: I03a7b74a39c966a2b8be610bca0e492d902f6b08
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13696
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Note, this change only sets defaults for the ID/KEY,
more specific use cases like NVMe/TCP may set the ID and KEY on a per connection basis.
Also simplify PSK identity string, that isn't NVMe focused.
NVMe libraries using this will need to construct more complicated
identity strings and pass them to the sock layer.
Example:
rpc.py sock_impl_set_options -i ssl --psk-key 4321DEADBEEF1234
rpc.py sock_impl_set_options -i ssl --psk-identity psk.spdk.io
./build/examples/perf --psk-key 4321DEADBEEF1234 --psk-identity psk.spdk.io
./build/examples/hello_sock --psk-key 4321DEADBEEF1234 --psk-identity psk.spdk.io
Change-Id: I1cb5b0b706bdeafbccbc71f8320bc8e2961cbb55
Signed-off-by: Boris Glimcher <Boris.Glimcher@emc.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13759
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
There were a few error cases that weren't caught
as errors, meaning the "invalid tpoint mask" string
wouldn't be printed.
But also change it so that when an invalid tpoint mask
is specified, it fails spdk_app_start and causes the
application to exit, rather than just silently
stopping processing of the tpoint group mask string.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I567a4eee740559914e089dca7d7c3865ed9ce35b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13986
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
For the case `nvmf_subsystem_remove_listener` RPC call when VM is connected,
we should not relisten the accept poller, because the endpoint will be
destroyed for this case.
Change-Id: Icf8299f26a3bbf7bbe44fd01edb4ede344692d25
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13548
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: John Levon <levon@movementarian.org>
ioccsz is specific for fabrics. spdk_nvme_ctrlr_is_fabrics() returns
true for custom fabrics transport. Hence we can use
spdk_nvme_ctrlr_is_fabrics() safely in nvme_ctrlr_update_nvmf_ioccsz().
Before this change, in the unit tests, ctrlr->trid.trtype was set to
zero at initialization. After this change, for most cases,
spdk_nvme_ctrlr_is_fabrics() should return false for most cases.
SPDK_NVME_TRANSPORT_PCIE did not work. Hence, initialize
ctrlr->trid.trtype by SPDK_NVME_TRANSPORT_CUSTOM_FABRICS instead.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I4bedcab4a9f2876c1c9463ff10ad0966754f1713
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13948
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
rdma_reg_msgs() was replaced by ibv_reg_mr() recently to support
persistent PD per RDMA device. The difference between rdma_dereg_mr()
and ibv_dereg_mr() is only return value and errno. For consistency,
replace rdma_dereg_mr() by ibv_dereg_mr().
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I55e0743690e74f9510863bfa122a75d0632dce4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13949
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Get a PD for the device from the PD pool managed by the RDMA provider
when creating a QP, and put the PD when destroying the PD.
By this change, PD is managed completely by the RDMA provider or the hooks.
nvme_rdma_ctrlr::pd was added long time ago but is not referenced
anywhere. Remove nvme_rdma_ctrlr::pd for cleanup and clarification.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: If8dc8ad011eed70149012128bd1b33f1a8b7b90b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13770
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
SPDK NVMe RDMA initiator used the default PD per RDMA device. Default PD
may be changed when all QPs for the RDMA device are destroyed and created
again.
For multipath, the RDMA zero copy feature require the PD per RDMA device
to be persistent when all QPs for the RDMA device are destroyed and
created again.
Maintain such persistent PDs in this patch.
Add two APIs, spdk_rdma_get_pd() and spdk_rdma_put_pd().
In each call of two APIs, synchronize RDMA device list with
rdma_get_devices().
Context may be deleted anytime by rdma-core. To avoid such deletion,
hold the returned array by rdma_get_devices().
RDMA device has PD, context, ref. count, and removed flag. If context
is missing in rdma_get_devices(), set the removed flag to true. Then,
if the ref count becomes zero, free the PD and the RDMA device.
The ref. count of a RDMA device is incremented when spdk_rdma_get_pd()
is called and decremented when spdk_rdma_put_pd() is called.
To simplify synchronization, sort the returned array by
rdma_get_devices().
To avoid resource leakage, add destructor function and free all PDs
and related data at termination.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I093cb4ec2c7d8432642edfbffa270797ccf3e715
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13769
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
An earlier commit added ctrlr_ready into struct
spdk_nvme_transport_ops. However, the major SO
version was not increased.
Fixes: 3dd0bc9e (nvme: Add transport controller ready step)
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Id903634f9aaf5bdaa62fd30e92a4fb39a985b86f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13981
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Instead of passing each parameter to create a module, just have the user
make one and pass it in. This makes it easier to change the module
definition later.
Change-Id: I3a29f59432a6f0773129d7b210fbc011175b2252
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13909
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Masked by how accel_perf was doing decomp verificiation which is
changed in the next few patches and verifies these fixes.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Icb03fc169bf8d2f05396addaf1db56d6de1827d1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13038
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Docs explaining how to use the RPC are in the next patch in the
series.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I7dab8fdbeb90cdfde8b3e916ed6d19930ad36e66
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12848
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
submit_inflight_desc() actually do some meaningful work, so when it really process tasks, the poller should return BUSY status.
Signed-off-by: YafeiWangAlice <yafei.wang@samsung.com>
Change-Id: I2103cea6d28e8b355dad4ddd603d917f10e44c08
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13486
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
There may be for_each operations outstanding on an
io_device when it is unregistered. Currently we just
return when this happens, not unregistering the
device but also not notifying the caller that this
happened (since it returns void, and the callback
function doesn't have a status parameter either).
We could just push this responsibility to the caller,
to never unregister an io_device if it knows it has
outstanding for_each calls waiting to complete. But
I think we can simplify this a lot by just handling
this inside of the thread library. Mark that the
device is pending registration, and unregister it
(on the original requesting thread!) when the
for_each count gets back to zero. Also don't
allow any new for_each operations either.
Note this requires a bit of refactoring on the
thread unit tests, since it is now possible to
unregister a device with outstanding for_each
operations.
Fixes issue #2631.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I978f2d99a25e65d2b7d71ce9b1926a79a6c94263
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13890
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
If spdk_for_each_channel is called on a device that doesn't
exist, we need to set a non-zero status (-ENODEV in this
case) to the completion function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I898ad5ea499fb6087338b621b2befcadd6a05414
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13889
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This has been implicitly required before, and all
in-tree apps (except accel_perf) set it, so let's
explicitly require it. This name gets used for
things like the shm name for spdk trace event file.
While here, add the name for accel_perf.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I47a22466550d4b31bacafee58d30339b4f22f4b4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13876
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
When decoupling a snapshot from its parent, we need to clear its parent.
So we should remove the xattr BLOB_SNAPSHOT. Modifying the xattrs of a blob
only works if its metadata are not in read-only mode.
By default, a snapshot is in read-only mode so this operation fails. When we
later want to delete the snapshot, we will see that it has a parent, so we will
try to remove the snapshot from its parent's clones list. This will cause a
crash.
The fix is to remove the BLOB_SNAPSHOT xattr only after setting the snapshot's
metadata in rw mode.
Signed-off-by: Alex Michon <amichon@kalrayinc.com>
Change-Id: I80efa6dd3dcb38b4c738ce2e97aa2ffc281cefa5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13723
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
DPDK may use this NULL pointer to access its member,
And then got segmentation fault. But we only need it
exit or report normal error.
To minimize the impact, and to prevent these going on,
we add check the error return for creating NULL mempool
in spdk_thread_lib_init_ext in spdk_reactors_init.
when error returning from spdk_thread_lib_init_ext in spdk_reactors_init.
It contains thread_lib_init which reports error for failed mempool.
Thus, codes will return and will not cause segmentation fault.
Fixes issue #2620.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I63369fdaeb231196e8f8daa826eb5b057ed829b8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13842
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Here should return -ENOMEM, and other places are
changed.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: Id81cd7485733e66d996b1501061a45f774f2b51a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13863
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Parameter `num_queues` for virtio_scsi PCI device means
maximum number of queues, it SHOULD include the `eventq`
and `controlq`, while for `vhost_user` RPC call, it means
the number of IO queues, so here we use it as `max_queues`
in lib/virtio and add the fixed number queues for `vhost_user`
SCSI device.
Also fix `vhost_fuzz` to get `num_queues` earlier than
negotiate the feature bits.
Change-Id: I41b3da5e4b4dc37127befd414226ea6eafcd9ad0
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13791
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The `vhost_user` socket transport APIs are already in the
same source file, so just call the function directly.
No code logic changes in this commit.
Change-Id: If471b9b0166d43591fb8614e95a17473c964e87c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13789
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Similar with NVMe device driver, here `virtio` is a specification
abstraction library, `pci` and `vhost_user` are transports layer,
here we merge vhost_user.c and virtio_user.c into one new source
file `virtio_vhost_user.c` so that to make code more clear.
No logic change, just code movement in this commit.
Change-Id: I8e3e5c477e7c45e6eeebad240b8cc3c9476b86d1
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13788
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
While running into this function, even the subsystem can't be
destroyed due to error subsystem state, it's better to continue
the execution.
Continue to fix#2590, QEMU is stuck for the failure case, and
nvmf target should process such error because it may support other
normal subsystems at the same time.
Change-Id: Ib05e24996378b52070d2b760519f476f9b2d7e76
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13839
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The RPC provides a list of initialized engine names along with
that engine's supported operations.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I59f9e5cb7aa51a6193f0bd2ec31e543a56c12f17
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13745
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
In prep for upcoming patch that will provide an RPC to override
and automatic assignment of an op code to an engine.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I17d4b962fb376a77f97ce051a513679d0fba698e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12829
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
fix: If the first six characters of two scsi lun's name are the same,
such as aaaaaa0 and aaaaaa1, so do theirs naa identifier
Signed-off-by: Bin Yang <bin.yang@jaguarmicro.com>
Change-Id: I4e0541b372a0e20e95e0a24d62dd3d85b7abe230
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13824
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This is another preparation to create and use ibv_context and pd.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: Id594fa1ccb2daf535b1aaaef0a397bda2ec98578
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13710
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The following patches will create and use ibv_context and pd
explicitly instead of using default ibv_context and pd created
by rdmacm.
As a preparation, pass pd instead of cm_id to nvme_rdma_reg_mr().
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: Ifdcd18ed363b8ba4a23a920bf3559237e38821c6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13599
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
spdk in interrupt, reactor dosen't correctly handle exited threads,
causing vhost threads still in reactor's lw_threads list. The fix
will do cleanup thread when it's state becomes EXITED. Though it's
exposed in v22.05.x, but the master branch also has the problem.
We will do this as below:
(1) When thread's state becomes SPDK_THREAD_STATE_EXITED, reactor
process thread exits first.
(2) Then reactor do remove lw_thread and destroy it.
Fix issue: #2574
Signed-off-by: Apokleos <oliverliyn@gmail.com>
Change-Id: I3ac2681d70480563db3a0aee4aff61c2f272b140
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13706
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
If Controller Fatal Status (CFS) bit is set, there's no point in waiting
for CSTS.RDY and the only way to move forward with the initialization is
to perform a controller reset.
This fixes issues with test/nvme/sw_hotplug.sh when running under qemu.
It seems that during that test, qemu marks the emulated NVMe drives as
fatal, so if we didn't check CSTS.CFS, the initialization would time
out.
Fixes#2201.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I97712debc80c3dd6199545d393c0f340f29d33b2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13820
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Sometimes VM may get a kernel panic when starting, and SPDK CI will kill
`nvmf_tgt` after 60 seconds, and for this exception, SPDK will raise an
assertion when destroying the subsystem, while here, we remove this
assertion and print the error information.
CI will still mark this case as a failed case, then we can use this error
information to understand error subsystem state in vfio-user.
Fix issue #2590.
Change-Id: I20b16f9e96a566730eca2dd9ea165645bd9160bd
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13773
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
this is a prework for further changes - with lock on generic layer
lock on specific transport (e.g. tcp, rdma) layer becomes optional
possibly it won't be required if some contract introduced on public
interfaces (to be considered)
- spdk_nvmf_poll_group_[create|destroy]
- spdk_nvmf_tgt_listen_ext, spdk_nvmf_tgt_stop_listen
- spdk_nvmf_get_optimal_poll_group
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: Ib132babf9e7022342129fe795991cdad834e7f53
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13665
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When there are not enought transport buffers for
multi SGL request in state NEED_BUFFER, WRs
received from the data_wr_pool are returned back
to the pool. However rdma_req->data.wr.next pointer
still points to the first WR from the pool. Usually
it doesn't cause any problems since rdma_req will
try to fill buffers again, but when qpair is being
destroyed, all requests are completed forcefully.
When the request is completed and data.wr.next
pointer is not NULL, we'll try to put already
released WRs into the pool one more time.
That corrupts the pool and leads to undefined
behavior.
Fixes#2541
Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: I238b92eec132d8d845330362af6f335421177454
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13760
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The function `nvme_ctrlr_init_ana_log_page` is exactly
same with `nvme_ctrlr_update_ana_log_page`, so remove it.
Change-Id: I1ad51635f47cf95cfa6de217e3b9144885c3b74e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13652
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
allow DSA device to async offload crc32 calculation in nvmf-TCP
This patch can use DSA to accelerate crc32 computation, making
the io performance of TCP paths using crc32 approach the io
performance of TCP paths that do not use crc32.
Using SLIST to minimize the performance drop. SLIST has less
operation compared to TAILQ.
Thinking about memory thrashing, we should use the same memory as
possible to receive new PDUs. So, insert newly freed PDU in to head
is better.
The performance drop is within 1% compared to the TCP path without
crc32.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I480eb8db25f0e730cb198ca5ec19dbe3b4d38440
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11708
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We have to process whole QoS queue on each QoS poll. It may contain
IOs that still have quota or not affected by QoS rules at all. If we
stop on the first queued IO, all IOs will be limited by the minimum
QoS rule even if they're not affected by this rule.
Here is an example and simple test. We have a NVMf target with Null
bdev and QoS configured with read bandwidth limited to 10 MB/s and
write bandwidth limited to 100 MB/s. First we start nvme_perf with
only write IOs and we see that reported bandwidth is 100 MB/s. Then we
start another instance of nvme_perf with only read IOs. We see that
reported read bandwidth is 10 MB/s but we also see that write
bandwidth also drops to 10 MB/s.
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I1edf09d038e65f873deef19ecb0f4bf9725a5ca5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13767
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Original implementation creates pollers and CQs for all discovered
devices at poll group creation. Device (ibv_context) that has no
references, i.e. has no QPs, may be removed from the system and
ibv_context may be closed by rdma_cm. In this case we will have a CQ
that refers to closed ibv_context and it may crash in ibv_poll_cq.
With this patch pollers are created on demand when we create the first
QP for a device. When there are no more QPs on the poller, we destroy
the poller. This also helps to avoid polling CQs that don't have any
QPs attached.
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I46dd2c8b9b2902168dba24e139c904f51bd1b101
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13692
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Both PCIE and VFIO-USER can use the same APIs to get IO queue
pair statistic data, so merge them here.
Change-Id: Iadf9ead2bd5abaf11d2ef5d1884acb67369f85bb
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13538
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
When a bdev is registered, it is examined by the bdev modules before the
bdev register even is notified.
Examination may be asychronous, e.g. when the bdev module has to perform
I/O on the new bdev.
This causes a race condition where the bdev might be destroyed while
examination is not finished. Then, once all modules have signaled that
examination is done, `bdev_register_finished` makes an invalid access to
the freed bdev pointer.
To fix this, defer the unregistration until the examine is completed by
opening a descriptor on the bdev.
Change-Id: I79a2faa96c1c893fc1cee645fbe31f689b03ea4a
Signed-off-by: Nathan Claudel <nclaudel@kalray.eu>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13630
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
