We do not support Soft RoCE anymore. Remove a workaround for Soft RoCE's
bug that we amy receive a completion without error status after qpair is
disconnected/destroyed. Then add a assert to check if rdma_req->req is
not NULL. This will simplify the code and the following patches.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I80c349053adc0f79679eaf8a5d7265d555d3c2b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14909
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The following patches will support SRQ and SRQ will be per poller.
We will need SRQ in nvme_rdma_cq_process_completions().
It is not possible to identify poller if poll_group is passed to
nvme_rdma_cq_process_completions().
Based on these thoughts, add poll_group pointer to poller and pass
poller to nvme_rdma_cq_process_completions() instead of poll_group.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I322a7a0cc08bdcc8e87e720ad65dd8f0b6ae9112
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14282
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
NVMe-RDMA target has a helper function get_rdma_qpair_from_wc() and
uses it to identify a qpair from a WC.
NVMe-RDMA initiator has a similar function
nvme_rdma_poll_group_get_qpair_by_id().
NVMe-RDMA initiator will support SRQ in the following patches, and
it will want to identify a qpair from a WC.
get_rdma_qpair_from_wc() of NVMe-RDMA target uses wc->qp_num internally
anyway.
However, the upcoming custom transport for RDMA will have to use other
variables of WC.
Hence, it will be convenient to pass WC instead of qp_num if we consider
future enhancements.
Based on these thoughts, for NVMe-RDMA initiator rename
nvme_rdma_poll_group_get_qpair_by_id() by get_rdma_qpair_from_wc().
remove unnecessary declaration, and pass WC instead of qp_num.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I01ead4730207e2c6ac53b83f151bd5f977a11465
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14279
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Poller will have more shared resources when SRQ is supported.
This is a preparation.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: Ic3d1cb93dde3f53653a9536a103e5518cebd58e1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14173
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
nvme_rdma_ctrlr_disconnect_qpair() does not poll the qpair until it is
actually disconnected if it is in a poll group even if its async mode
is disabled. Hence, spdk_nvme_ctrlr_free_io_qpair() removes the qpair
from a poll group when it is being disconnected.
On the other hand, I/O qpair is destroyed after it is actually
disconnected.
When SRQ is enabled and used, a SRQ is destroyed if the corresponding
poller does not have any I/O qpair after an I/O qpair is removed from
the poller.
In particular, if we use spdk_nvme_ctrlr_free_io_qpair(), a SRQ is
destroyed before the corresponding I/O qpairs are destroyed.
Destroying a SRQ failed because it is still referenced by I/O qpairs.
This bug was found when running the SPDK NVMe perf tool with SRQ.
The reason was we had nvme_rdma_poll_group_process_completions() to call
disconnected_qpair_cb after the qpair is actually disconnected.
However, it is ensured that nvme_rdma_poll_group_process_completions()
calls disconnected_qpair_cb for any disconnected qpair.
Hence, remove a check if qpair->poll_group is not NULL from
nvme_rdma_ctrlr_disconnect_qpair() and update the comment.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I0fde0d827eec3280e1cc5a0fce34d163a6069bc4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14908
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
With RDMA, the admin poller can experience a remote disconnect when
processing completions. The admin qpair will be disconnected to handle
this. The disconnect code path will manually complete queued aborts.
However, the completion callback for the abort will attempt to resubmit
other queued aborts from the queue, which will result in a very large
stack and can eventually cause a segfault.
The fix is to not resubmit queued aborts if the admin qpair is in any
kind of failed state.
Change-Id: I4a6f959232c8a1bd30c87ca50459014e556cbaa0
Signed-off-by: Vasuki Manikarnike <vasuki.manikarnike@hpe.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15114
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
A loop inside 'nvme_tcp_qpair_process_completions' makes
'max_completions' actually behaving like a minimum:
do {
rc = nvme_tcp_read_pdu(tqpair, &reaped);
[...]
} while (reaped < max_completions);
Before this change 'max_completion' constraint, in its true sense,
was actually not respected and a loop inside 'nvme_tcp_read_pdu'
could be executed indefinitely as long as a recv state changed.
To prevent this behavior, max_completion must be passed to
'nvme_tcp_read_pdu' and used as an additional exit condition.
Signed-off-by: Szulik, Maciej <maciej.szulik@intel.com>
Change-Id: I28da962f4a62f08ddb51915b5d0dae9611a82dee
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15136
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Some reset/disable paths are freeing the shadow doorbells without
switching the SQs back to BAR0. Fix this up, and add a small cleanup
when initializing the shadow doorbells.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ia5e5b91b7dc696a558eb0ad59cc554abced47cca
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14901
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
To support SQs allocated to a poll group other than the controller's
main poll group, we need to make sure to poll those SQs when we wake up
and handle the controller interrupt. As they will be running in a
separate SPDK thread, we will arrange for all poll groups to wake up
when we receive an interrupt corresponding to a vfio-user message
arriving.
This can mean needless wakeups: we don't (yet) have a mechanism to only
wake up the poll groups that correspond to a particular SQ write.
Additionally, as we don't have any notion of a poll group per
controller, this ends up polling all SQs in the entire poll group, not
just the ones corresponding to the controller we were handling.
As this has potential performance issues in many cases, it defaults to
disabled.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I3d9f32625529455f8d55578ae9cd7b84265f67ab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14120
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When crc32c is invoked with a multiple entry input iov,
only the last op has crc_dst set in order to write the final
crc value into the user supplied location.
spdk_idxd_process_events() for every successfully completed
CRC op writes the value into *op->crc_dst
UNLESS it is NULL.
The problem is that _idxd_prep_batch_cmd() that allocates
new ops left op->crc_dst uninitialized.
This results in a memory corruption (use after free)
in the following scenario:
1) op A is allocated an crc_dst is set to point to user memory X.
2) Op A is compeleted
3) User memory X is freed.
4) Ops B and C are allocated (chained), C has crc_dst set.
=> B reused op A memory and crc_dst still points to the
now stale user location (1)
5) B is complered, spdk_idxd_process_events() writes into X
as B->crc_dst = X.
Fix: _idxd_prep_batch_cmd() should initialize crc_dst to NULL.
Signed-off-by: Anton Eidelman <anton@lightbitslabs.com>
Change-Id: I9e7d57ec43a8fbcb3750906015a5cb7291278c35
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15115
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We were missing a check when ISAL uses the complete output buffer
on compression to determine whether it was s perfect fit or if
simply not enough buffer was provided.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I73532666f50cb9fbef3c42f6bfb25fc5c7de01c6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14874
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Prevent user from switching back to static scheduler after
different scheduler has been selected. Currently we
do not have a way to save initial thread distribution
configuration, so each time user switches from dynamic
scheduler back to static, the SPDK threads may end up on
different reactors. This would cause discrepancy in
performance statistics of SPDK managed by static scheduler.
Change-Id: Ic17a6be55eaea0e1a748f92e01f7075540403637
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15055
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This helps generate slightly better code in this function,
which can have a noticeable impact for high trace
event workloads.
Tested with bdevperf, single malloc or null bdev,
qd=32, 512B randreads on a single Xeon core.
Specify "-e bdev" to enable bdev trace events.
Null:
Before: 8.09M/s (123ns per IO)
After: 8.68M/s (115ns per IO)
Malloc:
Before: 4.21M/s (237ns per IO)
After: 4.34M/s (230ns per IO)
Note that each bdev I/O generates two trace events (START
and END) - meaning this change removes 7-8ns of overhead
for every 2 trace events, at least on my system.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7021b7f9e28b4a7cb16f8a97b4d4004ae165efd2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15096
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
When spdk_idxd_submit_crc32c() handles input
with multiple iovs (or multiple ops are generated
due to physically discontinuous buffers),
the first op has the original seed, while the
subsequent ops instruct the hardware to
to fetch the seed from the output of the previous op
(op->hw.crc32c_val):
void *prev_crc;
...
desc->flags |= IDXD_FLAG_FENCE | IDXD_FLAG_CRC_READ_CRC_SEED;
desc->crc32c.addr = (uint64_t)prev_crc; <<< virtual addr
The problem is the prev_crc is a virtual address,
so the hardware (at least with no IOMMU configured)
reports: DSA_COMP_HW_ERR1
spdk_idxd_process_events: Completion status 0x20
Solution:
Set crc32c.addr to the physical address of
the crc32c_val field in the previous desc.
Since desc->completion_addr already holds the physical address
of the dsa_hw_comp_record, we use this with the crc32c_val offset.
Signed-off-by: Anton Eidelman <anton@lightbitslabs.com>
Change-Id: I330e98c2f3fd6da5cb4fc03d0745df09a9ff0e0c
Signed-off-by: Anton Eidelman <anton@lightbitslabs.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14954
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
It allows the users to specify the path to the RPC socket on a NFS
mounted filesystem. This is necessary, because flock(2) on NFS requires
write access to place an exclusive lock.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: If197498ed5bdcb4e02c5f2f2b2c1ef388872c457
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14993
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
When Link Time Optimization is enabled, compiler can sometimes produce
additional warnings saying that some variables may be uninitialized.
To supress the warning it is enough to add explicit initialization
of the variable causing the issue, in this case '*module_name = NULL'
and "*writer = NULL".
Signed-off-by: Szulik, Maciej <maciej.szulik@intel.com>
Change-Id: I30492115b28a18554b08a6f575cbcc9538f3b848
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14849
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Fixes#2693
spdk threads should not be placed in interrupt mode
if the application does not have interrupt mode enabled.
This resulted in race condition, while reactor was placed
in interrupt mode, thread was scheduled on it.
Such operation is a valid one, but never should be attempt
to change the threads mode in this case.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I10b0bbacac1df812badb91b37064528f66743e51
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14815
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Patch below added copies of pci realted headers to keep
compatiblity with <= DPDK 22.07.
(1eb35ac) env_dpdk: add copies of 22.07 pci-related header files
Unfortunetly the rte_bus/bus_pci/dev headers from DPDK 22.07 are
not compatibile going back to DPDK 20.11.
The issues are:
- lack of RTE_TAILQ_ENTRY defined in rte_os.h
- rte_intr_handle being part of rte_pci_device rather than pointer
pci_dpdk_2207.c even before this patch is not binary compatible with
DPDK 20.11 - see pci_device_*_interrupt_2207() functions.
There would need to be another copy of headers matching that version
of DPDK to resolve this issue.
SPDK supports up to two latest LTS releases. Which right now includes
DPDK 20.11, but soon will be dropped due to DPDK 22.11 release.
Having compile time defines here, keeps the older DPDK working.
Meanwhile backwards compatiblity in SPDK is no worse than before.
The recent changes to env_dpdk, are aiming to improve support
with newer versions of DPDK.
Change-Id: If4dc601cb03e18c2cad61f3a93080e8265ca5fcc
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14795
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Add new bdev property split_on_write_unit which, if set to true, causes
writes to be split to match write_unit_size and fail if not aligned to
or not multiple of write_unit_size.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Id49f58a3288ddf5cfe4921ce4020ae4bcdd67298
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11390
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The client vfio_user library doesn't require this flag as
it is totally owned in SPDK, so remove it.
Change-Id: I8f7b1df18017ceac24dbb8a0417871f25f6bee0d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13895
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Previously SPDK use libvfio-user library to provide emulated NVMe
devices to VM, but it's limited to NVMe device type only. Here we
add SPDK vfu_target library abstraction based on libvfio-user which
supports more PCI device types.
We will add virtio-blk and virtio-scsi devices emulation based on
vfu_tgt library in following patches, actually this library can
support NVMe emulation too, due to the fact that the NVMe emulation
is already exist, so we will keep the NVMe emulation which based on
libvfio-user directly as it is.
Change-Id: Ib0ead6c6118fa62308355fe432003dd928a2fae9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12597
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This allows eliminating dpdk_pci_device_vtophys and
dpdk_pci_device_map_bar, reducing the amount of
code we need to maintain in the per-DPDK version
implementations.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I73d15eb75bf7fe8340d85494425e15651fec5425
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14722
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Break this function up into three APIs instead:
* dpdk_pci_device_get_addr
* dpdk_pci_device_get_id
* dpdk_pci_device_get_numa_node
This more clearly delineates the requirements we
have from the DPDK PCI device/driver APIs.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie585c8252d63c15c6e6884d60f8a064c3f0ab94f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14684
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Moving forward, we want to still be able to run against
<= 22.07 versions of DPDK, which exposed the necessary
data structures in public header files. But since we
will be building against newer versions of DPDK which
don't expose them publicly, we need a copy of the 22.07
header files in our tree.
Exclude these header files from astyle and POSIX include
file checks in check_format.sh
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Icd8a067af41a2ba031ce8f875a8a2b63f722ab69
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14683
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
This was a remnant from ages ago when we had rte_vhost
DPDK code copied into our repo. We actually have a file
named rte_vhost_user.c which is not DPDK code that was
getting excluded from astyle checking.
So this also includes the astyle violations that had
crept into this file. In a couple of places, change
the enum return type to int, this reduces astyle
confusion on function and if brace style.
Same applies to POSIX include checking - we don't need
to exclude rte_vhost_user.c from this either.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If3a25011ad54c694c15a91f7be66d862c765c5db
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14688
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
For example, in the calling from spdk_bdev_get_current_qd(), if
spdk_for_each_channel() failed to allocate struct spdk_io_channel_iter,
it will just return and the ctx allocated in spdk_bdev_get_current_qd()
is not released.
Instead to change the public API of spdk_for_each_channel() to return
the failed status to let the caller properly handle the NOMEM case and
release the allocation, it just adds the assert here.
Change-Id: I6a95207dd390586bdae4e86e5d550cdac709e10a
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14657
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
max_aq_depth should be not smaller than 2 or greater
than 4096
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I205fbb4345cfdc41ebaf30c953da263fe9f0e9a8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14691
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
max_queue_depth should be not smaller than 2 or greater
than 65536
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I0f2a4b8df6eb1b140a11936fc6929f1285a7d717
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14619
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Refine the macro definition name about queue depth and
prepare for next patch.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I85bee2528ae4ab70292fc11aa62d05bae0c28a77
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14664
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Delete bit masks from trace help (found inside
build/bin/spdk_tgt -h help text), as they do not
provide useful information, are much harder to
remember and use, and migh leave user confused.
Since we provide trace group names anyway, bit masks
are excessive.
Change --tpoint-group-mask parameter name to
--tpoint-group, because we do not provide
bit masks anymore.
Drop "default" tpoint group mask from help text,
since it does not enable any tracepoints and
may confuse the user.
Change-Id: I2ca780883dfa7822e76523e9ba1fc65a7bfe5a99
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14656
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When Link Time Optimization is enabled, compiler can sometimes produce
additional warnings saying that some variables may be uninitialized.
To supress the warning it is enough to add explicit initialization
of the variable causing the issue, in this case 'iovcnt = 0'.
Signed-off-by: Szulik, Maciej <maciej.szulik@intel.com>
Change-Id: I080b20a6008643ae78c8e3a6c2d183193ef6c1bf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14674
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Community-CI: Mellanox Build Bot
When data_local.num_async_events >
SPDK_NVMF_MIGR_MAX_PENDING_AERS, data_local.async_events
was already indexed by 256, and it was out of bounds.
Signed-off-by: Liu Xiaodong <xiaodong.liu@intel.com>
Change-Id: I15cfdeb9bc165de0c73fbc9171b0ce6d8689c0aa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14666
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
If the kernel is booted with the IOMMU enabled and Shared Memory mode
enabled (which are the expected boot parameters for production servers),
then the kernel idxd driver will automatically register a dedicated work
queue with the PASID for the process that opens it. This means that the
descriptors written into the portal for that work queue should be
*virtual* addresses.
If the IOMMU is enabled but Shared Memory mode is disabled, then the
kernel has registered the device with the IOMMU and assigned it I/O
virtual addresses. We have no way to get those addresses from user
space, so we cannot use the kernel driver in this mode. Add a check to
catch that.
If the IOMMU is disabled, then physical addresses are used everywherre.
Change-Id: I0bf079835ad4df1128ef9db54f5564050327e9f7
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14019
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
The DSA specification calls out that software must use a memory barrier
such as sfence prior to writing a descriptor or incorrect data may be
transferred during the operation.
Change-Id: I12f20e5a748e41616c7a542ccdb158c6b548eea4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14018
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
By doing the registration immediately upon mapping the BAR instead of
when the memory is inserted into the spdk_mem_map, we're able to
register BARs that are not 2MB multiples in size and alignment. The SPDK
API for registering a BAR already returns the physical/io address in the
map call, and it can be used directly without a call to
spdk_mem_register().
If the user does elect to later register the BAR using
spdk_mem_register(), we attempt to insert the 2MB aligned segments we
can into the spdk_mem_map. Users may still need to register memory for a
few reasons, such as making spdk_vtophys() work, or for setting up the
BAR as a target for RDMA. These cases still require 2MB aligned and
sized segments.
Change-Id: I395ae8803ec4bf22703f6f76db54200949e82532
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14017
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
These variants did not exist in DPDK 20.11 which is
still supported by SPDK.
So we will instead need to scan the rte_version()
string to get these values.
Fixes issue #2715.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I79657002a7a605a38a0d98b944ac53c02fa6d78c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14661
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
In-capsule data length should be the same with the SGL data length.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I7eefecb8baebb76850a48689907aff27a8946f98
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14602
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Fixed error handles which are violated with spec:
1. 'data length > MAXH2CDATA' is a fatal error.
2. 'ICDOFF != 0' should abort the IO.
Other errors which are not defined in spec:
1. invalid sgl type
2. In-capsule Data length > In-capsule Data size
Because this function runs before data part receiving, it is hard
to skip the following data segment if we want to handle some error
as non-fatal.
Currently, we have to handle all undefined errors as fatal errors.
I think after this release, we can change receving process. This will
be helpful for error handling. But this work is not small.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I8fc0d2d743505e49a93be19fd217e7ad6ca06622
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14580
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Fuzzing vfio-user require access to send request api
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I6c58b8ab4fd3394150bbb3e64b4f95bff93dae6e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13881
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
During fuzzing vfio-user client and server are started from same
process causing deadlock. SO_PEERCRED return pid of process
connected to vfio endpoint.
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I6fc2db5d58a459a30fec116a9de3c69d48acf75e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14559
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This checks the current version to make sure we have
a dpdk_fn_table that supports it.
This is easy for now, since the DPDK PCI API is
public. Moving forward, DPDK 22.11 will likely make
these APIs private, requiring us to carry header file
copies for different DPDK versions so that we can
not only build against DPDK but also use the correct
data strucures and APIs to interact with those private
DPDK interfaces. We will also need to consider
minor (i.e. stable or point) releases since they
could technically change PCI ABI as well - the current
year + month checks won't be sufficient.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic9f41d9d13778f3d078b20b08da48d8d16362b11
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14637
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This allows it to return error codes. Have the
init code check the return value and fail the init
process when pci_env_init() returns error.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7c8a4f9a6da6b3438ed09a881153b7a4ceef3a83
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14635
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Get ready to have multiple implementations of the
dpdk_fn_table. We could do some fancy self-registering
constructor functions, but let's just keep it simple
for now and extern declare each implementation in
the pci_dpdk.h header file.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8f5621412d1c8bd22c95ab74ef66c5bcc41d1380
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14636
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This is the next step in supporting multiple DPDK
PCI device/driver ABIs once those APIs are no longer
public and subject to ABI versioning rules.
This patch does the following:
1) introduce dpdk_fn_table
2) rename the existing dpdk_xx functions to xx_2207,
to denote these functions are valid for DPDK versions
up to and including 22.07
3) create a dpdk_fn_table pointing to the xx_2207
functions
4) create a global dpdk_fn_table pointer that points
directly to the 2207 fn_table
5) create new dpdk_xx functions that just redirect
to the associated dpdk_fn_table function pointer
Future patches will add the machinery to register
multiple function tables and pick the one to use at
run time based on rte_version() calls.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I1171fbdb4f72ff117416ac1fb282ff6f9fa5cadf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14634
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In prep for upcoming iovec based compression/decompression patches.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I413493f764bead9e56266e488b74f8bca979e225
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14633
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In prep for adding both src and dst iovec support for compression.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I704b8d2bd459de03deb7f8ee45d76261910a3727
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13746
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
in_capsule_data_size should not be larger than max_io_size.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I636724c888b9e5abc4cffac96bff24021e172498
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14618
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
A new parameter io_drain_timeout has been added to spdk_bdev
structure. If this value is unset, the bdev reset behavior
does not change.
The io_drain_timeout controls how long a bdev reset must wait for IO
to complete prior to issuing a reset to the underlying device.
If there is no outstanding IO at the end of that period, the reset
is skipped.
Change-Id: I585af427064ce234a4f60afc3d69bc9fc3252432
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14501
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Rename it to dpdk_pci_driver_register. This way we
follow the dpdk_pci_xxx naming convention for all
DPDK PCI structure/API dependent functions.
Also move it to the end of the file, to prepare for
moving it into the separate file.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ifca4110f737095a94f9db3d27525f5b9af0546c9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14546
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
This touches the rte_pci_device structure, so let's
make a separate accessor function just for that.
We will start putting the definitions for these
new dpdk_pci_device_xxx functions at the end of
pci.c. At the end of this series, we will then
just lop off the end of pci.c containing all of
the dpdk_pci_device functions and move them to
a DPDK-dependent pci_22_07.c file.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0323fc19b51d21d1bac899df21d6ebf4354ab339
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14542
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
struct rte_pci_driver will become private, and its
size may change between DPDK releases. But we want
to keep the spdk_pci_driver structure generic. So
allocate 256 bytes of space for the rte_pci_driver
structure, which is far more than the 104 bytes it
currently occupies. We will keep a struct
rte_pci_driver pointer to this memory in spdk_pci_driver
which can be set up in the generic code. This will
make it easier in future patches to make sure that
anything actually touching the rte_pci_driver
structure will be in the separate DPDK dependent
files.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I29aa7e71137da25a5480b34c71f2e0d5c9c02eae
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14541
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
The previous version missed the case of return value of _nbd_poll
equals to 0,and thus,when using nbd with no io,spdk_top shows high
cpu utilization.Return idle when _nbd_poll return 0.
Fixes#2697
Signed-off-by: Xinrui Mao <xinrui.mao@intel.com>
Change-Id: Ifa2ca3010e10250b5320a8282dfed3d97bea5105
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14615
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
hpda value should be in range of 0 to 31.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: Ie1329c831af06ccc8943a562c3f6396b635be518
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14575
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
This function is small and called only once.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: Ie4b11668e42a8920b3a9a11aa8cb83512f32942c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14576
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
spdk_bdev_close should be called on the caller thread. Saving the thread
now for both unmap and get stats, and executing the close in the
appropriate context.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I82192817d6012b0d41bbe2078fbd3f7dc01a7282
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14597
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
If both allocation paths would fail, then the same mngt path would
execute rollback twice, leading to use after free error.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I55c9ea5131faabc930fd8ff92ddd9f8d0fd9a0b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14596
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
An example:
There are 3 c2h data PDUs for one read request. Data digest is
enabled, accel_poller is enabled. The first PDU will be offload
to accel_poller. Then the others will use CPU to calc the crc32c.
If the last PDU is calc done and the first PDU is not calc down,
SPDK will direct success the read request, and free some objects.
When accel_poller calc down, it will find the request is freed,
and abort the SPDK.
Disable multi c2hs async process to prevent this situation.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I03c9e5b30622bbe84523c0836aa93cfed672896
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14079
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When emitting the JSON-RPC text for saving the
current configuration, add the listeners last.
This is usually the preferred order when
configuring a new subsystem - it is better to have
all of the namespaces and hosts added to the subsystem
before adding the listener to allow hosts to connect
to it. We support namespace hotplug but there's
no need to unnecessarily generate hotplug events
if we can avoid it.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I79e8a0a496eeb128efbb7e314ac835b6110d3cc8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14586
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This function is called only once and can be eliminated.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I0b3e80c025b60a816e2113f859907f95e96dd183
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14578
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
'nvmf_tcp_pdu_payload_insert_dif' can be done after receiving
whole payload data as an optimization.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I3054079427c25d102477ef8ec1b288631741d7a3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14577
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
This is consistent with the use of terms in other parts of SPDK and fits
with the code living under module/
Change-Id: If182f7cf2d160d57443a1b5f24e0065f191b59b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
psh len is not the same with header len.
Add an assert in nvme_tcp.c to prevent this happen again.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: Ibc250752bedf3da8994f79c51fb01577a222d364
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14521
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This "if" is of no use here.
The state machine has the "NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_CH"
state means the pdu does not receive enough length of header.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: Id50943f77b570fd337e2bb4e3b45281018d159e4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14504
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
RDMA transport registers MRs for in-capsule
data buffers, commands and completions. Since
these structures are allocated using huge pages,
MR for these buffers are already registered, we
only need to translate addresses.
Signed-off-by: Aleksey Marchuk <alexeymar@nvidia.com>
Change-Id: I90c53d8276d72077f7983e9faf9160e9ede52a7d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14430
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
These functions used to allocate resources
using calloc/spdk_zmalloc depending on the
g_nvme_hooks pointer. Later these functions
were refactored to always use spdk_zmalloc,
so they became simple wrappers of spdk_zmalloc
and spdk_free. There is no sense to use them,
call spdk memory API directly.
Signed-off-by: Aleksey Marchuk <alexeymar@nvidia.com>
Change-Id: I3b514b20e2128beb5d2397881d3de00111a8a3bc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14429
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Since now cmds and rsps buffers are allocated
from huge pages, there are already registered
MR for this memory. In that way we can avoid
registering 2 additional MRs per qpair, just
perform memory translation to get lkey.
Signed-off-by: Aleksey Marchuk <alexeymar@nvidia.com>
Change-Id: I2cb39a15e5d224698c293ac18af00a909840eaa8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14428
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Don't rely on compiler for metadata packing to 4KiB size and add
reserved fields manually. For compatibility reasons against metadata
relying on automatic padding the reserved fields are also added in-between
existing fields as needed.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5e342d5bf5948c213d455590d09597ae120b3c62
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14307
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Moving the superblock of the base device to sector 0, in order to
prevent other bdevs (e.g. GPT or blobstore) from potentially hijacking
the base device during startup (if their metadata by 'luck' manages to
find itself at sector 0 of band 0, which depending on the order of
operations could be very likely).
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I8a6eb3c89a229f443ef23d975a8ff0880ba65b08
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14143
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Retrying on write errors is generally not needed, by default FTL will
fail now in such cases. If retry is preferable, an additional build flag
must be supplied.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I8ed1fe140564f08905bdf7fc6d6aa86a7585693a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14114
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This should prevent accidental reordering/removal of regions from
causing problems after loading against such changed metadata.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I75c62810157db4bb0de4dfc84f5656fd187befde
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13614
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adds user write throttling - since writing to cache must be balanced
against the ability to compact the data to the base device, this
throttling mechanism allows for a smoother, more stable performance
levels - tying the user write speed to the compaction drain speed.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ia85efeb387f17c6c080b23ae4e658a6d7e47a2fb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13392
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Add gathering of some performance counters and RPC for printing them.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I2e77d37fb66459240ff2e241f2b1f77c60f4eef4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13390
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Since P2L, Band, Chunks start at version 1, adding some code blocking the loading
of version 0 for them.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4f5d3a8bb3ed1e39bea18803ffb8ba319a815ae8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13387
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Layout of metadata will be part of the superblock at the end of the upgrade.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: If888866806e948ee07f0777612da73ab8b7548b1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13385
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Added the ability for minor metadata upgrade - updating the internal
fields of metadata structures, without changing the overall layout.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Iec98c62b45b099d6d476d486ba7e4ff6b648bb95
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13384
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds extra functions which will be used during upgrade (changing
versions) of superblock metadata.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I08642deaf509f613cc8b22043dcdded6c329daa9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13383
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Preparing for potential 22.11 changes, refactor this code using DPDK api:
- a bus device list can be walked through via RTE_DEV_FOREACH,
- a reference to the bus object is directly available under the device,
Signed-off-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id3a21a6e62dfa1619a92465fac5a82afb9b43cb0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14532
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Also remove all pci-related DPDK includes from
env_internal.h, and add rte_bus_pci.h to pci.c
only.
Now pci.c has all references to DPDK pci-related
header files and data structures.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5f1727d465eaa73cf71d2f3589cecd3ebb83eb85
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14531
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This moves the only references to the rte_pci_device
data structure from memory.c to pci.c. This helps
prepare SPDK for possible changes to DPDK around
visibility of these DPDK data structures, making it
easier for SPDK to manage if only one file is
affected.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I26b1907fabd7a6c23701523811abd1ce12606683
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14530
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Following discussion in a recent SPDK community meeting,
it was determined that we no longer need to carry ISA-L as
a user configuration option. It will be enabled by default.
If running on an architecture that ISA-L isn't fully supported
on, the configure script will disable associated features and
display a warning and will also not build ISA-L. Same case if
there are issues with dependencies.
Note that --without-isal is no longer supported as a configure
option.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ibd1e5e9454d1b090462c3e757b2f51c52e6cb774
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14393
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Previously we would always allocate the shm file based on
max (128) cores which is unnecessary. So use
spdk_env APIs to only allocate shm file size based
on the cores we might possible use.
With default settings, an shm file was 135MB before this
change, now an app using cores 0-7 will just use
about 9MB.
A lot of the trace-related code depended on there
*always* being a history for every core, even unused
ones, so a few additional changes were needed,
mainly the trace_parser library.
Tested by starting an app using a 0x4 core mask and
enabling a trace mask, generating some events, then
checking both the size of the shm file and that
spdk_trace works properly with the resulting file.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie868b3e3658d6f82b2fea37cb87453e8a9e0abc4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14044
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
When doing live migration, there are some spdk_nvmf_ctrlr internal
data structures which need to be saved/restored, these data
structures are designed only for vfio-user transport, for
the purpose to extend them to support other vendor
specific transports, here we move them as public APIs,
users can use SAVE|RESTORE to restore a new nvmf controller
based on original one.
And remove the register from vfio-user transport, these registers
are stored in the common nvmf library.
Change-Id: I9f5847ef427f7064f8e16adcc963dc6b4a35f235
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11059
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Move memtable register out of start_device, into
post_handler for vhost-msg SET_MEMTABLE;
And unregister memtable in destroy_connection
instead of destroy_device
If memtable info not changed in the msg, then we
don't need to register it multi times.
Signed-off-by: Liu Xiaodong <xiaodong.liu@intel.com>
Change-Id: I0f8c76c1ee43b6f981d703beeba92da5dac4dbd6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14263
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Fix mkfs fail when using lvol as backend of nbd.Predefined
NBD_FLAG_SEND_FLUSH and NBD_FLAG_SEND_TRIM are defined by default,
so the operations of trim and flush are supported,but in fact lvol
doesn't support trim and flush operations.Therefore add judgement for
NBD_FLAG_SEND_FLUSH and NBD_FLAG_SEND_TRIM to check.
Signed-off-by: Xinrui Mao <xinrui.mao@intel.com>
Change-Id: I3d21034d12a038c8fc694d3383028103239ea6bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14099
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
state change function do not need to use swtich to do some work.
Do memset in state machine.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: Ie66454d8f31860f403171f20858a6b4a24e3c76f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14502
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Some methods are allowed to be run in both
STARTUP and RUNTIME states and current implementation
calls such methods twice. That can be a problem
in some cases, so use the new spdk_rpc_get_method_state_mask
function to skip such methods in RUNTIME state.
Signed-off-by: Aleksey Marchuk <alexeymar@nvidia.com>
Change-Id: I0a109805db428f60072a8c82161805dcde763da7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14407
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The new API will be used in the next patch
to prevent calling metods for the seconds time
when subsystem is initialized with config file
Signed-off-by: Aleksey Marchuk <alexeymar@nvidia.com>
Change-Id: I60ac8196e46ccb3b22b3af0607e1ba35a11a66a6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14406
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These functions start from a given offset and seek for next
data or for next hole. For bdevs that do not support seeking,
it is assumed that only data and no holes are present
Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: I6bc831970223333b25683f60ce3fcbbfebb5bb81
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14361
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
These functions start from a given offset and seek for first
io_unit belonging to an allocated cluster or first io_unit
belonging to an unallocated cluster
Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: I0c632e2b3dfd2e96aa22e21796e25a36f2f55f9f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14360
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
This function returns the number of io_units per cluster
Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: I8f33d24a63876a0a918830b9eeaa69a91ff21193
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14431
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Adding `psk` field to `spdk_nvme_ctrlr_opts`
Adding `psk` parameter to `bdev_nvme_attach_controller` RPC
Change-Id: Ie6f0d8b04ce472e6153934e985c026acded6cdfc
Signed-off-by: Boris Glimcher <Boris.Glimcher@emc.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14046
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
before this change, we cannot pass a `const struct option*` to
spdk_app_parse_args() even the callee does not mutate the value pointed
by the pointer. in other words, we are not able to write something like:
static const option g_options[] = {...};
// ...
spdk_app_parse_args(argc, argv, &opts, "",
g_options, app_parse_arg, app_usage);
after this change, the requirement of the type of the `option` argument
is relaxed, so we can pass a `const struct option*` to this function
now.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Change-Id: I8794fcf92090f538743850a28ef4a2a8c357f121
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14082
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
While waiting for a new PDU, target will not do too many useless
memcpy.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: Ie0825c2b1e44444b210040c4a1761010e0e4cfe5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14444
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Since only L2P pages as a whole are marked as invalid during trim, the
specific L2P entries won't be updated until someone touches that page.
The unmap process will slowly invalidate pages during runtime, by paging
them in. This will allow compaction and relocation to benefit from the
trim as the user data gets invalidated.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I239b9adf0aaaeac58f440145f4ab78b0d78d98b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13381
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Trim is now also available as a management operation via RPC.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I05b778a611e9809a14bfed50b01986bb4649a35c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13379
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds ability to send trim commands to FTL - only 4MiB aligned requests (both
for offset and length of request) will be processed. During a trim
operation an L2P page (containing 1024 4B entries, 1 per user LBA; which
is where the 4MiB alignment comes from) will be marked as unmapped.
After this point any L2P access to that page will actually set the
entries themselves as FTL_ADDR_INVALID. This is done to make the trim as
fast as possible, since for large requests it's probable that most of
the L2P pages aren't actually in DRAM.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4a04ee9498a2a6939af31b06f2e45d2b7cccbf19
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13378
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
At the end of the recovery step, all chunks will be transferred to closed state.
Missing write pointer data filled with LBA_INVALID
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Id496e465e46fa24b04b30f2558bdacfdd668e8a4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13375
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
Recovers the free/open/close chunk state, initializing them to any
specific lists.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Idf689f4fbcd6fc6bd986104dc89f5079c758845a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13373
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
For all the vhost-user messages processed in SPDK except
VHOST_USER_GET_VRING_BASE, DPDK rte_vhost "vhost-events"
thread already holds all VQ's access lock, before return
response to "vhost-events" thread, SPDK should not call
`rte_vhost_vring_call`, here we set a flag to TRUE for
these vhost-user messages, and avoid to kick VM. The
deferred IRQs will be posted in next round poll or
after restarting the device.
Fix issue #2518.
Change-Id: I82f14b97d0b0ce602a93fd66d5fdeef64f07d179
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14402
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Currently we will call `rte_vhost_vring_call` in the DPDK
"vhost-events" thread context when starting the device, and
DPDK vhost library already holds all VQ's access lock when
starting device, with new DPDK/dpdk@c573699 commit, it will
cause deadlock to call `rte_vhost_vring_call` in "vhost-events"
context, so here we increase 1 to `used_req_cnt` to make sure
one more `rte_vhost_vring_call` will be executed later in SPDK
thread context.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: Iab53941942335744bf25ab6e9b8747bd08b0c698
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14328
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
`rte_vhost_vring_call` may return error, then we can
try to call it in next poll.
Change-Id: I8f6a591837225079e004c6f57f2d7b01063f87a1
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14342
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
We do not support dynamic memory allocation with the virtio-user
library - it results in SET_MEM_TABLE vhost messages for every
change which is not supported by the vhost target.
Add '-s 256' to vhost fuzz script, to ensure it does not
violate the new restriction.
This is a follow-on patch for issue #2596.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If851f53d7d670ac8443f0d9c8f4e3cbe82e0df7c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14249
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Before this change, a negative `--shm-id` value is rejected by
`spdk_app_parse_args()` and this function simply errors out after
detecting it. However, `build_eal_cmdline()` has a dedicated branch
checking for a negative `opts->shm_id` and passes `--no-shconf` down
to DPDK as a parameter, so we cannot disable the shared config support
in DPDK.
After this change, a negative value `--shm-id` is accepted, but if it
cannot be parsed as an integer, `spdk_app_parse_args()` errors
out as before. In result we can disable shared config support in DPDK
by passing `--shm-id=-1` to SPDK application.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Change-Id: Ibe089f13638eefa9ac28c5c99e303bcc3102f307
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14097
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The generic bdev layer has a public API spdk_bdev_get_qd() but its
value is the most recently measured value and it requires qd sampling
to be enabled. We will have bdev modules to want to wait until
all bdev_ios are aborted by a reset. Unfortunately, spdk_bdev_get_qd()
is not suitable for the custom bdev module. Furthermore,
spdk_bdev_channel::io_outstanding is not accessible from bdev modules.
Hence, add a new public API spdk_bdev_get_current_qd().
This function should be used only from the bdev module and it should
be ensured that the bdev is not unregistered during execution.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ica30a8d8fe3264e28f0772a39bdf5f9ba72933e1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12791
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Some use cases want to abort every bdev_io submitted to the bdev by
traversing the bdev channels.
However, struct spdk_bdev_channel is private in lib/bdev/bdev.c.
Hence, add a helper function spdk_bdev_for_each_bdev_io() to execute
the function on the appropriate thread for every bdev_io submitted
to the bdev.
This function should be used only from the bdev module and it should
be ensured that the bdev is not unregistered during execution.
We keep this function as generic as possible because we may have
other use cases in future.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ic0209361bd1228ea8d4cb3241d0df07106be58d9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12751
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Former code, there're many repeated defines. And some add
asserts checking valid event and some don't add. To get the right
reports from debugging mode and catch the errors, so encapsulate
a common function to do these. And add assert in this function.
This will help get the right failure point.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I23d71eac6652c4104ceff80419f39634ac5ce395
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14335
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We already define a convenient variable for the admin CQ: use it.
Suggested-by: Alexis Lescouet <alexis.lescouet@nutanix.com>
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If6570f30844a52113633bdb5f3543eec700f05d7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14391
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In order to fit inside the maximum memory usage limit, recovery needs
to be split into multiple parts. During each iteration, part of L2P needs
to be read, modified as necessary and saved back to the cache.
This patch introduces the load/save steps, initialization of seq_id array
and valid map recovery. The actual L2P recovery is done in the followup patch.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I8ceadc5ef280542a173d83b932a983d5d86604a1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13371
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds recovery of open bands from P2L metadata region. Recovers the
commited P2Ls and write pointers for them.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I943c53f55e653dd075035cef7ddba448c990be87
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13370
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Recovers the open/close/free state of bands after shutdown, initializing
necessary lists.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4a6bd4ed1013ce8d04f44d1772dcd1f0e4e365bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13368
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Since base device doesn't require VSS, FTL introduces a mechanism that
will allow for recovering both the P2L and write pointer of open bands
after a dirty shutdown. After writing 1MiB of data to a band, a 4KiB
block describing the P2L will be persisted to cache device, effectively
emulating VSS for the base device.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ic6be52dc09b237297a5cda3e752d6c038e98b70e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13367
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Track the relative sequence of opening and closing bands and chunks.
Necessary for detecting the most recent user data during dirty shutdown recovery.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I682030e58284d7b090667e4e5a9f4bbc7615708a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13366
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
hexlify and unhexlify utils from vbdev_crypto.h have been moved so that
they could be included and reused outside of vbdev_crypto module.
Signed-off-by: Blachut, Bartosz <bartosz.blachut@intel.com>
Change-Id: Ia074250176907f4803b84024239ecd4e9d8a5fc1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14191
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The word engine was both used (interchangeably with module) to refer to
the things that plug into the framework and to the framework itself.
This patch eliminates all use of the word engine that meant the
framework. It leaves uses of the word that meant "module".
Change-Id: I6b9b50e2f045ac39f2a74d0152ee8d6269be4bd1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13918
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
spdk_accel_module_finish
Also move it into the internal header that defines the interface used by
modules.
Change-Id: I3aeb41e643f27a69556099cb8d166f64c9e5d67f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13917
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
For non-fabric controllers, the corresponding I/O qpairs are simply
re-enabled at controller reset.
This had a issue when I/O qpairs span multiple threads and poll group
is used.
spdk_nvme_ctrlr_reconnect_poll_async() calls
nvme_transport_ctrlr_connect_qpair() with qpair->async being false.
Then nvme_transport_ctrlr_connect_qpair() calls
spdk_nvme_poll_group_process_completions() until the qpair is connected.
spdk_nvme_poll_group_process_completions() may poll other qpairs.
This may cause I/O to complete on a wrong thread.
For PCIe controller, spdk_nvme_poll_group_process_completions() calls
spdk_nvme_qpair_process_completions() simply for each qpair.
Hence change nvme_transport_ctrlr_connect_qpair() to call
spdk_nvme_qpair_process_completions() if the controller is non-fabrics.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ieb270c2fb154124021ef6d25577b817d05e5ca9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14295
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Writing to unallocated cluster triggers copy-on-write sequence. If
this cluster is backed by zeroes device we can skip the copy part. For
a simple thin provisioned volume copy this shortcut is already
implemented because `blob->parent_id == SPDK_BLOBID_INVALID`. But this
will not work for thin provisioned volumes created from snapshot. In
this case we need to traverse the whole stack of underlying
`spdk_bs_dev` devices for specific cluster to check if it is zeroes
backed.
This patch adds `is_zeroes` operation to `spdk_bs_dev`. For zeroes
device it always returns 'true', for real bdev (`blob_bs_dev`) always
returns false, for another layer of `blob_bs_dev` does lba conversion
and forwards to backing device.
In blobstore's cluster copy flow we check if cluster is backed by
zeroes device and skip copy part if it is.
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I640773ac78f8f466b96e96a34c3a6c3c91f87dab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13446
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
There's no reason for this parameter to be non-const and it makes this
functions pain to use when you want to hardcode a specific sock
implementation.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifed4426a02ab54cbd51c8a2051b1eac010f86db9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14303
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
If initiator and target run on the same application, and initiator
uses SRQ, target may get async events for initiator, e.g.,
IBV_EVENT_QP_LAST_WQE_REACHED unexpectedly.
The reason is initiator and target may use the same device
simultaneously and only target polls async events.
Target sets attr.qp_context to rqpair when creating QP, but initiator
sets attr.qp_context to NULL when creating QP.
Hence one simple fix is to ignore async events whose qp_context is
NULL.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Id9ead1934f0b2ad1e18b174d2df2f1bf9853f7e1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14297
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
By the previous patches, a qpair is destroyed after it is actually
disconnected.
But after the qpair is destroyed, it is checked if drained by using
rqpair->current_num_sends and rqpair->current_num_recvs.
However, if the qpair is the last of a poller of a poll group,
CQ is destroyed before checking if the qpair is drained.
If CQ is destroyed, at least rqpair->current_num_recvs is not updated,
and we may get one second timeout.
This should be avoided.
Hence, destroy the qpair after it is disconnected and drained.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ibd6c83e8a3e7b6e11e9b45cee42669da6d42a621
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14278
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
If the being disconnected qpair is the last of a poller of a poll group,
CQ is destroyed and the poller is released before the qpair is actually
disconnected.
This patch destroy CQ and release the poller after the qpair is actually
disconnected.
One exception is when spdk_nvme_ctrlr_free_io_qpair() is called to a
connected qpair. In this case, the qpair is removed from a poll group
before the qpair is actually disconnected. In this case, destroy CQ and
release the poller when the qpair is removed from the poll group.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Idf266bbb6dbb40f04ae6313db724fabf80865763
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14253
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We have two cases to call nvme_rdma_poll_group_put_poller().
For consistency, make the two cases the same sequence.
This will make the next patch easier. The next patch will release
poller from poll group when qpair is actually disconnected as
possible as we can.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I4178113d5277240e287e83a57e97cf32fd0f7457
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14252
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds ability for FTL to startup after clean shutdown.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I2f1b83bb3eb1487b6665c95e76c48881e8899b16
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13364
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds additional debugging functionality - ability to check the validity
of all L2P entries and valid map to check for inconsistencies after FTL
startup. Since this is a very time consuming process, it's controlled by
an environment variable and not executed during normal operations.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4766a1576c058f69fa047f45d2d8be6d0ad0b3cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13363
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds necessary functions for setting up the state of FTL components
based on loaded in metadata.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I3a4c05230c877850e61d4f31d495d38121d27b3f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13362
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds the ability to persist only the most important metadata. The rest
is stored in shared memory.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4084c04ba09115a7a08ff66fd33552a2ec60d801
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13360
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add an extra step during FTL shutdown to save all metadata.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Idc2f77e15bbd02028548cc88355cd450175830e8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13359
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Flat L2P (all L2P in memory) needs to be specifically built against, due
to large memory consumption for big devices.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ib8906e10868455f88725b69b2b033b70a9f7256c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13358
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adds eviction of least recently used pages from the L2P cache - dirty
pages will be persisted.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ic646f7e9da777d077b5cb9b409c3f03ef05b1273
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13357
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Adds paging in from the cache device to memory.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I250009d12e9ed5ad52ee861ec5157cf983cf8cfc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13356
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
There is a set amount of pinned pages available. If exceeded they will be deferred
and processed in the future, using eviction logic.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ic642a5870db009ccf57152dd8a4178a6b2098ee1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13355
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This commit also introduces ranking pages, based on usage for determining the least used
page to be evicted.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Iaf3812177b61376bb38aa209e4ba8576d784ffb5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13354
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
L2P cache allows for partial storing of L2P in memory, paging in and out
as necessary, lowering the total memory consumption.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I727fec9d2f0ade4ca73e872d62a2ec10cfdb0a88
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13353
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
vhost_blk_start and vhost_scsi_start are now just
a single vhost_user_session_send_event() call, so
make this more generic by adding a top-level
start_session function.
Now this function will do the
vhost_user_session_send_event(), using the
user_dev_backend's start_session function pointer.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia89ba15011e231f0474405fb7225e713dcc920bf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14327
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Currently scsi sets it's svdev from the vhost
thread context, while blk does it from the
spdk thread context.
Make scsi match what blk does, to make the code
more consistent. This also will allow for an
upcoming simplification.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I609513bc8e05b49dd9455f2f61ba0cedc35236e6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14326
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Add parsing json as invalid cases:
1.json content that not enclosed in {}, it should be parsed as invalid, e.g.
"abc":"not encloesed in {}"
2.json content that 'subsystems' not associate with array, it will report error and return failure, e.g.
{"subsystems":"123"}
3.handle other invalid json formats, report and return failure, e.g. duplicate keys.
Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}.
json config with content:
1."not enclosed in {}"
2."'subsystems' not be an array"
3."duplicate key in json"
and some other invaild cases will be regarded as invalid json config, and will fail to start app.
Fixes#2599
Signed-off-by: tongkunkun <tongkunkun_yewu@cmss.chinamobile.com>
Change-Id: I02574c9acd7671e336d4c589ebbff8ed21eb3681
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13754
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Added a new RPC, vmd_rescan, that forces the VMD driver to do a rescan
of all devices behind the VMD. A device that was previously removed via
spdk_vmd_remove_device() will be found again during vmd_rescan.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ide87eb44c1d6d524234820dc07c78ba5b8bcd3ad
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13958
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Added new RPC, vmd_remove_device, that allows users to remove a PCI
device managed by the VMD library simulating a hot-remove.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifb84818ce8d147d1d586b52590527e85fe9c10de
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13957
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
It'll make it possible to remove a PCI device from within the callback.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I4cea2207a29bb145aee968715e873076a8c0993c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13956
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
This doesn't work anyway and can cause creating duplicate bus objects if
vmd_scan_single_bus() is called on a parent bus with previously
allocated child buses. Also, while here, removed a few unused functions
and flags in struct vmd_adapter.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ic757070188157d9851f648acd074ca4943a14c39
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13955
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
This is done in order to avoid having to decrement this counter in case
of a failure. Also, it makes the result valid for the few error cases
when we didn't decrement it.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia944fb8b810ce69caa8db5bc7c941e0905c9d3bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13954
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
This makes it possible to notify other PCI device providers (VMD) that a
PCI device is no longer used. The VMD will driver will unhook that
device and free any resources tied to it.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I42752afbb371a1d33972dac50fd679f68d05b597
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13887
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Idc9c7d0e5d0ebce8278e089bcfe5b7f76b86c270
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13953
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This patch implements the callback for attaching devices behind the VMD
with a given PCI address.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I07cf92c94cc7e6d3c8e31af7a8615e9a4ca641bf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13886
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
It makes it possible to call this function even if the VMD library
wasn't initialized.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I3d0f4677c4a1189f9d8acf07baee50a4e2050459
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14260
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Now that we have a attach_device() callback, the devices can be hooked
during spdk_pci_device_attach(). With DPDK, driver->cb_fn() is called
in pci_device_init(), so we need to do the same in
spdk_pci_hook_device().
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iada8b83ce7592aa62561530192072a50ec3a904b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13884
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This allows to free resources tied to a vmd_pci_device that isn't on the
dev_list or wasn't hooked to the PCI driver. Also, use that function
whenever a vmd_pci_device is freed instead of regular free().
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifca177a7eb6d8180d6f2ee2a9d9e36d58810e8ad
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14259
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
That way, we don't have to do TAILQ_REMOVE if vmd_assign_base_addrs()
fails.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id7a5df2093e4f9dfc95ee1fe415eb644c61bc971
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14258
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I79c35600fc9a758bbd9d58393b7eb98c8ac82acc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14257
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
It'll make it easier to reuse this part of the code.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id26f3f00abeeea6205df4f44689ffab1d367d777
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13885
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
The primary motivation for this patch is to allow the VMD driver to be
notified of when users wants to attach a device under a given BDF and to
make it more similar to the regular PCI path. Currently, the way the
VMD driver scans for the devices is a little bit different. The initial
scan is done during initialization and there's a separate poller for
checking hotplugs. Also, there's no device_attach() interface, so with
hotplug poller disabled, it isn't possible to attach to a device not
present in the initial scan, even if the BDF is known.
This causes a few issues. First of all, the VMD library isn't notified
when a device is stopped being used (i.e. user calls
spdk_pci_device_detach()), so when such a device is hotremoved, it never
gets unhooked. But we cannot simply add a spdk_pci_device.detach()
callback, as this would break cases when user detaches a device (without
hotremove) and then tries to reattach it again (via
spdk_pci_device_attach()), as the VMD doesn't get notified about the
device_attach() call.
So, in order to resolve this, a device_attach() callback is added, which
will notify the VMD library that the user wants to attach a device under
a specific PCI address. Then, in subsequent patches, a
spdk_pci_device_provider.detach_cb() callback is added to make sure that
devices are unhooked once they're no longer used.
Once that is done, it'll be also possible to get rid of the VMD hotplug
poller by adding something like scan_cb() to spdk_pci_device_provider and
call it from spdk_pci_enumerate().
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I084a27dcd12455f0f841440b7692375e80d07e84
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13883
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Hyper-V NVMe SSD controllers require admin queue
size to be even multiples of a page. Add quirk to
adjust the admin queue size if user overrides the
default value to something other than an even
multiple.
As part of this change, set the quirks earlier
when constructing a pcie controller, so that the
quirks value can be used in the generic
nvme_ctrlr_construct() function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I417cd3cdc7e3ba512ec412f4876b0e0b7432341c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14220
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The last line doesn't need the line break, otherwise
it will wrongly include the next line.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I06257b18d25c060b7c6bb00853fa44963fe5b439
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14241
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Better not to cache a value especially for there's an error return.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I3b243a66f4db9af34bc2ea01bafdac33004be128
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13650
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This was correct back when we only supported PCIe, but doesn't
in the newfangled world of fabrics and vfio-user.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I565edd2dab1eff862844585df8c25da508e4816d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14136
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Adds a debug function, that scans the whole P2L of band, when
it's getting closed. The P2L is compared against both L2P and
valid map to check for any discrepancies.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ia4d7be65415e6af3752d676de69b6fdcb73effb4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13352
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Utilize the valid map when picking physical blocks to compact/relocate,
speeding up the process.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I860e3cf25a5907591e4f3043def67156fec8b0df
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13351
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adds P2L validity map tracking - a bitmap marking all physical LBAs
as containing valid (current) user data or not. A clear bit denotes
the location has no valid data and may be skipped during relocation
or compaction. A set bit means it may have valid data (it's still
necessary to do the necessary comparision against L2P).
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I6a831a97b3080eb7c880d9c4feab41b523467885
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13350
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adding API for the bringup part of fast shutdown/startup. Adds
shared memory utilization for necessary functions during initialization.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Iab2da102fd0ccaa56fbdb9b3c765be5eeefff145
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13349
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adds API for fast shutdown - the ability for FTL to skip most
of the metadata persists made during clean shutdown, and relying
on their representation in shared memory instead. This allows for
faster update of SPDK (or just FTL, assuming no metadata changes),
with downtime reduction from 2-5 seconds to 500-1000 ms (for
14TiB+800GiB base and cache drives).
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5999d31698a81512db8d5893eabee7b505c80d06
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13348
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Helper functions which determine which md regions will be stored
in shm.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I94cbfca66dfb56457a350874dbd1de63a2e07661
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14159
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Stores P2L map of open bands in shared memory, allowing for
faster recovery times from application crash.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I519441af05e4d0f57768835bf01c800556873c58
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13347
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Allows for using shared memory in memory pools. Adds API for
accessing such pools after dirty shutdown (claiming them, ie.
marking an entry as actively used; calling the
ftl_mempool_initialize_ext will reclaim all unused entries back
to the pool). Also introduces API for accessing objects, since
using direct pointers is not possible (as addresses may change
inbetween application startups).
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5325b39d68aef7e231945cee9d92c925cab2fb2a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13346
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Main use case is to allow for keeping it in shared memory, to
speed up the recovery time after application crash.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I36b6b8331cd6483c5bd202e5f9103c351d705da8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13345
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Relocation will
1. Read LBA map of a given band
2. Pin the LBAs
3. Issue writes of valid LBAsto the new location
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ie753a790e56a86bfa1e451b5eda78b88eeacd3cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13344
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Multiprocess is only supported by a few libraries (e.g. NVMe driver).
Other libraries that don't support it will often fail on mempool
initialization when running as a secondary process, as the mempools are
already created by the primary process. But the error messages are
vague and don't indicate why this happened. So, this patch adds a check
to see if a mempool exists after spdk_mempool_create() fails and prints
an error message informing users that multiprocess is unsupported.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I6f915a94266e64dda380e3b269424cc579372a10
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14234
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
When we specify source address for admin and I/O qpairs,
rdma_resolve_addr() succeeded only for admin qpair and failed for
following all I/O qpairs because rdma_resolve_addr() returned
-EADDRINUSE.
To reuse source address among multiple qpairs, set the REUSEADDR option
for each CM ID before executing rdma_resolve_addr() if source address
is specified.
We may miss something. Even if rdma_set_option() fails, execute
rdma_resolve_addr().
Fixes issue #2604
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: If03f82d4499cf83c0e428a62e91c9d9e6aad28e0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14229
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Add indication of support for compare, compare & write and abort
in json bdev info dump.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ifc8dc1a1b180f08fcd9e9d58684eab1fd50356ff
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14137
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Commit a119799b ("test/nvme/aer: remove duplicated changed NS list log")
changed the nvme driver to read the CHANGED_NS_LIST log page before
calling the application's AER callback (previously it would read it
after).
Commit b801af090 ("nvme: add disable_read_changed_ns_list_log_page")
added a new ctrlr_opts member to allow the application to tell the
driver to not read this log page, and will read the log page itself
instead to clear the AEN. But we cannot add this option to the 22.01
LTS branch since it breaks the ABI. So adding this API here, which
can then be backported manually to the 22.01 branch for LTS users
that require it.
Restoring the old behavior is not correct for applications that
want to consume the CHANGED_NS_LIST log page contents itself to
know which namespaces have changed. Even if the driver reads the
log page after the application, that read could happen during a
small window between when a namespace change event has occurred and
the AEN has been sent to the host. The only safe way for the
application to consume ChANGED_NS_LIST log page contents itself
is to make sure the driver never issues such a log page request
itself.
Fixes issue #2647.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaeffe23dc7817c0c94441a36ed4d6f64a1f15a4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14134
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Fixes #issue 2636.
The existing allocation method (nvmf_rdma_get_optimal_poll_group())
is traversal and unperceived link disconnection. A more fair method
considering the number of real-time connections to allocate a poll
group is implemented.
Signed-off-by: liuqinfei <18138800392@163.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
Change-Id: Ic1e6283e386dbb0dd6655bedebe26aeedb16c333
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14002
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
If the bdev does not natively support compare we use
the fall-back which performs a read instead of a compare
operation. We then compare the results of the read with
the buffer provided by the user. In case the bdev has
metadata, there are two options:
1) md is interleaved -> the md will be part of the data
buffer allocated for the read and compared accordingly
2) md is separate -> currently we do not compare
the metadata but just ignore it.
This patch fixes 2) by comparing the md buffer after
the read is done.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: I1018b8c02540bffcba69408eb283bdc8f06bb747
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14132
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
bdev_io is allocated from a memory pool and is not zeroed on reuse.
So set bdev_io->u.bdev.ext_opts = NULL for io ops where it is not
supported (yet) so we can test against it.
Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ia579ea6b0787cf62572ea3a6bf2251867602e952
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14056
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Adds functions for reading end metadata and initializing band reloc state.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I3d12c4a7edd36f0437bf10316114c83efe449f0f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13343
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
After chunk is compacted it can be moved to the free state, able to be used
for new user IO again.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I7f9c341169b171ee246c5aa161d74903b91bdc2f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13338
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
During compaction FTL moves valid user data from the nv cache drive to
the bottom device.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ia200af39cec80014fac3a10f20d2859b10a81088
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13337
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add writer - tracks and manages band state transitions and write pointer as IO
is issued to it.
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I5f878dc15bc1c1ac84835f75fe440672fad541d5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13335
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This allows mapping an nvme_request back to the
nvme_bdev_io.
This requires bumping up the max number of arguments per
tracepoint. 5 was previously chosen as max since it
exactly fit in 64 bytes (1 cacheline) when all
arguments were stored as uint64_t, but now that we
support uint32_t arguments we can afford extra
arguments when some of them are uint32_t. I've
bumped it to 8 so we can avoid having to touch
this value multiple times if we find some cases
where we need 7 or 8 args.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie2ef5e59d10549860b47542e68c1c34efa63047f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13995
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
This will allow us to map spdk_bdev_io events
to nvme_request events coming in a future patch.
Since we pass the nvme_bdev_io to the nvme driver
(not the spdk_bdev_io), we need to add tracepoints
for the nvme_bdev_io so that spdk_trace can
do the spdk_bdev_io->nvme_bdev_io->nvme_request
mapping.
An alternative would have been to pass the spdk_bdev_io
as the cb_arg to the nvme driver, but that change
seemed to invasive, and I think we will find other
uses for the nvme_bdev_io events anyways.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id7519e689b01875093359f41a1ca2af912061a8b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13994
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Calculates general priorities and trigger points for writers
(gc and compaction) dependent on number of free bands. GC will
be started at SPDK_FTL_LIMIT_START level, while at SPDK_FTL_LIMIT_CRIT
compaction needs to be stopped and only GC is allowed to work.
This is done to make sure FTL doesn't run out of free bands and deadlock
itself.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I1aab98503c2e79e97f8e4e9fb1257530fa9770e2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13326
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We'll likely need this eventually to address silicon version
specific workarounds.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ie6957674113cf0c7b7d695b468c694668ebbf2bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13571
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is targeted to fix the following error seen under clang:
ftl_nv_cache.c:54:1: error: unused function 'nvc_data_blocks'
[-Werror,-Wunused-function]
nvc_data_blocks(struct ftl_nv_cache *nv_cache)
^
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I11d52e76df5872819770d9468b6fa4ae54d8927c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14055
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
In multi-process, we need to make sure we don't
complete a register_operation in the wrong process. So
save the pid in the nvme_register_completion structure
when it is inserted into the STAILQ, then only complete
operations where the pid matches.
Fixes issue #2630.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I58c995237db486fecdd89d95e9e7a64379d0b0e5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13940
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
To prevent the processing of outstanding commands from starving the rest of the system
Fixes: #2586
Signed-off-by: Chen, You <you.chen@intel.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: I392db2359408cdef32cc1f46b76ecd94f0c3332c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13685
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This ensures that when fields are added, that the
size of the structure will change, ensuring different
versions of the structure can be detected using
sizeof.
Adding -Wno-address-of-packed-member to Makefiles
here, although we should consider disabling this
warning globally in SPDK just like DPDK.
Suppress abidiff errors around spdk_app_opts -
structure size and offsets of all existing members
were unchanged, so there is no ABI breakage here.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2249eddb604d7b44180cadb92ba30edcd946b9bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14091
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Similar to the disable_read_ana_log_page ctrlr_opt,
this enables the application to tell the NVMe
driver to *not* read the CHANGED_NS_LIST log
page in response to a NS_ATTR_CHANGED AEN, and
will do the read itself.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie447734187d4a4cb95ceef6e0131b640b8ba5984
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14088
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Various opts structures in SPDK have a size member, to enable
ABI compatibility should fields be added in the future.
But this requires the strucures to be packed, otherwise for
example a structure may be padded at the end, and a new
field added may just consume some of that padding.
So add STATIC_ASSERTS for the current sizes in this
patch. Upcoming patches will make the structures packed
and add in reserved fields to fill in holes.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9107d01d7b533f8542385a3538894bcd9f8c465d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14086
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Mellanox Build Bot
Wait for all bdevs to finish examination before
proceeding with the spdk_bdev_finish shutdown
logic. This ensures the bdev layer and its
modules are not trying to examine bdevs after
the bdev layer has reported it has shut down.
Theoretically, bdev modules could all defer their
fini callbacks until any outstanding examinations
are complete, but it is WAY simpler to just use
the existing spdK_bdev_wait_for_examine API
instead.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If90cc2a786281d348b82de8beb17ac37ba269c64
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13850
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
If the queue was on another poll group, we need to send a message back
to the admin CQ's thread to post the completion from the correct
context.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I997987d5d6b822a1a5124f54fc29ce5d7f03190d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14057
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Swapnil Ingle <swapnil.ingle@nutanix.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
spdk_nvme_qpair_process_completions() had called
always _nvme_qpair_complete_abort_queued_reqs() at its end.
However, the call was accidentally removed by a commit
59c8bb527b
to fix an issue.
By this removal, aborting request was not completed for some error
cases.
Fix the degradation by restoring the call.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I0099eb7a008f823e1282576504423cdc248911d7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14045
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
glibc 2.36 added arc4random(), which breaks
the SPDK iSCSI build since it always implements its
own arc4random() implementation on non-FreeBSD OS
(meaning always on Linux).
So instead add a CONFIG_HAVE_ARC4RANDOM and remove
the explicit FreeBSD dependency - this will work on
FreeBSD as well as Linux with >= glibc 2.36.
Also fix check_format.sh, so that it does not
enforce spdk/stdinc.h checks on code snippets in
the configure file.
Fixes issue #2637.
Reported-by: Karl Bonde Torp <k.torp@samsung.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iab9da8ae30d62a56869530846372ffddf7138eed
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14028
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Avoid putting a new req on the outstanding_reqs
TAILQ until we know it can be initialized
successfully. This avoids adding to the TAILQ
only to remove it just after.
This allow simplifies the outstanding_reqs TAILQ
handling, since reqs are now only inserted and
removed in one place each.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5ccc41c14abd541ffcf2a602246e0671386840c7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13991
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We were using "TR" for "tracker" previously, but
we are tracing the nvme_requests, not nvme_trackers,
so use the right names for the trace object to avoid
confusion.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia3886d74b162138c2cdbe0017224d9494f74966c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13990
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
