Commit Graph

9571 Commits

Author SHA1 Message Date
yidong0635
b813f998ea nvme_pcie_common: Move group right before using.
Better not to cache a value especially for there's an error return.

Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I3b243a66f4db9af34bc2ea01bafdac33004be128
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13650
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-01 08:26:34 +00:00
Jim Harris
3d59045a2a nvme: remove incorrect comment about spdk_nvme_ctrlr structs
This was correct back when we only supported PCIe, but doesn't
in the newfangled world of fabrics and vfio-user.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I565edd2dab1eff862844585df8c25da508e4816d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14136
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-08-30 16:20:23 +00:00
Artur Paszkiewicz
8fad5718e1 ftl: validate band metadata in debug mode
Adds a debug function, that scans the whole P2L of band, when
it's getting closed. The P2L is compared against both L2P and
valid map to check for any discrepancies.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ia4d7be65415e6af3752d676de69b6fdcb73effb4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13352
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-30 14:48:50 +00:00
Artur Paszkiewicz
57cfab6808 ftl: use valid map to optimize compaction and reloc
Utilize the valid map when picking physical blocks to compact/relocate,
speeding up the process.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I860e3cf25a5907591e4f3043def67156fec8b0df
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13351
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-30 14:48:50 +00:00
Artur Paszkiewicz
cea8dadecf ftl: valid map
Adds P2L validity map tracking - a bitmap marking all physical LBAs
as containing valid (current) user data or not. A clear bit denotes
the location has no valid data and may be skipped during relocation
or compaction. A set bit means it may have valid data (it's still
necessary to do the necessary comparision against L2P).

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I6a831a97b3080eb7c880d9c4feab41b523467885
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13350
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-30 14:48:50 +00:00
Artur Paszkiewicz
1e904e2b75 ftl: fast startup
Adding API for the bringup part of fast shutdown/startup. Adds
shared memory utilization for necessary functions during initialization.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Iab2da102fd0ccaa56fbdb9b3c765be5eeefff145
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13349
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-30 14:48:50 +00:00
Kozlowski Mateusz
0e33da4974 ftl: fast shutdown
Adds API for fast shutdown - the ability for FTL to skip most
of the metadata persists made during clean shutdown, and relying
on their representation in shared memory instead. This allows for
faster update of SPDK (or just FTL, assuming no metadata changes),
with downtime reduction from 2-5 seconds to 500-1000 ms (for
14TiB+800GiB base and cache drives).

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5999d31698a81512db8d5893eabee7b505c80d06
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13348
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-30 14:48:50 +00:00
Kozlowski Mateusz
811a027e43 ftl: Add helper functions for creating md regions
Helper functions which determine which md regions will be stored
in shm.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I94cbfca66dfb56457a350874dbd1de63a2e07661
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14159
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-30 14:48:50 +00:00
Kozlowski Mateusz
101a039923 ftl: p2l map on shm
Stores P2L map of open bands in shared memory, allowing for
faster recovery times from application crash.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I519441af05e4d0f57768835bf01c800556873c58
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13347
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-30 14:48:50 +00:00
Artur Paszkiewicz
71a1762821 ftl: mempool support for durable format objects
Allows for using shared memory in memory pools. Adds API for
accessing such pools after dirty shutdown (claiming them, ie.
marking an entry as actively used; calling the
ftl_mempool_initialize_ext will reclaim all unused entries back
to the pool). Also introduces API for accessing objects, since
using direct pointers is not possible (as addresses may change
inbetween application startups).

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5325b39d68aef7e231945cee9d92c925cab2fb2a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13346
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-30 14:48:50 +00:00
Artur Paszkiewicz
f1b079b49f ftl: bitmap on external memory
Main use case is to allow for keeping it in shared memory, to
speed up the recovery time after application crash.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I36b6b8331cd6483c5bd202e5f9103c351d705da8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13345
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-30 14:48:50 +00:00
Kozlowski Mateusz
43a4d47a1c FTL: Add relocation logic
Relocation will
1. Read LBA map of a given band
2. Pin the LBAs
3. Issue writes of valid LBAsto the new location

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ie753a790e56a86bfa1e451b5eda78b88eeacd3cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13344
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-30 14:48:50 +00:00
Jim Harris
ffa823557a blob: add assert that cluster_sz > 0
Avoids divide-by-zero scanbuild warning on Fedora36.

Fixes issue #2667.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib2793c793725e8bb8ba25fb779ffc14334929da0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14238
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-08-29 11:41:50 +00:00
Konrad Sztyber
475b86aa8d print better errors when creating mempools from secondary process
Multiprocess is only supported by a few libraries (e.g. NVMe driver).
Other libraries that don't support it will often fail on mempool
initialization when running as a secondary process, as the mempools are
already created by the primary process.  But the error messages are
vague and don't indicate why this happened.  So, this patch adds a check
to see if a mempool exists after spdk_mempool_create() fails and prints
an error message informing users that multiprocess is unsupported.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I6f915a94266e64dda380e3b269424cc579372a10
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14234
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
2022-08-29 11:41:32 +00:00
Shuhei Matsumoto
4a6f858872 nvme_rdma: Set REUSEADDR to reuse source address among multiple CM IDs
When we specify source address for admin and I/O qpairs,
rdma_resolve_addr() succeeded only for admin qpair and failed for
following all I/O qpairs because rdma_resolve_addr() returned
-EADDRINUSE.

To reuse source address among multiple qpairs, set the REUSEADDR option
for each CM ID before executing rdma_resolve_addr() if source address
is specified.

We may miss something. Even if rdma_set_option() fails, execute
rdma_resolve_addr().

Fixes issue #2604

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: If03f82d4499cf83c0e428a62e91c9d9e6aad28e0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14229
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
2022-08-29 11:41:17 +00:00
Jonas Pfefferle
29977e8506 bdev: add additional io types in dump bdev info
Add indication of support for compare, compare & write and abort
in json bdev info dump.

Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ifc8dc1a1b180f08fcd9e9d58684eab1fd50356ff
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14137
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-29 10:51:31 +00:00
Jim Harris
4300c62167 nvme: add spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page()
Commit a119799b ("test/nvme/aer: remove duplicated changed NS list log")
changed the nvme driver to read the CHANGED_NS_LIST log page before
calling the application's AER callback (previously it would read it
after).

Commit b801af090 ("nvme: add disable_read_changed_ns_list_log_page")
added a new ctrlr_opts member to allow the application to tell the
driver to not read this log page, and will read the log page itself
instead to clear the AEN.  But we cannot add this option to the 22.01
LTS branch since it breaks the ABI.  So adding this API here, which
can then be backported manually to the 22.01 branch for LTS users
that require it.

Restoring the old behavior is not correct for applications that
want to consume the CHANGED_NS_LIST log page contents itself to
know which namespaces have changed.  Even if the driver reads the
log page after the application, that read could happen during a
small window between when a namespace change event has occurred and
the AEN has been sent to the host.  The only safe way for the
application to consume ChANGED_NS_LIST log page contents itself
is to make sure the driver never issues such a log page request
itself.

Fixes issue #2647.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaeffe23dc7817c0c94441a36ed4d6f64a1f15a4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14134
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-25 07:31:44 +00:00
liuqinfei
cd1b7ab0e7 nvmf: balance the get optimal poll group
Fixes #issue 2636.

The existing allocation method (nvmf_rdma_get_optimal_poll_group())
is traversal and unperceived link disconnection. A more fair method
considering the number of real-time connections to allocate a poll
group is implemented.

Signed-off-by: liuqinfei <18138800392@163.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
Change-Id: Ic1e6283e386dbb0dd6655bedebe26aeedb16c333
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14002
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-23 07:46:03 +00:00
Jonas Pfefferle
9e50d53b1a bdev: add compare fall-back separate md support
If the bdev does not natively support compare we use
the fall-back which performs a read instead of a compare
operation. We then compare the results of the read with
the buffer provided by the user. In case the bdev has
metadata, there are two options:
1) md is interleaved -> the md will be part of the data
buffer allocated for the read and compared accordingly
2) md is separate -> currently we do not compare
the metadata but just ignore it.
This patch fixes 2) by comparing the md buffer after
the read is done.

Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: I1018b8c02540bffcba69408eb283bdc8f06bb747
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14132
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-23 07:18:56 +00:00
Jonas Pfefferle
7ba89d1e48 bdev: set ext_opts=NULL if not used
bdev_io is allocated from a memory pool and is not zeroed on reuse.
So set bdev_io->u.bdev.ext_opts = NULL for io ops where it is not
supported (yet) so we can test against it.

Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>
Change-Id: Ia579ea6b0787cf62572ea3a6bf2251867602e952
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14056
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
2022-08-23 07:18:56 +00:00
Kozlowski Mateusz
711759a029 FTL: Add reloc helper functions
Adds functions for reading end metadata and initializing band reloc state.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I3d12c4a7edd36f0437bf10316114c83efe449f0f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13343
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-22 20:21:15 +00:00
Artur Paszkiewicz
f45c007512 ftl: superblock in shared memory
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I86e2cbf364ae3075aad2e09429754027df33eadf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13342
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-22 20:21:15 +00:00
Artur Paszkiewicz
818b9c053b ftl: support for metadata on shared memory
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ibc259f61f0ef2aeadb0e5ac7230969e29d77f184
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13340
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-22 20:21:15 +00:00
Kozlowski Mateusz
19613862ae FTL: Add free chunk logic
After chunk is compacted it can be moved to the free state, able to be used
for new user IO again.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I7f9c341169b171ee246c5aa161d74903b91bdc2f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13338
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-19 17:37:14 +00:00
Kozlowski Mateusz
71f20c9a74 FTL: Add compaction logic
During compaction FTL moves valid user data from the nv cache drive to
the bottom device.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ia200af39cec80014fac3a10f20d2859b10a81088
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13337
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-19 17:37:14 +00:00
Artur Paszkiewicz
1dadcd8786 ftl: ftl_rq helpers for compaction
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I614b29e7bc7f6db20b10395bc780ff633c497b59
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13336
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-19 17:37:14 +00:00
Kozlowski Mateusz
31cf633679 FTL: Add writer logic
Add writer - tracks and manages band state transitions and write pointer as IO
is issued to it.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I5f878dc15bc1c1ac84835f75fe440672fad541d5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13335
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-19 17:37:14 +00:00
Artur Paszkiewicz
0291b2845a FTL: Add read path
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ib5bac109b59d5a21a7dad1f8e79b5da7633ffa9d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13334
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-19 17:37:14 +00:00
Kozlowski Mateusz
5af491a2ee FTL: Add band state change functions
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I6a985f0b54a05fbebb8d65343cffaed7e47ed60d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13332
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-19 17:37:14 +00:00
Artur Paszkiewicz
7c9d3ea595 FTL: Add helper functions for IO to band regions
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I17443ba83afd0ccee0cb84e02329b150562cfd63
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13331
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-19 17:37:14 +00:00
Jim Harris
e36f0d363e nvme/pcie, nvme/tcp: add cb_arg context tracepoint argument
This allows mapping an nvme_request back to the
nvme_bdev_io.

This requires bumping up the max number of arguments per
tracepoint.  5 was previously chosen as max since it
exactly fit in 64 bytes (1 cacheline) when all
arguments were stored as uint64_t, but now that we
support uint32_t arguments we can afford extra
arguments when some of them are uint32_t.  I've
bumped it to 8 so we can avoid having to touch
this value multiple times if we find some cases
where we need 7 or 8 args.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie2ef5e59d10549860b47542e68c1c34efa63047f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13995
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-08-19 11:06:31 +00:00
Jim Harris
54f1603954 bdev/nvme: add tracepoint support
This will allow us to map spdk_bdev_io events
to nvme_request events coming in a future patch.
Since we pass the nvme_bdev_io to the nvme driver
(not the spdk_bdev_io), we need to add tracepoints
for the nvme_bdev_io so that spdk_trace can
do the spdk_bdev_io->nvme_bdev_io->nvme_request
mapping.

An alternative would have been to pass the spdk_bdev_io
as the cb_arg to the nvme driver, but that change
seemed to invasive, and I think we will find other
uses for the nvme_bdev_io events anyways.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id7519e689b01875093359f41a1ca2af912061a8b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13994
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-08-19 11:06:31 +00:00
Kozlowski Mateusz
81dfe157f3 FTL: Add calculation of device size
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I1f57ea699d7613f89270f9a47f044d1b85c72b60
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13330
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 19:09:50 +00:00
Kozlowski Mateusz
9dbdb02975 FTL: Initialize band metadata on startup
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ie27b3c5058ae6029262ad3861d5c64dd1ac5794f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13329
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 19:09:50 +00:00
Kozlowski Mateusz
88d1c3a69a FTL: Add debug function for dumping band information
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I6edef1e8e822f8428dff5f5f5da2df923191f6fc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13328
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 19:09:50 +00:00
Kozlowski Mateusz
8c519d31bd FTL: Add internal band state changes
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Icaecc4e77996919a23f70c1ffad15b783741fd5e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13327
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 19:09:50 +00:00
Artur Paszkiewicz
0f99700db9 ftl: user write limits
Calculates general priorities and trigger points for writers
(gc and compaction) dependent on number of free bands. GC will
be started at SPDK_FTL_LIMIT_START level, while at SPDK_FTL_LIMIT_CRIT
compaction needs to be stopped and only GC is allowed to work.
This is done to make sure FTL doesn't run out of free bands and deadlock
itself.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I1aab98503c2e79e97f8e4e9fb1257530fa9770e2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13326
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-08-18 19:09:50 +00:00
Artur Paszkiewicz
c7213b9c6d FTL: Add band P2L map usage
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I7f526c80667ab548a2903689066ac76a8d8d3c53
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13325
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 19:09:50 +00:00
Artur Paszkiewicz
6448f33672 FTL: Add band structure and helper functions
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I986746a008e716705304906ab4f2bdabce0a84c3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13324
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 19:09:50 +00:00
Kozlowski Mateusz
1bbefed63b FTL: Remove leftover ZNS code
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ica358805a69582d78e0d6c4f17b5a97ff38e44ca
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14112
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 19:09:50 +00:00
paul luse
c746637df8 lib/idxd: add some flag overrides when doing PMEM writes
Per upcoming specification changes.

Fixes: 2486

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ic2534148a87b3dec7512f7b01384f484fee4c30f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13572
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: <wayne.gao@intel.com>
2022-08-18 18:47:02 +00:00
paul luse
61631dadb3 lib/idxd: Save device version during kernel and user initialization
We'll likely need this eventually to address silicon version
specific workarounds.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ie6957674113cf0c7b7d695b468c694668ebbf2bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13571
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-18 18:47:02 +00:00
Michal Berger
5f6ce57fb2 lib/ftl: Fix "unused function" error under clang
This is targeted to fix the following error seen under clang:

  ftl_nv_cache.c:54:1: error: unused function 'nvc_data_blocks'
     [-Werror,-Wunused-function]
  nvc_data_blocks(struct ftl_nv_cache *nv_cache)
  ^

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I11d52e76df5872819770d9468b6fa4ae54d8927c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14055
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
2022-08-18 10:10:09 +00:00
Jim Harris
0f068506ca nvme: complete register_operations in the correct process
In multi-process, we need to make sure we don't
complete a register_operation in the wrong process.  So
save the pid in the nvme_register_completion structure
when it is inserted into the STAILQ, then only complete
operations where the pid matches.

Fixes issue #2630.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I58c995237db486fecdd89d95e9e7a64379d0b0e5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13940
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-08-18 10:09:55 +00:00
Chen, You
43ebecdf60 lib/idxd: break spdk_idxd_process_events loop after processing DESC_PER_BATCH ops
To prevent the processing of outstanding commands from starving the rest of the system

Fixes: #2586

Signed-off-by: Chen, You <you.chen@intel.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: I392db2359408cdef32cc1f46b76ecd94f0c3332c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13685
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 10:09:23 +00:00
Jim Harris
92335c01cf event: make opts structures packed
This ensures that when fields are added, that the
size of the structure will change, ensuring different
versions of the structure can be detected using
sizeof.

Adding -Wno-address-of-packed-member to Makefiles
here, although we should consider disabling this
warning globally in SPDK just like DPDK.

Suppress abidiff errors around spdk_app_opts -
structure size and offsets of all existing members
were unchanged, so there is no ABI breakage here.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2249eddb604d7b44180cadb92ba30edcd946b9bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14091
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-08-18 10:08:40 +00:00
Jim Harris
b801af090a nvme: add disable_read_changed_ns_list_log_page
Similar to the disable_read_ana_log_page ctrlr_opt,
this enables the application to tell the NVMe
driver to *not* read the CHANGED_NS_LIST log
page in response to a NS_ATTR_CHANGED AEN, and
will do the read itself.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie447734187d4a4cb95ceef6e0131b640b8ba5984
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14088
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-08-18 10:08:40 +00:00
Jim Harris
c50cb569de include: add STATIC_ASSERTS for opts structures with size member
Various opts structures in SPDK have a size member, to enable
ABI compatibility should fields be added in the future.

But this requires the strucures to be packed, otherwise for
example a structure may be padded at the end, and a new
field added may just consume some of that padding.

So add STATIC_ASSERTS for the current sizes in this
patch.  Upcoming patches will make the structures packed
and add in reserved fields to fill in holes.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9107d01d7b533f8542385a3538894bcd9f8c465d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14086
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Mellanox Build Bot
2022-08-18 10:08:40 +00:00
Jim Harris
af0d907604 bdev: wait_for_examine during spdk_bdev_finish.
Wait for all bdevs to finish examination before
proceeding with the spdk_bdev_finish shutdown
logic.  This ensures the bdev layer and its
modules are not trying to examine bdevs after
the bdev layer has reported it has shut down.

Theoretically, bdev modules could all defer their
fini callbacks until any outstanding examinations
are complete, but it is WAY simpler to just use
the existing spdK_bdev_wait_for_examine API
instead.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If90cc2a786281d348b82de8beb17ac37ba269c64
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13850
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-08-18 08:35:18 +00:00
Kozlowski Mateusz
e8c5ccf039 FTL: Add write path
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I41985617b5879bd3f4bf6d49d2a03eaffdd5ccb5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13322
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-08-18 08:34:47 +00:00