Commit Graph

18947 Commits

Author SHA1 Message Date
Kozlowski Mateusz
0991dfd449 ftl: Remove unnecessary bdev_nvme_detach_controller from test
The target application is shutdown shortly afterwards anyway. This
prevents an intermittent failure in the lvol layer in the meantime.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I8913dff33bea985400c85fc477c6d2a7876593d3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14574
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
920c1cca18 ftl: Change metadata to use structure packing
Don't rely on compiler for metadata packing to 4KiB size and add
reserved fields manually. For compatibility reasons against metadata
relying on automatic padding the reserved fields are also added in-between
existing fields as needed.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I5e342d5bf5948c213d455590d09597ae120b3c62
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14307
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
c332181331 FTL: Move base device sb to LBA 0
Moving the superblock of the base device to sector 0, in order to
prevent other bdevs (e.g. GPT or blobstore) from potentially hijacking
the base device during startup (if their metadata by 'luck' manages to
find itself at sector 0 of band 0, which depending on the order of
operations could be very likely).

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I8a6eb3c89a229f443ef23d975a8ff0880ba65b08
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14143
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
759e176927 lib/ftl: Don't retry on write failure
Retrying on write errors is generally not needed, by default FTL will
fail now in such cases. If retry is preferable, an additional build flag
must be supplied.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I8ed1fe140564f08905bdf7fc6d6aa86a7585693a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14114
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Artur Paszkiewicz
d1dd6ca814 ftl: check structure sizes for future ABI compatibility
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ic32f6fe085d94b00d025b6cab7e5073341169a73
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13677
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
4759b0b6a6 ftl: Add explicit values to the ftl_layout_region_type
This should prevent accidental reordering/removal of regions from
causing problems after loading against such changed metadata.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I75c62810157db4bb0de4dfc84f5656fd187befde
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13614
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
7f5a982f3c FTL: Update documentation
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I78141ecf086d8ace07f9d8194c8eb9f64201a939
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13393
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Artur Paszkiewicz
63b2fecb3f ftl: nv cache write throttling
Adds user write throttling - since writing to cache must be balanced
against the ability to compact the data to the base device, this
throttling mechanism allows for a smoother, more stable performance
levels - tying the user write speed to the compaction drain speed.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ia85efeb387f17c6c080b23ae4e658a6d7e47a2fb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13392
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Artur Paszkiewicz
8a76d5500d ftl: I/O tracing
Adds tracepoints in FTL.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I661703e42b8b531822a2ba74a09cdc716daa1c46
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13391
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Artur Paszkiewicz
1790ee8a8d ftl: I/O statistics
Add gathering of some performance counters and RPC for printing them.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I2e77d37fb66459240ff2e241f2b1f77c60f4eef4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13390
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
d748bc41e2 ftl: Add layout upgrade to management path
Execute the upgrade management path during startup. Will attempt to
update metadata and verify layout validity.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I2cff15cbe87836ca8b7700d0e3f4eee0f331ac56
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14450
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
8c41c40331 ftl: Add md upgrade templates for P2L/Band/Chunk from version 0 to version 1
Since P2L, Band, Chunks start at version 1, adding some code blocking the loading
of version 0 for them.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4f5d3a8bb3ed1e39bea18803ffb8ba319a815ae8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13387
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
c8ab874d7c ftl: Add upgrade of superblock from version 2 to version 3
Layout of metadata will be part of the superblock at the end of the upgrade.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: If888866806e948ee07f0777612da73ab8b7548b1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13385
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
7ff285193f ftl: Add metadata upgrade framework
Added the ability for minor metadata upgrade - updating the internal
fields of metadata structures, without changing the overall layout.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Iec98c62b45b099d6d476d486ba7e4ff6b648bb95
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13384
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 19:24:26 +00:00
Artur Paszkiewicz
44b6d585ca FTL: Add helper functions for superblock upgrade
Adds extra functions which will be used during upgrade (changing
versions) of superblock metadata.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I08642deaf509f613cc8b22043dcdded6c329daa9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13383
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 19:24:26 +00:00
Kozlowski Mateusz
1bc356bb21 ftl: Fix abort in compaction retry path
Don't try to abort when return code is actually 0.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Id93a43173ae54324dc61ba419d929fdec4d90264
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14449
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 19:24:26 +00:00
Ben Walker
75aded8439 spdk_dd: Use FIXED_BUFFERS
This improves performance for io_uring

Change-Id: Ifff7f87e76f7ce3074c25d409dfd2b171c800e19
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12514
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-20 16:57:47 +00:00
Ben Walker
c284fa9ec4 spdk_dd: Use FIXED_FILES
This reduces overhead from fget/fput on each operation.

Change-Id: I43ddfd6e4ec9ed210206a5378eff9e7ab2f64f82
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12513
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 16:57:47 +00:00
Ben Walker
84cec154e8 spdk_dd: Allow io_uring to use O_NONBLOCK
This works fine on recent Linux kernels.

Change-Id: If3bf7feb88666d5c3e4b2f3e5071d6be6937fdf8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12515
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 16:57:47 +00:00
Ben Walker
cec27746ae spdk_dd: Report speed correctly
There were several places where the integers were likely to overflow,
plus we can avoid checking the timestamp every time because the poller
already fires on a fixed interval.

Change-Id: I7ec8f40f13b6c7a9c0f3f2de6d0aa23e9479c100
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12574
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-20 16:57:47 +00:00
Ben Walker
128083ec42 spdk_dd: If both input and output are block devices, use
IORING_SETUP_IOPOLL

Change-Id: Ie1ab8ab896ce0d2e41290645bf3ee275e1a5c578
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12022
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-20 16:57:47 +00:00
Michal Berger
a342847df7 pkgdep: Add clang packages to rhel-based distros
Currently these packages are installed through (sometimes weak)
dependencies. Install them explicitly instead.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I5d8f7c92be4a74028ce028eef33ec4ac6d2ad792
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14447
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
2022-09-20 16:00:28 +00:00
wanghailiangx
c43a50ea00 test/nvme: Counting io_submitted after starting I/O success
Change-Id: I1e79c40d25dd67c84f583b8aab362e0eb65f522e
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14536
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-20 10:19:39 +00:00
Jim Harris
b313652b30 env_dpdk/pci: Refactor PCI bus scan
Preparing for potential 22.11 changes, refactor this code using DPDK api:
- a bus device list can be walked through via RTE_DEV_FOREACH,
- a reference to the bus object is directly available under the device,

Signed-off-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id3a21a6e62dfa1619a92465fac5a82afb9b43cb0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14532
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-09-20 10:19:22 +00:00
Jim Harris
36644ef32f env_dpdk: move spdk_pci_driver definition to pci.c
Also remove all pci-related DPDK includes from
env_internal.h, and add rte_bus_pci.h to pci.c
only.

Now pci.c has all references to DPDK pci-related
header files and data structures.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5f1727d465eaa73cf71d2f3589cecd3ebb83eb85
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14531
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 10:19:22 +00:00
Jim Harris
2bb7185f1b env_dpdk: add dpdk_pci_device_vtophys()
This moves the only references to the rte_pci_device
data structure from memory.c to pci.c.  This helps
prepare SPDK for possible changes to DPDK around
visibility of these DPDK data structures, making it
easier for SPDK to manage if only one file is
affected.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I26b1907fabd7a6c23701523811abd1ce12606683
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14530
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 10:19:22 +00:00
Jim Harris
92e63a9cc6 env_dpdk: remove unused SPDK_PCI_DRIVER_MAX_NAME_LEN
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7b6f8d165b56b079fbab0f9dd4a354bf82533d59
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14529
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 10:19:22 +00:00
paul luse
dd2c08d2d1 configure/misc: make ISA-L a hard dependency
Following discussion in a recent SPDK community meeting,
it was determined that we no longer need to carry ISA-L as
a user configuration option.  It will be enabled by default.

If running on an architecture that ISA-L isn't fully supported
on, the configure script will disable associated features and
display a warning and will also not build ISA-L. Same case if
there are issues with dependencies.

Note that --without-isal is no longer supported as a configure
option.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ibd1e5e9454d1b090462c3e757b2f51c52e6cb774
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14393
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-20 10:18:54 +00:00
KanKuo
b4f9bb444a UT/bdev/part.c:add the test of function spdk_bdev_part_free()
Signed-off-by: KanKuo <kuox.kan@intel.com>
Change-Id: I99bdb67eea08e27276fca4c8659b71ced076573c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14074
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
2022-09-20 10:17:57 +00:00
Jim Harris
18c8b52afa trace: allocate shm filesize based on number of cores used
Previously we would always allocate the shm file based on
max (128) cores which is unnecessary.  So use
spdk_env APIs to only allocate shm file size based
on the cores we might possible use.

With default settings, an shm file was 135MB before this
change, now an app using cores 0-7 will just use
about 9MB.

A lot of the trace-related code depended on there
*always* being a history for every core, even unused
ones, so a few additional changes were needed,
mainly the trace_parser library.

Tested by starting an app using a 0x4 core mask and
enabling a trace mask, generating some events, then
checking both the size of the shm file and that
spdk_trace works properly with the resulting file.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie868b3e3658d6f82b2fea37cb87453e8a9e0abc4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14044
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2022-09-20 10:17:45 +00:00
Changpeng Liu
982c25feef nvmf: add spdk_nvmf_ctrlr_[save|restore]_migr_data() APIs
When doing live migration, there are some spdk_nvmf_ctrlr internal
data structures which need to be saved/restored, these data
structures are designed only for vfio-user transport, for
the purpose to extend them to support other vendor
specific transports, here we move them as public APIs,
users can use SAVE|RESTORE to restore a new nvmf controller
based on original one.

And remove the register from vfio-user transport, these registers
are stored in the common nvmf library.

Change-Id: I9f5847ef427f7064f8e16adcc963dc6b4a35f235
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11059
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 10:17:24 +00:00
Michal Berger
04dd028292 perf/nvmf: Gather power statistics on the target side
Use collect-bmc-pm to gather power usage stats during fio workloads.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I20d1f3009c8a32c3927ad2cf2f932a8e97d3e7c5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14164
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-20 10:17:04 +00:00
Michal Berger
b43a0112dd scripts/perf: Add script for collecting system power statistics
This script is a wrapper around ipmitool which attempts to extract
power statistics from the BMC. It's mainly based around DCMI spec
to make sure it's common for most of the platforms that we use
in our CI. If given BMC does not conform to DCMI, there's a fallback
in place which attempts to find proper sensor in the SDR which may
report overall power usage of the underlying system.

By default, current ("NOW") reading is extracted in context of DCMI.
Different intervals may be defined, and when matched with supported
averaging time periods, special version of the command will be used
to extract additional avg, min, max information for given time (e.g.
current, max, min, avg readings from last 1m). For the SDR fallback,
only current reading can be extracted.

When terminated, script dumps all readings + their averaged value
into separate .txt files.

Change-Id: Ifc8c1defe862254864d69de7edf5cce635aff258
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14144
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-20 10:17:04 +00:00
Jacek Kalwas
b9717f0109 perf: change qdepth from int to uint32_t
this is due to further change and compiler warning related to
comparison of variables with different signedness

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: Ia1e5acd85fce27fc53f63597b0e44753b86a7801
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14518
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-09-20 00:46:38 +00:00
Michal Berger
c738575823 autotest: Move nvmf's fuzzing outside the main suite
This test is static in nature and not dependent on actual nvmf setup
that comes from the CI - everything is driven via a static json config
that comes with the test itself.

Instead, keep fuzzing directly in autotest.sh, under a top dedicated
test flag SPDK_TEST_FUZZER - this should allow users to run only
fuzzing part without touching the majority of the nvmf suite.

Change-Id: I9c33c46e05f7c4b977b041673fe200d9d2989265
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14287
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-09-20 00:46:11 +00:00
Denis Barakthanov
044bb17525 bdev/daos: support custom DAOS installation dir
./configure --with-daos=[DIR] now accepts DAOS installation directory.

Signed-off-by: Denis Barakhtanov <denis.barahtanov@croit.io>
Change-Id: I16f4e0e41e6f73a32dec452c20aa7b558ca50f33
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14522
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-20 00:45:40 +00:00
Sebastian Brzezinka
d326389d96 test/sma: use smaller indention for the JSONs
Both tests sma/vhost_blk.sh and sma/vfiouser_qemu.sh
use two spaces indention for JSON

Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I59aedabe40852d45a606c82a875a49a182137f91
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14293
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-19 19:43:35 +00:00
Sebastian Brzezinka
9f0d4f33c3 test/sma: vhost_blk create cntrl with crypto bdev
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I72c1340d927101d171509b1a42c4f14c0f7dc3e9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13998
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-19 19:43:35 +00:00
Sebastian Brzezinka
7cefe7f62d test/sma: vfiouser attach crypto bdev
The test uses bdev_crypto and vfiouser devices to verify that
volumes with crypto parameters are correctly attached

Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I9abed43f3df803870af11d7f28601f5f9390474c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13997
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-19 19:43:35 +00:00
Sebastian Brzezinka
a70732c9f1 test/sma: move crypto helper function to common.sh
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I55d58a5959d5b67d99c2e2dfd9ea787a1e31b023
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13996
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-19 19:43:35 +00:00
Sebastian Brzezinka
aa55c23dbc sma: crypto support for vhost_blk devices
The vhost_blk devices will now support volumes with encryption enabled.
Reorder 'imports' according to PEP 8.

Change-Id: Ia4ae0832f4cf91578153da7ca0a28a83707ccdca
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13983
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-19 19:43:35 +00:00
Sebastian Brzezinka
62e8255460 sma: crypto support for nvmf/vfiouser devices
The nvmf/vfiouser devices will now support attaching volumes with
encryption enabled. Reorder 'imports' according to PEP 8.

Change-Id: I9862c655c2b57b8bb307f6091d4967afc21c11da
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13982
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-19 19:43:35 +00:00
Konrad Sztyber
30ff5c6f14 test/sma: crypto test
The test uses bdev_crypto and NVMe/TCP devices to verify that volumes
with crypto parameters are correctly attached and any errors are handled
as intended.

It starts up two spdk_tgt processes, one is being orchestrated by SMA
and the other one acts as an NVMe-oF target.  This ensures that we can
verify that SMA terminates discovery service connections whenever a
failure occurs during crypto setup.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I74001c256d4a05d2d8d295fb6fd8375d30ab66a1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13873
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-19 19:43:35 +00:00
Sebastian Brzezinka
a9d10dade8 test/common: add with-crypto to sma autotests
Change-Id: I25cff079e021418ec66c6d04cfd5a3daf969ed77
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14154
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-19 19:43:35 +00:00
Konrad Sztyber
2bd552fbfe sma: crypto support through bdev_crypto
This patch adds CryptoEngine implementation using bdev_crypto.  Only a
single crypto drvier can be used at a time and it's configured during
startup in a config file, e.g.:

```
crypto:
  name: 'bdev_crypto'
  params:
    driver: 'crypto_aesni_mb'
```

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I13f4fd1227a02cf9f1bee61a1686904b43c0fc55
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13872
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
2022-09-19 19:43:35 +00:00
Konrad Sztyber
c16dab7e37 sma: crypto support for nvmf/tcp devices
The nvmf/tcp devices will now support attaching volumes with encryption
enabled.  This basically boils down to getting the crypto bdev name
through the CryptoEngine.get_crypto_bdev() interface (instead of just
using volume_id) and specyfing UUID/NGUID when attaching namespaces
to a subsystem.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iefddbf07675152aa2e042564eb87c457b6995b9b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13871
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
2022-09-19 19:43:35 +00:00
Konrad Sztyber
e1068a862c sma: configure crypto when attaching volumes
Crypto parameters are now checked and crypto is configured when a volume
is attached.  Since configuring crypto can lead to creating new bdevs on
top of the attached volume, each device manager will also need to be
changed to retrieve the bdev through the CryptoEngine.get_crypto_bdev()
interface.  This will be done in subsequent patches.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Idb5f804cf88aa5e34bbee6817acdb8f3a42a2320
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13870
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
2022-09-19 19:43:35 +00:00
Konrad Sztyber
cc3f842cd1 sma: initial crypto definitions
This patch defines the interface for crypto engines, which provide
support for configuring crypto on a given volume.  Only a single crypto
engine can be active at a time and it's selected in the "crypto" section
of the config file.  Similarly to device managers, external crypto
engines can be loaded from plugins.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id942ef876e070816827d7ad1937eb510a85c8f8d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13869
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
2022-09-19 19:43:35 +00:00
Konrad Sztyber
b30edf643d sma: volume crypto interface
This interface will make it possible for users to configure data at rest
encryption for volumes exposed through SMA.  The crypto parameters are
part of volume parameters specified when a volume is attached.  Once a
volume is attached it is not possible to change it.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I795cfab67523ace47d366d50e33b253737716668
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13868
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-19 19:43:35 +00:00
Michal Berger
11d13b85e6 test/nvmf: Test different abort qd sizes under SPDK and kernel targets
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I79ede7670477ed0d8eb38fa3eced4a9b70c64620
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8962
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-19 13:13:42 +00:00