Commit Graph

6505 Commits

Author SHA1 Message Date
Konrad Sztyber
cc3f842cd1 sma: initial crypto definitions
This patch defines the interface for crypto engines, which provide
support for configuring crypto on a given volume.  Only a single crypto
engine can be active at a time and it's selected in the "crypto" section
of the config file.  Similarly to device managers, external crypto
engines can be loaded from plugins.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id942ef876e070816827d7ad1937eb510a85c8f8d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13869
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
2022-09-19 19:43:35 +00:00
Michal Berger
11d13b85e6 test/nvmf: Test different abort qd sizes under SPDK and kernel targets
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I79ede7670477ed0d8eb38fa3eced4a9b70c64620
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8962
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-19 13:13:42 +00:00
wanghailiangx
69b4e8b17f test/nvme: Replace -s with -o for "io size in bytes"
For most of apps, -o is for "io size in bytes",
-s("io size in bytes") was changed to -o a long time ago.
-s is now for "memory size in MB for DPDK".

Change-Id: I0977d4a7bc30a4f2e135707ff6e77b6254240fb9
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14533
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-19 13:11:55 +00:00
Krzysztof Karas
905e519adc autotest_common.sh: increase read timeout inside rpc_cmd
Currently we use 5 seconds as a timeout inside rpc_cmd's
"while read" loop, which is enough for a system with
a single NVMe. That value is insufficient for machines
with more devices and causes the timeout, erroring out
the tests (verified with the same number of devices,
as in the CI node with the most NVMes available - 4).

For that reason increase the read timeout to 15 seconds.

Fixes #2641

Change-Id: If7e9fd169a75ea16ec243c63821a4e5d70995063
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14525
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-19 13:11:30 +00:00
Jim Harris
88055b422f test/spdkcli: reduce amount of HUGEMEM memory requested
We do not need to force usage of 4096MB of memory for these
tests.  Let's just remove how that is being forced.

This allows us to also remove the setup.sh call from
the spdkcli test common.sh script.  Its only purpose
was to increase the amount of memory allocated to satisfy
the spdkcli tests, but now that the spdkcli tests aren't
forcing so much memory to be allocated, we can remove
that too.

Fixes issue #2694.

Note: these limits were in the original commit for the
test scripts - 7e8206c - but it is not clear why they
were added instead of just relying on DPDK dynamic
memory allocation..

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2210765e69117205a039aa6bc468efa5d5d537cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14520
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-19 13:11:06 +00:00
Changpeng Liu
f167986b6c test/vfio-user: skip NVMeoF target coremask for VMs
For vfio-user tests, SPDK NVMeoF target uses "0xF"
as the core mask, so we should avoid to use these
cores when starting VMs.

Fix issue #2600.

Change-Id: I1927cd70670b30d59944d5571bcfc37c9e7e35d6
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14485
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-19 12:47:37 +00:00
Xinrui Mao
c3f628f141 lib/nbd:export bdev flush and trim ability
Fix mkfs fail when using lvol as backend of nbd.Predefined
NBD_FLAG_SEND_FLUSH and NBD_FLAG_SEND_TRIM are defined by default,
so the operations of trim and flush are supported,but in fact lvol
doesn't support trim and flush operations.Therefore add judgement for
NBD_FLAG_SEND_FLUSH and NBD_FLAG_SEND_TRIM to check.

Signed-off-by: Xinrui Mao <xinrui.mao@intel.com>
Change-Id: I3d21034d12a038c8fc694d3383028103239ea6bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14099
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2022-09-16 13:32:13 +00:00
Michal Berger
6872c67e4f perf/vhost: Add NUMA-aware cgroup setup for the VMs
Each VM will be placed under dedicated cgroup pinned to proper set of
cpus and mem nodes as defined in the provided configuration.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Id5d284ee9cbde541b8344ab38c3f77aa5430b1b1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13704
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-15 20:15:57 +00:00
Michal Berger
3d5e27df7d test/cgroups: Add helper functions for more granular setup
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I6b6baae849a472599adc36a04d268a7a65979b41
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13703
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-15 20:15:57 +00:00
MengjinWu
31fc5f196f nvme/tcp: simplify state change function
state change function do not need to use swtich to do some work.

Do memset in state machine.

Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: Ie66454d8f31860f403171f20858a6b4a24e3c76f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14502
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
2022-09-15 19:25:02 +00:00
Aleksey Marchuk
515419ac66 rpc: Add API to get method state mask
The new API will be used in the next patch
to prevent calling metods for the seconds time
when subsystem is initialized with config file

Signed-off-by: Aleksey Marchuk <alexeymar@nvidia.com>
Change-Id: I60ac8196e46ccb3b22b3af0607e1ba35a11a66a6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14406
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-15 08:25:18 +00:00
Damiano Cipriani
67bb33160a vbdev_lvol: Implement SEEK_[DATA,HOLE] io type
Two functions have been added to implement bdev io type
SEEK_DATA and SEEK_HOLE. Blob functions to find next
[un]allocated io_unit are used

Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: Ic3be3c0c86bd3010a23ba2681f0f00c62abcaaba
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14362
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-15 08:23:56 +00:00
Damiano
6defafc913 bdev: Add functions to [hole,data] seek
These functions start from a given offset and seek for next
data or for next hole. For bdevs that do not support seeking,
it is assumed that only data and no holes are present

Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: I6bc831970223333b25683f60ce3fcbbfebb5bb81
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14361
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
2022-09-15 08:23:56 +00:00
Damiano
d8a3dee1c1 blob: Add functions to find [un]allocated io_unit
These functions start from a given offset and seek for first
io_unit belonging to an allocated cluster or first io_unit
belonging to an unallocated cluster

Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: I0c632e2b3dfd2e96aa22e21796e25a36f2f55f9f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14360
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
2022-09-15 08:23:56 +00:00
Boris Glimcher
35f7f0ce1e nvme/tcp: Allow to choose SSL socket implementation
Adding `psk` field to `spdk_nvme_ctrlr_opts`

Adding `psk` parameter to `bdev_nvme_attach_controller` RPC

Change-Id: Ie6f0d8b04ce472e6153934e985c026acded6cdfc
Signed-off-by: Boris Glimcher <Boris.Glimcher@emc.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14046
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-14 07:44:53 +00:00
Michal Berger
f352e9f477 pkgdep/git: Skip -pre tags
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I85daca195e0b7c272bc18a610b20e25408989d5b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14496
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-14 07:43:47 +00:00
Changpeng Liu
6045458cb6 test/bdevperf: apply g_abort to job->abort
Also give an error log when the g_timeout_in_sec parameter
isn't set but g_abort is set.

Fix issue #2643.

Change-Id: Iafd9e94952319b89891e37bc6e9e0db6892ddd7d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14435
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-13 07:29:24 +00:00
yidong0635
1ad140676e test/aer: Complete requests before detaching.
The failed stack:
The calling processes:
	spdk_nvme_detach_async->nvme_ctrlr_detach_async->
	nvme_ctrlr_get_ref_count->nvme_ctrlr_proc_put_ref->
	nvme_ctrlr_remove_inactive_proc->nvme_ctrlr_get_current_process->
	nvme_ctrlr_remove_process

The proc->active_reqs list processing steps:
	spdk_nvme_ctrlr_process_admin_completions->
	spdk_nvme_qpair_process_completions->
	nvme_transport_qpair_process_completions->
	nvme_pcie_qpair_process_completions->
		*nvme_pcie_qpair_complete_tracker
		*nvme_pcie_qpair_insert_pending_admin_request
	and
		*nvme_pcie_qpair_complete_pending_admin_request
Issue assert:
	assert(STAILQ_EMPTY(&proc->active_reqs));
The assert means there are outstanding active requests in
proc->active_reqs list.

According the test app's coding style, it needs to foreach the dev and
complete the outstanding commands before spdk_nvme_detach_async
especially  it launched spdk_nvme_ctrlr_register_aer_callback .

Bring the aer tests back, to check it.

Fixes issue #2674.

Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: Ic0d3946f8c8743155cf3067aea1c9b4676748d2f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14433
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2022-09-12 07:25:41 +00:00
Kozlowski Mateusz
156b48b298 ftl: Trim tests
Unmap FIO and dd tests for FTL.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I6d01c4e729755c12d6c27159a7b12c575fc62e0a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13382
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-09 19:44:29 +00:00
Kozlowski Mateusz
66fe5f75bb ftl: Unmap functionality
Adds ability to send trim commands to FTL - only 4MiB aligned requests (both
for offset and length of request) will be processed. During a trim
operation an L2P page (containing 1024 4B entries, 1 per user LBA; which
is where the 4MiB alignment comes from) will be marked as unmapped.
After this point any L2P access to that page will actually set the
entries themselves as FTL_ADDR_INVALID. This is done to make the trim as
fast as possible, since for large requests it's probable that most of
the L2P pages aren't actually in DRAM.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I4a04ee9498a2a6939af31b06f2e45d2b7cccbf19
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13378
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-09 19:44:29 +00:00
Kozlowski Mateusz
ea80f87e0e ftl: Dirty shutdown tests
Add additional tests checking against dirty shutdown recovery. Also adds
'write after write' test - checking if two simultaneous writes to the
same LBAs return the same data before and after dirty shutdown.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Idcf9b51d9c00d0d065f7e9655387668f5eeb646d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13376
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-09 19:44:29 +00:00
Kozlowski Mateusz
42899f1a74 FTL: rewrite restore tests to use spdk_dd
Changing tests to using spdk_dd instead of using mounted nbd and dd on
it. Should be faster.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I17ffeae8441ba37d8e3348c715889fb568e8cd88
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14425
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-09 19:44:29 +00:00
Jim Harris
f869197b76 virtio: assert and ERRLOG for virtio-user dynamic mem allocations
We do not support dynamic memory allocation with the virtio-user
library - it results in SET_MEM_TABLE vhost messages for every
change which is not supported by the vhost target.

Add '-s 256' to vhost fuzz script, to ensure it does not
violate the new restriction.

This is a follow-on patch for issue #2596.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If851f53d7d670ac8443f0d9c8f4e3cbe82e0df7c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14249
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-09-09 13:06:15 +00:00
Ben Walker
bbd36d0dd8 test: Wait for 'nvme disconnect' operations to remove disks
Use waitforserial_disconnect to wait for all nvme disconnect operations
to actually finish.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I91f410d9a84db33cf5d8e9d7ecdd4e7a7003aae0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9314
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-09 12:57:55 +00:00
Shuhei Matsumoto
dd3460582b bdev/nvme: Rename check_multipath_params by check_io_error_resiliency_params
These checked parameters are necessary themselves even for single path
configuration.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ie1eb2f51eeec1dbc634c6bae462a41d4c209d6ac
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12052
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <yidong0635@126.com>
Community-CI: Mellanox Build Bot
2022-09-09 12:56:12 +00:00
Changpeng Liu
4bd26da22f UT/nvmf/subsystem: add subsystem state change UT cases
Change-Id: Icdbb95694d8eb8aa495aeab89dd36523ff5a1470
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13772
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
2022-09-08 07:20:33 +00:00
Krzysztof Karas
9b0730b326 run_openstack_tests.sh: move rpc_py declaration below source commands
Since commit: eaebf84102
rpc_py is defined in autotest_common.sh.
To avoid overriding the variable, move its declaration
below source commands.

Change-Id: I4f4e59120d4202c00a9fd2c685f3bf4febae48a5
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14387
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-08 07:20:17 +00:00
Michal Berger
7d4f7b7f01 test/common: Include list of running processes upon failure
This is done in order to have a bigger picture on the state of the
system (especially the phy nodes) when the autotest suite fails.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ib6e0b740b616284bda48ed82d92ac32236acaa3d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14261
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-08 07:16:39 +00:00
Jim Harris
004ab98dac test/unit: add spdk_json_find test for an array
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic63277fbf7f82b58231b2554939a7faca8e4b41e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14162
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-09-07 14:37:04 +00:00
Artur Paszkiewicz
1738488e41 ftl: p2l checkpointing
Since base device doesn't require VSS, FTL introduces a mechanism that
will allow for recovering both the P2L and write pointer of open bands
after a dirty shutdown. After writing 1MiB of data to a band, a 4KiB
block describing the P2L will be persisted to cache device, effectively
emulating VSS for the base device.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: Ic6be52dc09b237297a5cda3e752d6c038e98b70e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13367
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-07 00:08:34 +00:00
Artur Paszkiewicz
36049672a3 ftl: sequence id tracking
Track the relative sequence of opening and closing bands and chunks.
Necessary for detecting the most recent user data during dirty shutdown recovery.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I682030e58284d7b090667e4e5a9f4bbc7615708a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13366
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-07 00:08:34 +00:00
Krzysztof Karas
35a9ddb5b6 test: restore rpc_py to point to rpc.py instead of rpc_cmd
rpc_cmd function does not tolerate piping arguments into it,
so code execution skips the parts responsible for generating
a return code. The code is equal to nothing or "", which then
is compared with a value, ultimately erroring out the whole test.
To avoid these problems restore rpc_py varible declaration
in multiconnection.sh and ext4test.sh scripts.

Additionally added a comment in compress.sh to match ext4test.sh
and multiconnection.sh.

Change-Id: I4fb5f737d4e97d602f522b810363bc0d4522bf0d
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14356
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-06 18:01:42 +00:00
wanghailiangx
809ae055b4 usage: Add ] for [-G enable debug logging
Change-Id: I02944e2c40c9a5d288d20b53548371d5e7b97069
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14334
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-06 07:16:32 +00:00
Ben Walker
34c48f1b3b accel: Do not refer to the "framework" as "engine"
The word engine was both used (interchangeably with module) to refer to
the things that plug into the framework and to the framework itself.
This patch eliminates all use of the word engine that meant the
framework. It leaves uses of the word that meant "module".

Change-Id: I6b9b50e2f045ac39f2a74d0152ee8d6269be4bd1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13918
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-09-06 07:16:17 +00:00
Shuhei Matsumoto
cdf61c2f22 nvme: Polls only the qpair if ctrlr is not fabrics when connecting synchronously
For non-fabric controllers, the corresponding I/O qpairs are simply
re-enabled at controller reset.

This had a issue when I/O qpairs span multiple threads and poll group
is used.

spdk_nvme_ctrlr_reconnect_poll_async() calls
nvme_transport_ctrlr_connect_qpair() with qpair->async being false.
Then nvme_transport_ctrlr_connect_qpair() calls
spdk_nvme_poll_group_process_completions() until the qpair is connected.
spdk_nvme_poll_group_process_completions() may poll other qpairs.
This may cause I/O to complete on a wrong thread.

For PCIe controller, spdk_nvme_poll_group_process_completions() calls
spdk_nvme_qpair_process_completions() simply for each qpair.

Hence change nvme_transport_ctrlr_connect_qpair() to call
spdk_nvme_qpair_process_completions() if the controller is non-fabrics.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Ieb270c2fb154124021ef6d25577b817d05e5ca9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14295
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-09-05 12:50:00 +00:00
Konrad Sztyber
ab58ddf107 sock: make impl_name const char * in all functions
There's no reason for this parameter to be non-const and it makes this
functions pain to use when you want to hardcode a specific sock
implementation.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifed4426a02ab54cbd51c8a2051b1eac010f86db9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14303
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-05 12:49:28 +00:00
Shuhei Matsumoto
1d58eb038b nvme_rdma: Release poller from poll group when qpair is actually disconnected
If the being disconnected qpair is the last of a poller of a poll group,
CQ is destroyed and the poller is released before the qpair is actually
disconnected.

This patch destroy CQ and release the poller after the qpair is actually
disconnected.

One exception is when spdk_nvme_ctrlr_free_io_qpair() is called to a
connected qpair. In this case, the qpair is removed from a poll group
before the qpair is actually disconnected. In this case, destroy CQ and
release the poller when the qpair is removed from the poll group.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Idf266bbb6dbb40f04ae6313db724fabf80865763
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14253
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-05 12:49:11 +00:00
Kozlowski Mateusz
70c2bd8858 FTL: Add restore tests
Additional tests testing clean shutdown/startup capability (both fast
and slow shutdown paths). Additionally FIO tests can now run on the same
FTL instance, instead of creating a new one for each test.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I20cbd2fe0303b3e75bf7efca1623e223e2611f17
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13365
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-02 17:40:09 +00:00
Kozlowski Mateusz
cbd7ae6df7 FTL: Add metadata restore functionality
Adds necessary functions for setting up the state of FTL components
based on loaded in metadata.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I3a4c05230c877850e61d4f31d495d38121d27b3f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13362
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-02 17:40:09 +00:00
Kozlowski Mateusz
e7e5bc07b2 FTL: Add initial L2P cache logic
L2P cache allows for partial storing of L2P in memory, paging in and out
as necessary, lowering the total memory consumption.

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I727fec9d2f0ade4ca73e872d62a2ec10cfdb0a88
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13353
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-02 17:40:09 +00:00
tongkunkun
bb432b4eea json: fix parsing json problems when json config is invalid.
Add parsing json as invalid cases:
1.json content that not enclosed in {}, it should be parsed as invalid, e.g.

"abc":"not encloesed in {}"

2.json content that 'subsystems' not associate with array, it will report error and return failure, e.g.

{"subsystems":"123"}

3.handle other invalid json formats, report and return failure, e.g. duplicate keys.

Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}.

json config with content:
1."not enclosed in {}"
2."'subsystems' not be an array"
3."duplicate key in json"
and some other invaild cases will be regarded as invalid json config, and will fail to start app.

Fixes #2599

Signed-off-by: tongkunkun <tongkunkun_yewu@cmss.chinamobile.com>
Change-Id: I02574c9acd7671e336d4c589ebbff8ed21eb3681
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13754
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-02 07:32:21 +00:00
Jim Harris
8216e70a14 Revert "test/vhost/initiator: Don't use spdk_tgt to generate json config"
This reverts commit 0c30154eb5.

Make a change in the revert to pre-allocate 1GB of memory for the
spdk_tgt initiator process, which is required for the virtio
initiator.

Fixes issue #2596.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id594fa1ccb2daf535b1aaaef0a397bda2ec98579
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13828
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-01 18:30:56 +00:00
Krzysztof Karas
c527d09f30 hotplug: add new software hotplug test
Add a new test to sw_hotplug.sh based on SPDK target
in addition to examples/hotplug application.

Change-Id: I352ee4dfe934fa88082101bb92c6db41da10c10c
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>

Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10088
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-01 11:15:03 +00:00
Krzysztof Karas
eaebf84102 autotest_common.sh: change rpc_py to rpc_cmd
Change rpc_py to use rpc_cmd function.
The former needs to be defined explicitly in the scripts
that uses function, while the latter is a function
already available in autotest_common.sh.
Additionally delete unnecessary rpc_py declarations
and override rpc_cmd where additional arguments are
needed (like -s sock_name.sock).

Change-Id: I835f7eb308cd77658a2a425e153ab7ca460b5b61
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12872
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-01 11:15:03 +00:00
Konrad Sztyber
f0441b29db vmd: rename enable_vmd RPC to vmd_enable
The new name is consistent with the naming scheme of
<subsystem>_<action> that all of our other RPCs use.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I2cae7af5715add8eba26501cd192a6ac4884ec69
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13952
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-01 08:48:32 +00:00
Konrad Sztyber
55bdd88506 env/pci: add detach() callback to pci_device_provider
This makes it possible to notify other PCI device providers (VMD) that a
PCI device is no longer used.  The VMD will driver will unhook that
device and free any resources tied to it.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I42752afbb371a1d33972dac50fd679f68d05b597
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13887
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
2022-09-01 08:48:32 +00:00
Konrad Sztyber
4b08c07a62 env/pci: call driver callback in pci_hook_device
Now that we have a attach_device() callback, the devices can be hooked
during spdk_pci_device_attach().  With DPDK, driver->cb_fn() is called
in pci_device_init(), so we need to do the same in
spdk_pci_hook_device().

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iada8b83ce7592aa62561530192072a50ec3a904b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13884
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tom Nabarro <tom.nabarro@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-01 08:48:32 +00:00
Michal Berger
51a8f16e72 test/vhost: Fix parse_fio_results() for a single client
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I5aa53d0d06564030a3dbae8a8aa5c6179e972980
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13713
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
2022-09-01 08:25:55 +00:00
Michal Berger
c1cb6b0d4f perf/vhost: Skip comments in disk map files
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ibb713d5a597c83730cc81f8ab2d7a16334407ff8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13740
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
2022-09-01 08:25:55 +00:00
Michal Berger
fb41398ae7 test/vhost: Replace error() with simple echo
In this particular case, error() is called before it's declared,
hence bash complains that it's missing, failing to print out the
actual error message.

To not shuffle code around, simply replace this call with an echo
to stderr.

Signed-off-by: Michal Berger <michallinuxstuff@gmail.com>
Change-Id: I6f0a6e5b0dfd6893aee6a66e6efdb9874245fafb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12427
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
2022-09-01 08:25:55 +00:00