Commit Graph

1845 Commits

Author SHA1 Message Date
Konrad Sztyber
5e911c0c06 check_format: search all declarations in naming conv check
Look for all declarations instead of only the ones introduced in the
tested patch.  It makes this test pass if a patch adds definition for a
function that already has a declaration in the header file (e.g.
the implementation depends on some #ifdef).

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I735cf10d4524318009f8c4282a86e4137bf8c769
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15428
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-18 08:40:05 +00:00
Krzysztof Karas
b5bdbbb959 bdev: enable bdevs based on physical device to generate UUID
Add an option "--generate-uuids" to bdev_nvme_set_options
RPC to enable generation of UUIDs for NVMes devices that
do not provide this value themselves. The identifier is
based on a serial number of the device, so a bdev
using this NVMe will always be assigned the same UUID.

Part of enhancement from #2516.

Change-Id: I86d76274e5702d14ace89d83d1e9129573f543e2
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15151
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
2022-11-18 08:38:13 +00:00
Rafal Stefanowski
8000cedbe3 bdev/ocf: Add runtime cache bdev flush support
Introduce two RPC calls for starting flush and getting flush
status of OCF cache bdev:
- bdev_ocf_flush_start
- bdev_ocf_flush_status

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
Signed-off-by: Rafal Stefanowski <rafal.stefanowski@intel.com>
Change-Id: I1d659da6fc51396e0d070af35372ee130c40ae8b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8961
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-11-18 08:38:03 +00:00
Michal Berger
c730f1040e perf/nvmf: Include CPU power measurements in the results
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I4bdfde28d4ee7b4987bda5f8152ed408c9f77516
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15183
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
6fb5eae644 perf/pm: Collect power statistics per CPU socket
This uses RAPL's powercap interface under sysfs. The alternative is
to use MSRs directly, but with this we don't have to bother about
different cpu models, etc. as kernel does that for us here.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I91ed5d67edf2669b9d7b271bbc02ecc61a6a3ea2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15182
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
daeadb177a perf/pm: Add number of all readings to the all_* log
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Id9f249926cf8a1ba3648e7a732b2a368dfb3d9e8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14655
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
f83a2d3aba perf/pm: Allow for fractional interval
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If5ddb9d7d14f3d7acc93490deade5e5524afe3b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14629
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
eb0159cc4e perf/nvmf: Align power measurement with other tools
Run collect-bmc-pm as a thread with execution time limited by count
and interval. This aligns it with the way how other tools - sar,
pcm - are used during the test.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I48fa4ea1be69f9dcae71bf01c3214102ab329bac
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14628
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Karol Latecki
ede1049911 scripts/nvmf_perf: move get_nvme_devices to SPDK Target
These functions are used only by SPDKTarget class, so
move them out of common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I489443f76fbdf4cb3def19a557f4a953a098bbce
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14853
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
0b995d4aba scripts/nvmf_perf: move get_nvme_devices to Kernel Target
This function is used only by this class, remove it
from common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I6cb01e68f0005ddc1604ff4014c6b346da3a970e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14852
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
3a359b7923 scripts/nvmf_perf: move results parsing from Target class
Results parsing is not Target specific. Result
files get copied from initiator the system where
the script is run. In future we might want to
add a feature to work with remote Targets which
would make the parse method fail.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I215391e0ff7a3ac0881f5a092513dd087fbd2e2c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14845
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
93e056ba6c scripts/nvmf_perf: clean up common.py
Remove unused imports and unused function.
Move nvmet_command function to Kernel Target
as it's specific only to this class.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I30699ca1d8541ff2f57ea609e5caf0304feb4282
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14740
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
4e07343fea scripts/nvmf_perf: common connect & disconnect methods names
Rename "kernel_init_connect/disconnect" methods
and create equivalents in SPDK Initiator class.

This allows to skip conditional checks in main
section of the script.

Change-Id: I23f620670d628c602dfdc1f24b2ddf493993ae12
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14739
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
b0760bd3d9 scripts/nvmf_perf: do not discover subsystems with nvmecli
Information about Target subsystems (port / nqn name /
ip address) is already saved at the time of their creation.
Avoid spawning nvme-cli processes and instead just filter
matching subsystems based on available information.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I24d16315d67e5eaade48790b7fae45664110afcb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14737
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
4f74eb12a3 scripts/nvmf_perf: use tuple to store subsystem information
Use tuple instead of list as this information is
supposed to be immutable later on.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Id9806202d06368f09ede0c0d903924b101d3b86c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14736
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
80e92723f5 scripts/nvmf_perf: use standard library logger
Change-Id: I1fa41339ff76fbba97071b62f311699ad870e199
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14735
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
3db789597d scripts/nvmf_perf: use sudo to print sysctl.conf
Permissions on /etc/systctl.conf file might be
for owner-only (root). Use sudo to make sure we
can read the file.

Change-Id: I98d77ac87f196aac82000cb42553285f22f8cdcd
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14390
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:15 +00:00
Krzysztof Karas
29784f35cd bdevperf: promote bdevperf from test to example application
bdevperf application is utilized in multiple test scenarios
and most prominently in SPDK performance reports.
Similar to perf and fio_plugins, it is used to measure performance.
It is intended to represent the expected behavior of users
application that use SPDK bdev layer.

Applications under --enable-tests are intended for specific test
scenarios and user interaction is usually via a test scripts.
Meanwhile --enable-examples are used more broadly to teach and
show usage of SPDK libraries.

As an added benefit with this patch, its possible to compile
bdevperf without need to satisfy additional requirements to
compile all the test applications.

Change-Id: I9aaec1f9d729a1cdee89b5fdc365d61c19b03f82
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14558
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-11-16 09:52:41 +00:00
Michal Berger
66c590e0b4 pkgdep/rhel: Upgrade pip under centos8, rocky8
Do this prior installing any pip package to make sure they are stable.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I842252fe33b62dddd16858806a7d334cf2139fac
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15292
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-11-10 13:20:20 +00:00
Mykola Golub
8a5d487d7e scripts/pkgdep: add RBD support
Adds "--rbd" option to install build dependencies for the rbd bdev.

This will be used by the Ceph NVMe-oF gateway.

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>
Change-Id: I7926433aaa398873133dfffea3c401ce3c1f7813
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11134
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-11-10 08:29:10 +00:00
paul luse
17538bdc67 add (c) and SPDX header to python files as needed
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc)

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line.  Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I6cd3f18d1b469d5ef249d26ddb2923ca6b970bd4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15208
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-10 08:28:53 +00:00
paul luse
a6dbe3721e update Intel copyright notices
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc).  Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line.  Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
2022-11-10 08:28:53 +00:00
Krzysztof Karas
344249069d event: add runtime cpu lock configuration
Allow CPU core locks to be enabled and disabled
during runtime. This feature will be useful
in cases like SPDK hot upgrade, where
locking should be disabled temporarily.

Change-Id: I9bc7292fd964abffc7214d074d191f38b13583c3
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15031
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2022-11-09 08:18:32 +00:00
Krzysztof Karas
0af934b38c event: add CPU lock files
When running SPDK application on a given set of
CPU cores, create lock files for each of them.
This wil prevent user misconfiguration and
assigning a core to more than one SPDK instance.

The introduced mechanism is based on device locks
implemented in spdk_pci_device_claim() function.

Add a command line option to disable lock files.
This feature will be useful in cases where differing
CPU cores is impossible (eg. setup with only one core
available).

The patch also fixes all existing cases of overlapping
core masks.

Change-Id: Ie9aacb7523a3597b9aa20f2c3fa9efe4db92c44c
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2022-11-09 08:18:32 +00:00
Kozlowski Mateusz
311d014d17 scripts: Update spdk_print_io_devices and spdk_print_threads gdb macros
The structs io_device and spdk_io_channel were moved from using tailq list to rb tree,
moving the parsing in gdb functions to use those too.

Fixes issue #2763

Signed-off-by: Kozlowski Mateusz <mateusz.kozlowski@intel.com>
Change-Id: I8fa5a69b5004e3d86fdfa201c2ba47e06fa47cfa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15330
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2022-11-09 08:16:49 +00:00
Michal Berger
b35a6f39a6 pkgdep/rhel: Make sure pip3 is installed in first instance
There was a code that attempted to to use pip3 before we try to
actually install it. Change the order.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I5dc74b79b3adf132589bbf9ef10360d2eeed51ba
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15293
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-08 08:21:59 +00:00
Michal Berger
71605a520d scripts/setup: Fix extglob match
The actual target files come with a .* suffix.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I1c6056c81261ae97bd272d0d99ad915ab4bb870e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15294
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-11-08 08:21:59 +00:00
Shuhei Matsumoto
3d2a3ee4df bdev/error: Support injecting data corruption into I/O buffer
Support injecting data corruption for processing read/write I/O.

Add two parameters, corrupt_offset and corrupt_value, and an error
type corrupt_data.

For write I/O, inject data corruption before submitting write I/O
to the underlying bdev. For read I/O, inject data corruption after
read I/O successfully returned.

Cause the data corruption by xoring the value at the corrupt_offset
with the corrupt_value. corrupt_value is required to be non-zero to
cause data corruption necessarily.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I67d8d252b06d7a221413e74996f7f894e6519556
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15028
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
2022-11-08 08:21:22 +00:00
Jim Harris
36aed79cc4 crypto: fix AET_XTS misspellings
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idaec6f6ca2c3492898ecf808e10642b4cbd35b78
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15200
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-11-02 10:52:24 +00:00
GangCao
e28e247954 RPC/Bdev: display the per channel IO statistics for required Bdev
Add a new parameter "-c" to display the per channel IO statistics
for required Bdev

./scripts/rpc.py bdev_get_iostat -b Malloc0 -h
usage: rpc.py [options] bdev_get_iostat [-h] [-b NAME] [-c]

optional arguments:
  -h, --help            show this help message and exit
  -b NAME, --name NAME  Name of the Blockdev. Example: Nvme0n1
  -c, --per-channel     Display per channel IO stats for specified device

This could give more intuitive information on each channel's processing
of the IOs with the associated thread on the same Bdev.

Please also be aware that the IO statistics are collected from SPDK
thread's related channel's information. So that it is more relating
to the SPDK thread. And in the dynamic scheduling case, different
SPDK thread could be running on the same Core.

In this case, any seperate channel's IO statistics are returned to
the RPC call and if needed, further parse of the data is needed to
get the per Core information although usually there is one thread
per Core.

On the other hand, user could run the framework_get_reactors RPC
method to get the relationship of the thread and CPU Cores so as
to get the precise information of IO runnings on each thread and
each Core for the same Bdev.

Change-Id: I39d6a2c9faa868e3c1d7fd0fb6e7c020df982585
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13011
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2022-10-28 06:51:19 +00:00
Shuhei Matsumoto
00bff560dd bdev/malloc: Support protection information for read and write
For write, verify DIF/DIX before submission and for read, verify
DIF/DIX after successful completion.

As same as the NVMe bdev module and the NULL bdev module, DIF/DIX
verification is done based on the DIF type and DIF insert/strip is
not supported.

In near future, the bdev I/O APIs bring an I/O flag to the underlying
bdev and the malloc bdev module will be able to decide DIF/DIX
verification based on the I/O flag.

One important feature is to setup protection information when
creating a malloc disk. Otherwise, all initial reads will fail
if protection information is enabled.

For users, add some explanation about the dif_type parameter
into doc/jsonrpc.md.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I93757b77c03cade766c872e418bb46d44918bee2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14985
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-10-28 06:49:40 +00:00
Shuhei Matsumoto
aef00d4420 bdev/malloc: Support both of interleaved and separated metadata
The malloc bdev module supports both of interleaved and separated
metadata in this patch.

Different from the NULL bdev module, opts->block_size is a data block
size and a block size is caculated internally as a sum of
opts->block_size and opts->md_size if opts->md_interleave is true, or
opts->block_size otherwise. This will be more intuitive. Additionally,
opts->md_size accepts only either of 0, 8, 16, 32, 64, or 128.

Protection information (T10 DIF/DIX) will be supported in the
following patches.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Icd9e92c8ea94e30139e416f8c533ab4cf473d2a8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14984
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
2022-10-28 06:49:40 +00:00
Michal Berger
0252bdbf60 scripts/setup: Always cleanup target directories
Currently, in case target directories are empty the directories
themselves are not being purged. Change that, as there is no
need to keep them around.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ia1a9d60a91508e74b7e106e0217414a8927029b6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15153
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-10-27 09:24:38 +00:00
Michal Berger
cb8174dc4a scripts/setup: Ignore potential readlink failure upon cleanup
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I21ba900f333f0dc3710fcf93fb7fa011301a51bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15152
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-10-27 09:24:38 +00:00
Michal Berger
d88f8ed409 scripts/sync_dev_uevents: Adjust termination of the udevadm
This is needed for newer Bash where $! becomes the PID of the
udevadm instead of being its child. Not killing udevadm properly
in this case may be problematic in case the script was run from
the ssh session - depending on the setup, if not all children
of said session are dead, ssh may block forever waiting for them
to terminate (which won't happen as udevadm gets attached to 1
process instead of being reaped).

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I3248d0937e9dd919c054a97473f9bb998aa8ca63
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15147
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-10-27 09:24:17 +00:00
Denis Barakhtanov
3d7851e011 bdev/daos: Add object class parameter
A new `oclass` parameter allow to specify DAOS DFS object class that is
responsible for data redundancy and protection.

Examples of the object classes could be found here: https://github.com/daos-stack/daos/blob/master/src/include/daos_obj_class.h

The default value is OC_SX for the max IOPS.

Signed-off-by: Denis Barakhtanov <denis.barahtanov@croit.io>
Change-Id: Ia48681832458c2266eb7c3bcae0df2055e59e309
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15006
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-10-25 07:12:06 +00:00
Michal Berger
af5b654df8 test/setup: Adjust use of nullglob, extglob in cleanup_linux()
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I4e5a28f916c5855fefba8594d549b56c63d71013
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15094
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-10-25 06:57:59 +00:00
Michal Berger
8c19851209 scripts/setup: Enable extglob, nullglob in parser's global scope
This done due changes in latest Bash releases (>=5.2) which is
currently shipped with fedora36.

Fixes issue #2740.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Id6e6f00eb4ae8b7f2b110848fd55e710368fcf94
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15081
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: wanghailiang <hailiangx.e.wang@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
2022-10-25 06:57:59 +00:00
Denis Barakhtanov
bdc683aaa9 bdev/daos: add resize rpc call
Signed-off-by: Denis Barakhtanov <denis.barahtanov@croit.io>
Change-Id: I71d643733e31eb2229649cff3db610d5bee07796
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14921
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
2022-10-21 07:19:17 +00:00
Yuhua
4c6a2e3daa bdev/aio: implement read-only
Support Aio bdev 'readonly' option in RPC call. The read-only flag
can be dumped from the bdev info. Any writes on a read-only aio bdev
will be fail.

Signed-off-by: Yuhua <yuhua@smartx.com>
Change-Id: I939f72479f8953a3678a8df3083ecce0f96844fb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14955
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-10-21 07:18:15 +00:00
Michal Berger
28bfb8763b scripts/setup: Don't overwrite already allocated number of hugepages
Don't touch existing hp allocations when they already meet the NRHUGE
requirement. Introduce new environment switch, SHRINK_HUGE, to
override that new behavior.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I0cd124b98c3deb7c21b4fbd57529d0995978c2e5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14912
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-10-19 07:49:22 +00:00
Alexey Marchuk
8d27a5f7ed check_format: Update a hint for Darwin tools
Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: I483cb49c56b02cd9d24dff2c71395a2e1b0a6a80
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14865
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-10-19 07:47:58 +00:00
Pawel Piatek
f658f463a4 autobuild: tiny version
Allow to run AUTOBUILD test in two versions:
* full - original one
* tiny - minimalistic, currently only check format and
         check so deps

We want to reduce first phase of per-patch (called Pre-Tests)
by running only basic tests. To accomplish that we will run
autobuild test in tiny version. Full version will be executed
in second phase of per-patch (called Tests).

Signed-off-by: Pawel Piatek <pawelx.piatek@intel.com>
Change-Id: I87d503ec272080ab238cf5129c44f92d8bb690ed
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14481
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-10-17 10:50:40 +00:00
Luo Yifan
1bab7cbc7d scripts: Fix a clerical error in gdb_macros.py
The add operation is meaningless here. New ns should be added to the returned string, so replace + with +=.

Signed-off-by: Luo Yifan <luoyifan_yewu@cmss.chinamobile.com>
Change-Id: I7f0eaaec48d544e21892320dfc1451ebd2e7174e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14899
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-10-13 13:51:21 +00:00
Jim Harris
274a9ffbd5 setup.sh: redirect 'hash lsblk' output to /dev/null
We are checking if lsblk exists already, but when it
doesn't let's not emit that detail to the user.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I34fedcc97fb7091229a11b2592cb7bd21a032b5b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14834
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2022-10-12 06:35:55 +00:00
Simon A. F. Lund
3bf8f8a39b pkgdep: bump liburing to v2.2
liburing v2.2 provides support for initialization of rings with "big"
submission and completion entries. Big entries allow embedding NVMe
submission and completion entries in the io_ring sqe/cqe and thereby an
enabler of "io_uring_cmd".

xNVMe utilizes the above. xNVMe v0.4.0 "vendored"/bundled liburing 2.2.
However, the bundling of liburing inside the xNVMe library is removed in
v0.5.0 to avoid linker issues. Instead, xNVMe v0.5.0 links liburing on
the system with discovery via pkg-config.

Thus, this change to update the liburing pkgdep to v2.2, in preparation
for upgrading the xNVMe submodule to v0.5.0.

Signed-off-by: Simon A. F. Lund <simon.lund@samsung.com>
Change-Id: Ibbacbcfa675f82257d41790b68c39dd7b1a82908
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14676
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-10-03 08:43:53 +00:00
Changpeng Liu
5004d7b8e8 module/vfu_device: add virtio-scsi emulation
Here we use vfu_tgt library and emulate a virtio-scsi device
as the next use case.

Compared with vhost-user-scsi, the packed ring is supported with this
patch.

Example usage method:

1. scripts/rpc.py bdev_malloc_create -b malloc0 $((512)) 512
2. scripts/rpc.py vfu_virtio_create_scsi_endpoint vfu.0 --cpumask 0x1 --num-io-queues=4 \
                                                  --qsize=128 --packed-ring
3. scripts/rpc.py vfu_virtio_scsi_add_target vfu.0 --scsi-target-num=0 --bdev-name malloc0
4. Start QEMU with '-device vfio-user-pci,socket=/spdk/vfu.0'

Change-Id: I8f35d1d21aaec34844d6ddb59dc997a64f141179
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12673
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-29 19:42:56 +00:00
Changpeng Liu
23ef63882c module/vfu_device: add virtio-blk emulation
Here we use vfu-tgt library and emulate a virtio-blk device
as the first use case of vfu-tgt library.

Usage example with QEMU:

1. build/bin/spdk_tgt
2. scripts/rpc.py bdev_malloc_create -b malloc0 $((512)) 512
3. scripts/rpc.py vfu_virtio_create_blk_endpoint vfu.0 --bdev-name malloc0 \
                                                 --cpumask=0x1 --num-queues=2 \
                                                 --qsize=256 --packed-ring
4. Start QEMU with '-device vfio-user-pci,socket=/spdk/vfu.0'

Change-Id: I45e45360c669584583b0b8a3f83250ab6c48efec
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12315
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
2022-09-29 19:42:56 +00:00
Changpeng Liu
da231290b2 lib/vfu_tgt: add library for PCI device emulation
Previously SPDK use libvfio-user library to provide emulated NVMe
devices to VM, but it's limited to NVMe device type only.  Here we
add SPDK vfu_target library abstraction based on libvfio-user which
supports more PCI device types.

We will add virtio-blk and virtio-scsi devices emulation based on
vfu_tgt library in following patches, actually this library can
support NVMe emulation too, due to the fact that the NVMe emulation
is already exist, so we will keep the NVMe emulation which based on
libvfio-user directly as it is.

Change-Id: Ib0ead6c6118fa62308355fe432003dd928a2fae9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12597
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-09-29 19:42:56 +00:00
Jim Harris
1eb35ac7e3 env_dpdk: add copies of 22.07 pci-related header files
Moving forward, we want to still be able to run against
<= 22.07 versions of DPDK, which exposed the necessary
data structures in public header files.  But since we
will be building against newer versions of DPDK which
don't expose them publicly, we need a copy of the 22.07
header files in our tree.

Exclude these header files from astyle and POSIX include
file checks in check_format.sh

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Icd8a067af41a2ba031ce8f875a8a2b63f722ab69
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14683
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-09-29 15:32:24 +00:00