Commit Graph

206 Commits

Author SHA1 Message Date
Michal Berger
3f912cf0e9 misc: Fix spelling mistakes
Found with misspell-fixer.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If062df0189d92e4fb2da3f055fb981909780dc04
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15207
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-09 08:16:18 +00:00
Kamil Godzwon
2be0335489 scripts/run_vhost_test: fix shellcheck SC2294 warning
The warning appeared during Fedora 37 VM image testing

Signed-off-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I2ceff22d08614fec0c27b9931d71753f57cf99bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15794
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-12-08 12:56:18 +00:00
Karol Latecki
de7c369173 scripts/nvmf_perf: stop checking self.mode in parent class
Make sure all attributes are initialized in both classes.
If something is not supported or works a bit differently -
move it to children's class method.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I963ef1e9fe4288992a15f65671c1b85e8e7f7fd6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14882
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
1e629c3219 scripts/nvmf_perf: allow dashes in instance names
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I4183549cc55b437a435fb024c4a9d234c456ccab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15041
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
894360fbf2 scripts/nvmf_perf: make Initiator num_cores actually optional
"num_cores" is described as optional in readme file.
For SPDK Initiator class it was required instead of
optional, and Kernel Initiator did not even take this
parameter into consideration.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Id6874ee8614aefdd9ee4a55dd200a6deab504b7a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14881
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
da55cb8756 scripts/nvmf_perf: CPU and power utilization to csv output
Find matching SAR and PM script output files, parse
them and put calculated average from accross all
workload iterations into final CSV with aggregated
results.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I9d3c445941e54eefbe93dd47d152d2ede22c9560
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15467
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
a6f6454073 scripts/nvmf_perf: fix enable_sar option
Wrong key was used to assign value from json config.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I3814b385824b435ae39cdf0a47cca42735873e86
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15757
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-02 10:40:21 +00:00
Karol Latecki
61692e8150 scripts/nvmf_perf: use common prefix for all target measurements
This should make it easier later to group matching
result files when parsing results.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I8d3c5641c8b71b6027e1f200ce53426a46e5234d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15466
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
9b3f5e98d5 scripts/nvmf_perf: enable DPDK memory dump by default
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I65e8299cb887edd23c9d8b29535d596c419bc63d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15381
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
babef5b127 scripts/nvmf_perf: gather additional metrics for all runs
Run additional measurements (PM, SAR, PCM, BWM-NG, DPDK mem)
for each configured test run.
For example: until now if fio parameter "retry" was set to 3
fio would run the workload 3 times, but additional measurements
would only be done for the first run.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I5a8aaa8eeb28f2a24f47a41650f9e2bd14a298dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15380
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
ecbb99e4d7 scripts/nvmf_perf: enable PM by default
Enable PM measurements by default.
Delay time and sample count variables are now taken
from fio ramp_time and run_time parameters, while
sample interval is hardcoded to 1 second.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Ibc08c690a8c8eea7591a5553b2478283c7574eec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15520
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
d9027e8140 scripts/nvmf_perf: group restore functions
Group and move all restore_* functions to classes
stop method.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I92cde5865a28343dbace702f54e4e06e4c2ba6cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14855
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
3d9b062852 scripts/nvmf_perf: enable PCM measurements by default
Enable PCM cpu, power and memory measurements as
default and make them run during fio workload.

Change-Id: I4d42cf9df8385bfaa3b5e8fac59283ef8be0e98b
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15341
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
b9403ae58b scripts/nvmf_perf: enable BW monitoring by default
bwm-ng interface bandwidth monitoring is now enabled
by default. Samples are captured  every second during
fio run_time duration.

Change-Id: I868f01d23e648e855161e6711236a68dcff8ff5a
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15340
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
90b95e2813 scripts/nvmf_perf: enable SAR by default
Enable SAR CPU measurements by default.
Delay time and sample count variables are now taken
from fio ramp_time and run_time parameters, while
sample interval is hardcoded to 1 second.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Icf2866820250dd6bbc5619b2777f6fe139f2a588
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15339
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
2e449d4166 scripts/nvmf_perf: group measurement options
Groupt variables used for enabling addtional
measurements together for better readibility.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: If61de427294fea6ae9a2d5c53f74b6c9e5aa3e6a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15338
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
f5d6d59482 scripts/nvmf_perf: add allowlist and blocklist
Use "allowlist" and "blocklist" options in JSON
configuration file to specify which NVMe drives
are OK to use when setting up Target side. If no
list is specified the user can continue with the
test and use all available NVMe drives by passing
"-f" option when running the script.

Previously the default behaviour was to use all
NVMe drives available in the system for test, which
could cause data loss on those drives.

This is a failsafe primarily aimed at KernelTarget
class, as SPDKTarget is already safer to use because
of existing checks in "setup.sh" script which must
be run prior to executing NVMe-oF performance test
script.

Change-Id: I5ff93672a92cb09f2aef8355542ad197d96e14e1
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14854
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
241d9c0a8a scripts/nvmf_perf: fix logging issue
"log_print" was removed in previous commit and
switched to Python's logging module instead.
This line was probably a result of a faulty rebase.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I6bbb2c37a947d0169689a014b2e110884bb70fdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15519
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-11-29 11:35:52 +00:00
Michal Berger
588dfe314b Add SPDX header to various files
They were missed by the initial set of patches which introduced this
header as a mandatory one across different types of files.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I3f9b37d41298c843e1648e72fe8593768ccd37e0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15423
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-11-29 08:27:51 +00:00
paul luse
eb53c23236 add (c) and SPDX header to bash files as needed
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc)

For intel copyrights added, --follow and -C95% were used.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I2ef86976095b88a9bf5b1003e59f3943cd6bbe4c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15209
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-29 08:27:51 +00:00
Karol Latecki
f67c74a2cf scripts/nvmf_perf: import missing subprocess exception
CalledProcessError was removed from common.py and
not moved to run_nvmf.py in:
https://review.spdk.io/gerrit/c/spdk/spdk/+/14740

Change-Id: Ib35c60aa779711f40c6fda990c721491ae000b43
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15543
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-11-24 10:10:16 +00:00
Michal Berger
c730f1040e perf/nvmf: Include CPU power measurements in the results
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I4bdfde28d4ee7b4987bda5f8152ed408c9f77516
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15183
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
6fb5eae644 perf/pm: Collect power statistics per CPU socket
This uses RAPL's powercap interface under sysfs. The alternative is
to use MSRs directly, but with this we don't have to bother about
different cpu models, etc. as kernel does that for us here.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I91ed5d67edf2669b9d7b271bbc02ecc61a6a3ea2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15182
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
daeadb177a perf/pm: Add number of all readings to the all_* log
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Id9f249926cf8a1ba3648e7a732b2a368dfb3d9e8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14655
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
f83a2d3aba perf/pm: Allow for fractional interval
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If5ddb9d7d14f3d7acc93490deade5e5524afe3b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14629
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
eb0159cc4e perf/nvmf: Align power measurement with other tools
Run collect-bmc-pm as a thread with execution time limited by count
and interval. This aligns it with the way how other tools - sar,
pcm - are used during the test.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I48fa4ea1be69f9dcae71bf01c3214102ab329bac
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14628
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Karol Latecki
ede1049911 scripts/nvmf_perf: move get_nvme_devices to SPDK Target
These functions are used only by SPDKTarget class, so
move them out of common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I489443f76fbdf4cb3def19a557f4a953a098bbce
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14853
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
0b995d4aba scripts/nvmf_perf: move get_nvme_devices to Kernel Target
This function is used only by this class, remove it
from common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I6cb01e68f0005ddc1604ff4014c6b346da3a970e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14852
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
3a359b7923 scripts/nvmf_perf: move results parsing from Target class
Results parsing is not Target specific. Result
files get copied from initiator the system where
the script is run. In future we might want to
add a feature to work with remote Targets which
would make the parse method fail.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I215391e0ff7a3ac0881f5a092513dd087fbd2e2c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14845
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
93e056ba6c scripts/nvmf_perf: clean up common.py
Remove unused imports and unused function.
Move nvmet_command function to Kernel Target
as it's specific only to this class.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I30699ca1d8541ff2f57ea609e5caf0304feb4282
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14740
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
4e07343fea scripts/nvmf_perf: common connect & disconnect methods names
Rename "kernel_init_connect/disconnect" methods
and create equivalents in SPDK Initiator class.

This allows to skip conditional checks in main
section of the script.

Change-Id: I23f620670d628c602dfdc1f24b2ddf493993ae12
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14739
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
b0760bd3d9 scripts/nvmf_perf: do not discover subsystems with nvmecli
Information about Target subsystems (port / nqn name /
ip address) is already saved at the time of their creation.
Avoid spawning nvme-cli processes and instead just filter
matching subsystems based on available information.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I24d16315d67e5eaade48790b7fae45664110afcb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14737
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
4f74eb12a3 scripts/nvmf_perf: use tuple to store subsystem information
Use tuple instead of list as this information is
supposed to be immutable later on.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Id9806202d06368f09ede0c0d903924b101d3b86c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14736
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
80e92723f5 scripts/nvmf_perf: use standard library logger
Change-Id: I1fa41339ff76fbba97071b62f311699ad870e199
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14735
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
3db789597d scripts/nvmf_perf: use sudo to print sysctl.conf
Permissions on /etc/systctl.conf file might be
for owner-only (root). Use sudo to make sure we
can read the file.

Change-Id: I98d77ac87f196aac82000cb42553285f22f8cdcd
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14390
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:15 +00:00
paul luse
17538bdc67 add (c) and SPDX header to python files as needed
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc)

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line.  Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I6cd3f18d1b469d5ef249d26ddb2923ca6b970bd4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15208
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-10 08:28:53 +00:00
paul luse
a6dbe3721e update Intel copyright notices
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc).  Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line.  Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
2022-11-10 08:28:53 +00:00
Karol Latecki
9a028f88ac scripts/perf_nvmf: fix Kernel Initiator regex issue
Get full regex match using ".group(0)" method.
Otherwise a non-iterable MatchObject is passed
to get_route_nic_numa() which causes an exception.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Ief8b08387e4233d42b5ac32bd4d7394596aabd78
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14649
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-23 10:03:33 +00:00
Karol Latecki
47b8795268 scripts/nvmf_perf: match CPUs/NICs NUMA on initiator side
For each generated fio job section "filename=" field try
to figure out what's the most dominant NUMA node in use
and use it for fio "numa_cpu_nodes" and "numa_mem_policy"
settings.

Change-Id: I96fda436dda8d4009dc191bf7c0c836e5b53a1a3
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13514
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-09-22 15:52:52 +00:00
Michal Berger
fc1c9aafe6 perf/vhost: Allow to only generate the fio configuration
This use case was missed during the initial refactor introduced via
008f8a8617. CI depends on it so allow to specify the target location
of the fio configuration and allow to skip call to the main perf
wrapper.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ie41f9db40769980047f42475ce74e61c374c364c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14590
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-21 22:51:24 +00:00
Karol Latecki
92b643b0b2 scripts/nvmf_perf: get systems NIC numa node function
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I254bd86336a5effaa7c78559b2b654809a673712
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13513
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-21 14:01:06 +00:00
Karol Latecki
23e767c4a5 scripts/nvmf_perf: add workload offset option
Use fio "offset", "offset_increment" and "size" to
split target file into chunks, so that each fio
job clone (specified by "numjobs") gets it's own
part of the disk to work with.

This is especially helpful in case of sequential
workloads run with numjobs > 1 and helps to keep
the workload to being as close to sequential as
possible.

Change-Id: I1ca88ae56136d22cf396f464d78b05eff5c07a2a
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13515
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-21 14:01:06 +00:00
Ben Walker
712e8cb7ef accel: Refer to plugins as 'modules' instead of 'engines'
This is consistent with the use of terms in other parts of SPDK and fits
with the code living under module/

Change-Id: If182f7cf2d160d57443a1b5f24e0065f191b59b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-21 08:17:48 +00:00
Michal Berger
04dd028292 perf/nvmf: Gather power statistics on the target side
Use collect-bmc-pm to gather power usage stats during fio workloads.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I20d1f3009c8a32c3927ad2cf2f932a8e97d3e7c5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14164
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-20 10:17:04 +00:00
Michal Berger
b43a0112dd scripts/perf: Add script for collecting system power statistics
This script is a wrapper around ipmitool which attempts to extract
power statistics from the BMC. It's mainly based around DCMI spec
to make sure it's common for most of the platforms that we use
in our CI. If given BMC does not conform to DCMI, there's a fallback
in place which attempts to find proper sensor in the SDR which may
report overall power usage of the underlying system.

By default, current ("NOW") reading is extracted in context of DCMI.
Different intervals may be defined, and when matched with supported
averaging time periods, special version of the command will be used
to extract additional avg, min, max information for given time (e.g.
current, max, min, avg readings from last 1m). For the SDR fallback,
only current reading can be extracted.

When terminated, script dumps all readings + their averaged value
into separate .txt files.

Change-Id: Ifc8c1defe862254864d69de7edf5cce635aff258
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14144
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-20 10:17:04 +00:00
Michal Berger
7a4cb7bc1c perf/vhost: Allow to pass multiple FIO configs to the benchmark script
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I96df5f21f04b9196083f91783f8dd9cc4c6920d1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13688
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-15 20:15:57 +00:00
Michal Berger
971451a84a perf/vhost: Allow to add extra fio config through $fio_extra_conf
$fio_extra_conf can either point at an existing file or be an
attached stdin the fio_conf() should read from.

Signed-off-by: Michal Berger <michallinuxstuff@gmail.com>
Change-Id: If902262b0100fb056491928b3f667ab15e6e0024
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12429
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-15 20:15:57 +00:00
Michal Berger
0ee37d256a perf/vhost: Add io_uring related options to fio configuration
https://trello.com/c/EFPM29gq/399-use-iouring-in-vhost-performance-tests

Signed-off-by: Michal Berger <michallinuxstuff@gmail.com>
Change-Id: I9bf8d9a121ffc1f52273fbe863bf9918e7399e59
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12428
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-15 20:15:57 +00:00
Michal Berger
008f8a8617 perf/vhost: Replace run_vhost_test.py with a simple bash wrapper
The wrapper glues execution of the perf/vhost tests in as simple form
as possible. The arguments are meant to be passed either via env or
the cmdline in a keyworded argument fashion. All defaults are
embedded inside the configuration which is built by the wrapper
itself.

https://trello.com/c/HSoRtQkO/401-deprecate-vhost-performance-python-helper-script

Signed-off-by: Michal Berger <michallinuxstuff@gmail.com>
Change-Id: Iad94bfdd65cb8871b819351ddae65e647621fb30
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12400
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-15 20:15:57 +00:00
Karol Latecki
7807960d46 scripts/nvmf_perf: load common drivers before test
Make sure needed drivers are loaded before we start
the test.

Change-Id: I93e16a2266f99e7bfa6fffaede460c6230659872
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13616
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-09-09 12:59:13 +00:00