Commit Graph

213 Commits

Author SHA1 Message Date
Michal Berger
c1e9ed6d4c perf/vhost: Auto generate VM cpu and disk map configs
The configuration is generated based on existing host's NUMA
topology (as seen via sysfs) instead of a total number of cpus as
it was done before. New logic attempts to load balance VMs and
their cpus based on nvme drives' NUMA location. If there is no
enough cpus left under the target node, all remaining nodes are
checked. For the sake of the performance, cpus are not mixed
between different numa nodes.

Disk map is created by mapping VMs to existing nvme drives
based on their NUMA location. Extra VMs are assigned in bus
order of the nvme drives.

SPDK cpus are split by matching the VM-to-nvme NUMA ratio.
Static list can be defined as well to override this behavior.

https://trello.com/c/HSoRtQkO/401-deprecate-vhost-performance-python-helper-script

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ia63c6f9a472a685d252efd110eaba7b114a87d2c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12401
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
2023-01-19 21:27:59 +00:00
Karol Latecki
cf0cf4790b scripts/nvmf_perf: add iobuf options to performance test
Recent changes done to iobuf and accel framework
require us to adjust iobuf pool sizes when running
tests with high number of NVMe-oF subsystems.

Change-Id: I72ce6e71446bea0af44bdad6258e7d7f5d765656
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16288
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2023-01-19 13:52:27 +00:00
Karol Latecki
83c8db4919 scripts/nvmf_perf: allow sharing initiator CPU cores
Each fio job section will get unique CPU cores
assigned based on the number of numjobs value.
In case of high number of fio jobs or high numjobs
parameter value we might eventually run out of
CPU cores to assign. Allow regenerating list
of available CPU cores. This allows to run the
test, but as a result CPU cores might be shared
between fio threads.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I69b1713e640d9343d68b26317f4caccdcec3cddd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16240
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2023-01-13 08:54:59 +00:00
Karol Latecki
cdab715b28 scripts/nvmf_perf: add irq_settings option
Allow user to specify the way IRQ alignment for
network interfaces used in test should be done.

Change-Id: Ib835d2ac2bc0c7b79474e617de32b96a483e436b
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15737
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-01-13 08:54:59 +00:00
Karol Latecki
7be3200f55 scripts/nvmf_perf: don't build unit tests on initiator side
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I44763a3fef9e05042623749b6fbc387a485becf1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15736
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2023-01-13 08:54:59 +00:00
Karol Latecki
70066e0c38 scripts/nvmf_perf: enable LTO on initiator side
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Icc62a586f59448242087e3d78e73e5fb76022a7d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15735
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2023-01-13 08:54:59 +00:00
Karol Latecki
87b94e3b1d scripts/nvmf_perf: manually pick fio allowed cpus
Fio "numa_cpu_nodes" parameter (which tells fio
which NUMA node to use to pick cpus for the job)
seemed promising, but manually adjusting
"cpus_allowed" seems to have better results.

Change-Id: I113566ad0dd1d32fb5a1e989725b257e3ccff4c0
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14842
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-01-13 08:54:59 +00:00
Michal Berger
3f912cf0e9 misc: Fix spelling mistakes
Found with misspell-fixer.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If062df0189d92e4fb2da3f055fb981909780dc04
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15207
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-09 08:16:18 +00:00
Kamil Godzwon
2be0335489 scripts/run_vhost_test: fix shellcheck SC2294 warning
The warning appeared during Fedora 37 VM image testing

Signed-off-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I2ceff22d08614fec0c27b9931d71753f57cf99bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15794
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-12-08 12:56:18 +00:00
Karol Latecki
de7c369173 scripts/nvmf_perf: stop checking self.mode in parent class
Make sure all attributes are initialized in both classes.
If something is not supported or works a bit differently -
move it to children's class method.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I963ef1e9fe4288992a15f65671c1b85e8e7f7fd6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14882
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
1e629c3219 scripts/nvmf_perf: allow dashes in instance names
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I4183549cc55b437a435fb024c4a9d234c456ccab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15041
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
894360fbf2 scripts/nvmf_perf: make Initiator num_cores actually optional
"num_cores" is described as optional in readme file.
For SPDK Initiator class it was required instead of
optional, and Kernel Initiator did not even take this
parameter into consideration.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Id6874ee8614aefdd9ee4a55dd200a6deab504b7a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14881
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
da55cb8756 scripts/nvmf_perf: CPU and power utilization to csv output
Find matching SAR and PM script output files, parse
them and put calculated average from accross all
workload iterations into final CSV with aggregated
results.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I9d3c445941e54eefbe93dd47d152d2ede22c9560
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15467
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
a6f6454073 scripts/nvmf_perf: fix enable_sar option
Wrong key was used to assign value from json config.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I3814b385824b435ae39cdf0a47cca42735873e86
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15757
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-02 10:40:21 +00:00
Karol Latecki
61692e8150 scripts/nvmf_perf: use common prefix for all target measurements
This should make it easier later to group matching
result files when parsing results.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I8d3c5641c8b71b6027e1f200ce53426a46e5234d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15466
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
9b3f5e98d5 scripts/nvmf_perf: enable DPDK memory dump by default
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I65e8299cb887edd23c9d8b29535d596c419bc63d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15381
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
babef5b127 scripts/nvmf_perf: gather additional metrics for all runs
Run additional measurements (PM, SAR, PCM, BWM-NG, DPDK mem)
for each configured test run.
For example: until now if fio parameter "retry" was set to 3
fio would run the workload 3 times, but additional measurements
would only be done for the first run.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I5a8aaa8eeb28f2a24f47a41650f9e2bd14a298dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15380
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
ecbb99e4d7 scripts/nvmf_perf: enable PM by default
Enable PM measurements by default.
Delay time and sample count variables are now taken
from fio ramp_time and run_time parameters, while
sample interval is hardcoded to 1 second.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Ibc08c690a8c8eea7591a5553b2478283c7574eec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15520
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
d9027e8140 scripts/nvmf_perf: group restore functions
Group and move all restore_* functions to classes
stop method.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I92cde5865a28343dbace702f54e4e06e4c2ba6cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14855
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
3d9b062852 scripts/nvmf_perf: enable PCM measurements by default
Enable PCM cpu, power and memory measurements as
default and make them run during fio workload.

Change-Id: I4d42cf9df8385bfaa3b5e8fac59283ef8be0e98b
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15341
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
b9403ae58b scripts/nvmf_perf: enable BW monitoring by default
bwm-ng interface bandwidth monitoring is now enabled
by default. Samples are captured  every second during
fio run_time duration.

Change-Id: I868f01d23e648e855161e6711236a68dcff8ff5a
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15340
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
90b95e2813 scripts/nvmf_perf: enable SAR by default
Enable SAR CPU measurements by default.
Delay time and sample count variables are now taken
from fio ramp_time and run_time parameters, while
sample interval is hardcoded to 1 second.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Icf2866820250dd6bbc5619b2777f6fe139f2a588
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15339
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
2e449d4166 scripts/nvmf_perf: group measurement options
Groupt variables used for enabling addtional
measurements together for better readibility.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: If61de427294fea6ae9a2d5c53f74b6c9e5aa3e6a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15338
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
f5d6d59482 scripts/nvmf_perf: add allowlist and blocklist
Use "allowlist" and "blocklist" options in JSON
configuration file to specify which NVMe drives
are OK to use when setting up Target side. If no
list is specified the user can continue with the
test and use all available NVMe drives by passing
"-f" option when running the script.

Previously the default behaviour was to use all
NVMe drives available in the system for test, which
could cause data loss on those drives.

This is a failsafe primarily aimed at KernelTarget
class, as SPDKTarget is already safer to use because
of existing checks in "setup.sh" script which must
be run prior to executing NVMe-oF performance test
script.

Change-Id: I5ff93672a92cb09f2aef8355542ad197d96e14e1
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14854
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
241d9c0a8a scripts/nvmf_perf: fix logging issue
"log_print" was removed in previous commit and
switched to Python's logging module instead.
This line was probably a result of a faulty rebase.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I6bbb2c37a947d0169689a014b2e110884bb70fdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15519
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-11-29 11:35:52 +00:00
Michal Berger
588dfe314b Add SPDX header to various files
They were missed by the initial set of patches which introduced this
header as a mandatory one across different types of files.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I3f9b37d41298c843e1648e72fe8593768ccd37e0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15423
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-11-29 08:27:51 +00:00
paul luse
eb53c23236 add (c) and SPDX header to bash files as needed
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc)

For intel copyrights added, --follow and -C95% were used.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I2ef86976095b88a9bf5b1003e59f3943cd6bbe4c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15209
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-29 08:27:51 +00:00
Karol Latecki
f67c74a2cf scripts/nvmf_perf: import missing subprocess exception
CalledProcessError was removed from common.py and
not moved to run_nvmf.py in:
https://review.spdk.io/gerrit/c/spdk/spdk/+/14740

Change-Id: Ib35c60aa779711f40c6fda990c721491ae000b43
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15543
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-11-24 10:10:16 +00:00
Michal Berger
c730f1040e perf/nvmf: Include CPU power measurements in the results
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I4bdfde28d4ee7b4987bda5f8152ed408c9f77516
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15183
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
6fb5eae644 perf/pm: Collect power statistics per CPU socket
This uses RAPL's powercap interface under sysfs. The alternative is
to use MSRs directly, but with this we don't have to bother about
different cpu models, etc. as kernel does that for us here.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I91ed5d67edf2669b9d7b271bbc02ecc61a6a3ea2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15182
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
daeadb177a perf/pm: Add number of all readings to the all_* log
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Id9f249926cf8a1ba3648e7a732b2a368dfb3d9e8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14655
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
f83a2d3aba perf/pm: Allow for fractional interval
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If5ddb9d7d14f3d7acc93490deade5e5524afe3b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14629
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
eb0159cc4e perf/nvmf: Align power measurement with other tools
Run collect-bmc-pm as a thread with execution time limited by count
and interval. This aligns it with the way how other tools - sar,
pcm - are used during the test.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I48fa4ea1be69f9dcae71bf01c3214102ab329bac
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14628
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Karol Latecki
ede1049911 scripts/nvmf_perf: move get_nvme_devices to SPDK Target
These functions are used only by SPDKTarget class, so
move them out of common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I489443f76fbdf4cb3def19a557f4a953a098bbce
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14853
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
0b995d4aba scripts/nvmf_perf: move get_nvme_devices to Kernel Target
This function is used only by this class, remove it
from common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I6cb01e68f0005ddc1604ff4014c6b346da3a970e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14852
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
3a359b7923 scripts/nvmf_perf: move results parsing from Target class
Results parsing is not Target specific. Result
files get copied from initiator the system where
the script is run. In future we might want to
add a feature to work with remote Targets which
would make the parse method fail.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I215391e0ff7a3ac0881f5a092513dd087fbd2e2c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14845
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
93e056ba6c scripts/nvmf_perf: clean up common.py
Remove unused imports and unused function.
Move nvmet_command function to Kernel Target
as it's specific only to this class.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I30699ca1d8541ff2f57ea609e5caf0304feb4282
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14740
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
4e07343fea scripts/nvmf_perf: common connect & disconnect methods names
Rename "kernel_init_connect/disconnect" methods
and create equivalents in SPDK Initiator class.

This allows to skip conditional checks in main
section of the script.

Change-Id: I23f620670d628c602dfdc1f24b2ddf493993ae12
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14739
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
b0760bd3d9 scripts/nvmf_perf: do not discover subsystems with nvmecli
Information about Target subsystems (port / nqn name /
ip address) is already saved at the time of their creation.
Avoid spawning nvme-cli processes and instead just filter
matching subsystems based on available information.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I24d16315d67e5eaade48790b7fae45664110afcb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14737
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
4f74eb12a3 scripts/nvmf_perf: use tuple to store subsystem information
Use tuple instead of list as this information is
supposed to be immutable later on.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Id9806202d06368f09ede0c0d903924b101d3b86c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14736
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
80e92723f5 scripts/nvmf_perf: use standard library logger
Change-Id: I1fa41339ff76fbba97071b62f311699ad870e199
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14735
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
3db789597d scripts/nvmf_perf: use sudo to print sysctl.conf
Permissions on /etc/systctl.conf file might be
for owner-only (root). Use sudo to make sure we
can read the file.

Change-Id: I98d77ac87f196aac82000cb42553285f22f8cdcd
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14390
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:15 +00:00
paul luse
17538bdc67 add (c) and SPDX header to python files as needed
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc)

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line.  Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I6cd3f18d1b469d5ef249d26ddb2923ca6b970bd4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15208
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-10 08:28:53 +00:00
paul luse
a6dbe3721e update Intel copyright notices
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc).  Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line.  Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
2022-11-10 08:28:53 +00:00
Karol Latecki
9a028f88ac scripts/perf_nvmf: fix Kernel Initiator regex issue
Get full regex match using ".group(0)" method.
Otherwise a non-iterable MatchObject is passed
to get_route_nic_numa() which causes an exception.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Ief8b08387e4233d42b5ac32bd4d7394596aabd78
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14649
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-23 10:03:33 +00:00
Karol Latecki
47b8795268 scripts/nvmf_perf: match CPUs/NICs NUMA on initiator side
For each generated fio job section "filename=" field try
to figure out what's the most dominant NUMA node in use
and use it for fio "numa_cpu_nodes" and "numa_mem_policy"
settings.

Change-Id: I96fda436dda8d4009dc191bf7c0c836e5b53a1a3
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13514
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-09-22 15:52:52 +00:00
Michal Berger
fc1c9aafe6 perf/vhost: Allow to only generate the fio configuration
This use case was missed during the initial refactor introduced via
008f8a8617. CI depends on it so allow to specify the target location
of the fio configuration and allow to skip call to the main perf
wrapper.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ie41f9db40769980047f42475ce74e61c374c364c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14590
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-21 22:51:24 +00:00
Karol Latecki
92b643b0b2 scripts/nvmf_perf: get systems NIC numa node function
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I254bd86336a5effaa7c78559b2b654809a673712
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13513
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-21 14:01:06 +00:00
Karol Latecki
23e767c4a5 scripts/nvmf_perf: add workload offset option
Use fio "offset", "offset_increment" and "size" to
split target file into chunks, so that each fio
job clone (specified by "numjobs") gets it's own
part of the disk to work with.

This is especially helpful in case of sequential
workloads run with numjobs > 1 and helps to keep
the workload to being as close to sequential as
possible.

Change-Id: I1ca88ae56136d22cf396f464d78b05eff5c07a2a
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13515
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-09-21 14:01:06 +00:00
Ben Walker
712e8cb7ef accel: Refer to plugins as 'modules' instead of 'engines'
This is consistent with the use of terms in other parts of SPDK and fits
with the code living under module/

Change-Id: If182f7cf2d160d57443a1b5f24e0065f191b59b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-21 08:17:48 +00:00