Commit Graph

1883 Commits

Author SHA1 Message Date
Tomasz Zawadzki
44290e13b4 env_dpdk: add script for comparing DPDK PCI API
Starting with DPDK 22.11, PCI API are no longer public.
In order to implement out-of-tree driver like
SPDK NVMe driver that is compatible with DPDK,
a copy of PCI API headers are required in SPDK.

check_dpdk_pci_api.sh script is intended simplify
the maintanance of the compatibility between
SPDK copies of the headers and multiple DPDK versions.

Please see the included README

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ief028c13564441560425761e7802c6cf07460876
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15857
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-12-16 09:27:11 +00:00
Konrad Sztyber
55b047a776 module/iobuf: introduce iobuf module
This gives us a place to initialize iobuf pools, specify subsystem
dependencies, and execute RPCs to configure the sizes of the pools.

We allow users to configure the size of the pools either through the
options in spdk_bdev_opts or through the new RPC, iobuf_set_options.
The second option has higher priority, so it will override the options
set by the bdev layer.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I7c45ace04bc71c8343658f98248fc7babcf24e5d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15765
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-12-16 09:06:07 +00:00
Michal Berger
4f857ce593 scripts/setup: Stop checking for broken uio_pci_generic driver
This check was put in place as a workaround for some specific kernel
builds which were mainly shipped under centos8. Since we already
dropped centos8 in the CI and replaced it with rocky8 (where kernel
is actually stable) there's no value in maintaining that anymore.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Idd39b5bae888a991d7d628a19bc18acac6570f81
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15819
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-12-14 08:03:17 +00:00
Michal Berger
eab0c6649d test/nvmf: Make sure ctrl is fully released
There may be some timing issue where attempt to rebind the ctrl from
vfio_pci driver back to nvme, right after issuing the
bdev_nvme_detach_controller() call, fails as the vfio_pci might have
not fully released the device yet.

To mitigate, simply kill the application (as it's not needed anymore
at at point) before starting the kernel_target test - this should give
enough time for the device to be properly released.

As a precaution, make setup.sh to retry the probe attempt in case it
fails.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ifc4f4c18a90605154bf33b078575c8b41129f1f3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15767
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-12-14 08:03:17 +00:00
Michal Berger
3f912cf0e9 misc: Fix spelling mistakes
Found with misspell-fixer.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If062df0189d92e4fb2da3f055fb981909780dc04
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15207
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-09 08:16:18 +00:00
Michal Berger
1f59abaebf scripts/setup: Use driver_override to bind devices
Using new_id attribute is global in scope, meaning that depending
on the kernel's setup seen prior running setup.sh, single write
to it may re-bind ALL matching devices. This doesn't play well
with our PCI_{ALLOWED,BLOCKED} options as they can't be enforced
in such a case. Consider the following example:

  > modprobe -r nvme # all nvme ctrls are detached from the kernel
  > echo 0xdead 0xbeef >/sys/bus/pci/drivers/uio_pci_generic/new_id

  # setup.sh-wise

  > modprobe -r nvme
  > PCI_BLOCKED=some:dead:beef.bdf setup.sh
  # PCI_BLOCKED device still ends up bound to userspace driver.

After this single write, ALL matching devices will end up bound to
uio_pci_generic. To avoid this, we should override preferred driver
on per-bdf basis.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Ic4613e33321303b92b47ce3f4d7e1f29ecca3036
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13813
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-12-08 14:16:40 +00:00
Alexey Marchuk
0ab4de566b pkgdep/debian: Install python3-dev
This package is required to install grpcio:

./scripts/pkgdep.sh --developer-tools
...
  Could not find <Python.h>. This could mean the following:
    * You're on Ubuntu and haven't run `apt-get install python3-dev`.
    * You're on RHEL/Fedora and haven't run `yum install python3-devel` or
      `dnf install python3-devel` (make sure you also have redhat-rpm-config
      installed)
    * You're on Mac OS X and the usual Python framework was somehow corrupted
      (check your environment variables or try re-installing?)
    * You're on Windows and your Python installation was somehow corrupted
      (check your environment variables or try re-installing?)

  ----------------------------------------
  ERROR: Failed building wheel for grpcio
Failed to build grpcio

Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: I1a9d6154de9ab86ba8062b53ba56002e226daa94
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15751
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: <qun.wan@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-12-08 14:15:52 +00:00
Michal Berger
5121b9a2fd scripts/qat_setup: Remove VFs' pci id from the target bind driver
Do this before the attempt to bind these devices is made. This is to
make sure script works on subsequent runs.

Scripts itself executes the qat_service which disables potential VFs
of the target PF. In case VFs were previously bound to one of the
uio drivers (via new_id) the next write to that attr will fail not
rebinding the devices again. To avoid that, remove the VFs' id first
and then attempt to bind them to the uio driver.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I4818a2e3e663a4136084ce4bbb85ccfcfde1ae42
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15427
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-12-08 12:56:56 +00:00
Kamil Godzwon
2be0335489 scripts/run_vhost_test: fix shellcheck SC2294 warning
The warning appeared during Fedora 37 VM image testing

Signed-off-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I2ceff22d08614fec0c27b9931d71753f57cf99bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15794
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-12-08 12:56:18 +00:00
Kamil Godzwon
bdc222cc27 vagrant/create_vbox: add fedora37
Signed-off-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Change-Id: Ibaff2db31f17e91a380922eb74dbcb3cc5578d43
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15790
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-12-08 12:56:18 +00:00
paul luse
19e2dc3853 configure: rename --with-reduce --with-vbdev-compress
This is in prep for adding a new compressDev accel_fw
module that will contain all of the DPDK compressDev specifics
on it, the vbdev will make calls to the accel_fw instead.

As the accel_fw has SW based compression, we want the configure
option to apply to building the vbdev module but not the accel_sw
software implementation or the upcoming compressdev module.

Renamed to "compress" as reduce is a term specific to the vbdev
implementation of the compression to be provided by the accel_fw
and thus the same reason why we leave the test flag called REDUCE
because it's controlling tests for the reduce library as well as
the vbdev module that is using reduce.  The flag does not apply
to the SW implementation of compression.

This does not affect upcoming accel_fw compressdev module, that
will have its own configure option.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: If8ed3e48e1e3dabcaad1cd161289e78122cd9d58
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15179
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-12-08 12:55:27 +00:00
Shuhei Matsumoto
d7ad7bca3c bdev: Add mode to bdev_reset_iostat RPC to reset only max/min fields
Both max and min should be reset periodically. We can use the queue
depth sampling poller to reset these but the queue depth sampling poller
is optional. We extend the bdev_reset_iostat RPC to support mode to
reset all or only max/min fields.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I9ce54892f6e808f6a82754b6930092f3a16d51ff
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15444
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-12-08 12:54:23 +00:00
Shuhei Matsumoto
cf4e8664bb bdev: Add bdev_reset_iostat RPC
Add a helper function bdev_reset_device_stat() to reset I/O statistics.
This funciton is used for the bdev_reset_iostat RPC.

We do not have any plan to use bdev_reset_device_stat() outside
lib/bdev. Hence, we do not add this as a public API.

Then, add a new RPC bdev_reset_iostat to reset I/O statistics of a
single bdev or all bdevs.

Resetting I/O statistics affects all consumers. Add a note to CHANGELOG
and doc/jsonrpc.md.

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I97af09107b5c3ad1f9c19bf3cbf027457c4fbae7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15350
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-08 12:54:23 +00:00
Artur Paszkiewicz
ec6d94b674 module/raid: show raid_bdev details in bdev_raid_get_bdevs rpc
Change-Id: I30a78b63c5ecc988e2a267d58716ea79f849789a
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14508
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
2022-12-07 08:43:52 +00:00
Xinrui Mao
cd4ac9c792 lib/trace: add trace_get_info RPC
Add rpc method trace_get_info to show name of shared memory file,
list of the available trace point groups and mask of the available trace points for each group.

Fixes #2747

Signed-off-by: Xinrui Mao <xinrui.mao@intel.com>
Change-Id: I2098283bed454dc46644fd2ca1b9568ab2aea81b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15426
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
2022-12-05 09:50:38 +00:00
Karol Latecki
de7c369173 scripts/nvmf_perf: stop checking self.mode in parent class
Make sure all attributes are initialized in both classes.
If something is not supported or works a bit differently -
move it to children's class method.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I963ef1e9fe4288992a15f65671c1b85e8e7f7fd6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14882
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
1e629c3219 scripts/nvmf_perf: allow dashes in instance names
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I4183549cc55b437a435fb024c4a9d234c456ccab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15041
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
894360fbf2 scripts/nvmf_perf: make Initiator num_cores actually optional
"num_cores" is described as optional in readme file.
For SPDK Initiator class it was required instead of
optional, and Kernel Initiator did not even take this
parameter into consideration.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Id6874ee8614aefdd9ee4a55dd200a6deab504b7a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14881
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
da55cb8756 scripts/nvmf_perf: CPU and power utilization to csv output
Find matching SAR and PM script output files, parse
them and put calculated average from accross all
workload iterations into final CSV with aggregated
results.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I9d3c445941e54eefbe93dd47d152d2ede22c9560
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15467
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-12-05 09:42:27 +00:00
Karol Latecki
a6f6454073 scripts/nvmf_perf: fix enable_sar option
Wrong key was used to assign value from json config.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I3814b385824b435ae39cdf0a47cca42735873e86
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15757
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Jaroslaw Chachulski <jaroslawx.chachulski@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-02 10:40:21 +00:00
Karol Latecki
61692e8150 scripts/nvmf_perf: use common prefix for all target measurements
This should make it easier later to group matching
result files when parsing results.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I8d3c5641c8b71b6027e1f200ce53426a46e5234d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15466
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
9b3f5e98d5 scripts/nvmf_perf: enable DPDK memory dump by default
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I65e8299cb887edd23c9d8b29535d596c419bc63d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15381
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
babef5b127 scripts/nvmf_perf: gather additional metrics for all runs
Run additional measurements (PM, SAR, PCM, BWM-NG, DPDK mem)
for each configured test run.
For example: until now if fio parameter "retry" was set to 3
fio would run the workload 3 times, but additional measurements
would only be done for the first run.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I5a8aaa8eeb28f2a24f47a41650f9e2bd14a298dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15380
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
ecbb99e4d7 scripts/nvmf_perf: enable PM by default
Enable PM measurements by default.
Delay time and sample count variables are now taken
from fio ramp_time and run_time parameters, while
sample interval is hardcoded to 1 second.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Ibc08c690a8c8eea7591a5553b2478283c7574eec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15520
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
d9027e8140 scripts/nvmf_perf: group restore functions
Group and move all restore_* functions to classes
stop method.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I92cde5865a28343dbace702f54e4e06e4c2ba6cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14855
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-01 10:59:13 +00:00
Karol Latecki
3d9b062852 scripts/nvmf_perf: enable PCM measurements by default
Enable PCM cpu, power and memory measurements as
default and make them run during fio workload.

Change-Id: I4d42cf9df8385bfaa3b5e8fac59283ef8be0e98b
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15341
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
b9403ae58b scripts/nvmf_perf: enable BW monitoring by default
bwm-ng interface bandwidth monitoring is now enabled
by default. Samples are captured  every second during
fio run_time duration.

Change-Id: I868f01d23e648e855161e6711236a68dcff8ff5a
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15340
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
90b95e2813 scripts/nvmf_perf: enable SAR by default
Enable SAR CPU measurements by default.
Delay time and sample count variables are now taken
from fio ramp_time and run_time parameters, while
sample interval is hardcoded to 1 second.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Icf2866820250dd6bbc5619b2777f6fe139f2a588
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15339
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
2e449d4166 scripts/nvmf_perf: group measurement options
Groupt variables used for enabling addtional
measurements together for better readibility.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: If61de427294fea6ae9a2d5c53f74b6c9e5aa3e6a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15338
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
f5d6d59482 scripts/nvmf_perf: add allowlist and blocklist
Use "allowlist" and "blocklist" options in JSON
configuration file to specify which NVMe drives
are OK to use when setting up Target side. If no
list is specified the user can continue with the
test and use all available NVMe drives by passing
"-f" option when running the script.

Previously the default behaviour was to use all
NVMe drives available in the system for test, which
could cause data loss on those drives.

This is a failsafe primarily aimed at KernelTarget
class, as SPDKTarget is already safer to use because
of existing checks in "setup.sh" script which must
be run prior to executing NVMe-oF performance test
script.

Change-Id: I5ff93672a92cb09f2aef8355542ad197d96e14e1
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14854
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-11-29 11:35:52 +00:00
Karol Latecki
241d9c0a8a scripts/nvmf_perf: fix logging issue
"log_print" was removed in previous commit and
switched to Python's logging module instead.
This line was probably a result of a faulty rebase.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I6bbb2c37a947d0169689a014b2e110884bb70fdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15519
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2022-11-29 11:35:52 +00:00
Michal Berger
c271381112 check_format: Add checker for the SPDX-license
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I0d38256afb918f1273def36b3d4915d5a1911df9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15422
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-29 08:27:51 +00:00
Michal Berger
588dfe314b Add SPDX header to various files
They were missed by the initial set of patches which introduced this
header as a mandatory one across different types of files.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I3f9b37d41298c843e1648e72fe8593768ccd37e0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15423
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2022-11-29 08:27:51 +00:00
paul luse
eb53c23236 add (c) and SPDX header to bash files as needed
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc)

For intel copyrights added, --follow and -C95% were used.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I2ef86976095b88a9bf5b1003e59f3943cd6bbe4c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15209
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-29 08:27:51 +00:00
Karol Latecki
f67c74a2cf scripts/nvmf_perf: import missing subprocess exception
CalledProcessError was removed from common.py and
not moved to run_nvmf.py in:
https://review.spdk.io/gerrit/c/spdk/spdk/+/14740

Change-Id: Ib35c60aa779711f40c6fda990c721491ae000b43
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15543
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-11-24 10:10:16 +00:00
Sebastian Brzezinka
7c79992e9c SC2295: Expansions inside ${..} need to be quoted separately
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: Ieb89027d41567f000f0f1a8c7375fde5201b06db
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15561
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-24 10:09:29 +00:00
Sebastian Brzezinka
6e7b3e4cba SC2048: Use "${array[@]}" (with quotes) to prevent whitespace problems.
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I64473e4904c82fa06d41e8a02d56070ebf966adc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15559
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-24 10:09:29 +00:00
Michal Berger
485f9f974a pkgdep/freebsd: Update name of the ceph package
Under latest 13.1 builds ceph package is now available as "ceph14"
so try to install it when the first attempt fails.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I4301f357d2c04881a1e8839c644cb000ba74e4e6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15495
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2022-11-21 12:02:48 +00:00
Konrad Sztyber
5e911c0c06 check_format: search all declarations in naming conv check
Look for all declarations instead of only the ones introduced in the
tested patch.  It makes this test pass if a patch adds definition for a
function that already has a declaration in the header file (e.g.
the implementation depends on some #ifdef).

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I735cf10d4524318009f8c4282a86e4137bf8c769
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15428
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-18 08:40:05 +00:00
Krzysztof Karas
b5bdbbb959 bdev: enable bdevs based on physical device to generate UUID
Add an option "--generate-uuids" to bdev_nvme_set_options
RPC to enable generation of UUIDs for NVMes devices that
do not provide this value themselves. The identifier is
based on a serial number of the device, so a bdev
using this NVMe will always be assigned the same UUID.

Part of enhancement from #2516.

Change-Id: I86d76274e5702d14ace89d83d1e9129573f543e2
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15151
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
2022-11-18 08:38:13 +00:00
Rafal Stefanowski
8000cedbe3 bdev/ocf: Add runtime cache bdev flush support
Introduce two RPC calls for starting flush and getting flush
status of OCF cache bdev:
- bdev_ocf_flush_start
- bdev_ocf_flush_status

Signed-off-by: Robert Baldyga <robert.baldyga@intel.com>
Signed-off-by: Rafal Stefanowski <rafal.stefanowski@intel.com>
Change-Id: I1d659da6fc51396e0d070af35372ee130c40ae8b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8961
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2022-11-18 08:38:03 +00:00
Michal Berger
c730f1040e perf/nvmf: Include CPU power measurements in the results
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I4bdfde28d4ee7b4987bda5f8152ed408c9f77516
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15183
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
6fb5eae644 perf/pm: Collect power statistics per CPU socket
This uses RAPL's powercap interface under sysfs. The alternative is
to use MSRs directly, but with this we don't have to bother about
different cpu models, etc. as kernel does that for us here.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I91ed5d67edf2669b9d7b271bbc02ecc61a6a3ea2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15182
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
daeadb177a perf/pm: Add number of all readings to the all_* log
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: Id9f249926cf8a1ba3648e7a732b2a368dfb3d9e8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14655
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
f83a2d3aba perf/pm: Allow for fractional interval
Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If5ddb9d7d14f3d7acc93490deade5e5524afe3b0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14629
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Michal Berger
eb0159cc4e perf/nvmf: Align power measurement with other tools
Run collect-bmc-pm as a thread with execution time limited by count
and interval. This aligns it with the way how other tools - sar,
pcm - are used during the test.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: I48fa4ea1be69f9dcae71bf01c3214102ab329bac
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14628
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-11-17 08:56:27 +00:00
Karol Latecki
ede1049911 scripts/nvmf_perf: move get_nvme_devices to SPDK Target
These functions are used only by SPDKTarget class, so
move them out of common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I489443f76fbdf4cb3def19a557f4a953a098bbce
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14853
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
0b995d4aba scripts/nvmf_perf: move get_nvme_devices to Kernel Target
This function is used only by this class, remove it
from common module.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I6cb01e68f0005ddc1604ff4014c6b346da3a970e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14852
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
3a359b7923 scripts/nvmf_perf: move results parsing from Target class
Results parsing is not Target specific. Result
files get copied from initiator the system where
the script is run. In future we might want to
add a feature to work with remote Targets which
would make the parse method fail.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I215391e0ff7a3ac0881f5a092513dd087fbd2e2c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14845
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2022-11-17 08:56:15 +00:00
Karol Latecki
93e056ba6c scripts/nvmf_perf: clean up common.py
Remove unused imports and unused function.
Move nvmet_command function to Kernel Target
as it's specific only to this class.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I30699ca1d8541ff2f57ea609e5caf0304feb4282
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14740
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
2022-11-17 08:56:15 +00:00