In prep for upcoming addition of IAA.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I3044f1e07dbfc65a6a55a7d68bbaff6a43e909d0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12761
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We try to modprobe msr so that DPDK can more easily figure
out TSC on x86 systems. But that just results in a
"Module msr not found" message on non-x86 systems. setup.sh
still works, but the error message makes it appear
otherwise. So just redirect the error message to reduce
this confusion.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I71074c7836519b003933551ce51a193070b1bfe6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12671
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
This is done to make sure we don't miss more complex setups where
target devices are not mounted but still hold some valid data that
shouldn't be touched in any way.
Also, adjust function names so they clearly indicate what is being
checked.
Signed-off-by: Michal Berger <michallinuxstuff@gmail.com>
Change-Id: Ibb0f1f21de68009a2f8f1faf4595a07ae527da35
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11111
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
If a VMD device present, add message to communicate to the user to use next steps.
Allow for DRIVER_OVERRIDE="none" to just unbind the driver without binding it to any other.
Fixes#2423
Signed-off-by: Samir Raval <samir.raval@intel.com>
Change-Id: Ifef6ed50dd619ce7629eabf458edd54e6bb22fa4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12035
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In case collect_driver() returns immediately due to missing modalias
attr, an empty $driver will be passed to check_for_driver() causing
grep inside to fail. Further false-positive then leads to call to
linux_bind_driver() as it's not able to locate proper paths for
given driver.
To avoid this, make sure collect_driver() always attempts to fallback
to a driver setup.sh supports for a target device.
Signed-off-by: Michal Berger <michallinuxstuff@gmail.com>
Change-Id: Id5631a731910f60b63c6afb3a412575bb69d784a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11747
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
SC2030: Modification of var is local (to subshell caused by pipeline)
This directive is raised since the for loop (or rather its contents)
is piped through to another process. In this context it's not harmful
since we are interested only in sending the stdout but since we operate
on variables that happen to be local to the process that handles the
loop shellcheck sees that as a potential issue. To make it happy
remove the pipe and sort pci addresses prior running the loop.
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I835c5ac1da9012129c5d01d62880307f70caab1b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8501
Community-CI: Mellanox Build Bot
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Current `test -f` never succeeds, because `/dev/cpu/0/msr` is a
character device file.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iaa28f04866ab24140668ad43351a37f0ccccef65
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7166
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Also, for consistency, use check_for_driver() while checking if
igb_uio is loaded too.
Change-Id: I00302b3cab169c77032fc0cef0ea384e9e5eb5be
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5303
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
DPDK 20.11 moved the kernel modules to separate
dpdk-kmod repository. It has to be built separate
from DPDK.
If needed for testing vm_setup.sh script now contains option
to build this driver.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I98a5eb956eb0cc60ec402d88fcdbd66d4854f19a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6033
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Michal Berger <michalx.berger@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This is done in order to make sure IOMMU extensions, which DPDK is
looking for, are enabled.
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Ia9ff8e5329d14a5eb60721f08ff7063cfb1d6eb7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5259
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Mike Gerdts
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Similarly replace PCI_BLACKLIST with PCI_BLOCKED.
Use of ALLOWED/BLOCKED matches similar changes made
in DPDK.
While here, replace use of term "blacklist" with "blocked"
in one of the nvme perf scripts. The usage there was
associated with how devices are blocked by using the
environment variables that are changed by this patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I720d99118ba5e050f436612c9fd415db44294a63
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5275
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
If DPDK was not told to unlink the hugepage backing files during the
hugepage_init, the application will keep them open while holding a
lock on each file. Check if said lock is held, if not, remove the
file.
Change-Id: I738a3d7756c335b63388a2efeb03debd734d9c9e
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5065
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Main changes:
- By default, allocate all hugepages on node0. On NUMA-aware systems,
processes will most often use default policy with local node as the
preffered one for allocations. This usually is node0, thus splitting
hugepages evenly across the nodes, in default setup, would force
allocations from a remote node, impacting overall performance (in
case hugepages on node0 run out). See 68740678e1 as a reference.
- Introduce HUGE_EVEN_ALLOC - force setup.sh to evenly distribute
hugepages across all the nodes.
- Introduce HUGEPGSZ - overrides default page size
- Introduce CLEAR_HUGE - remove all hugepages on demand before
allocation is performed.
- HUGENODE - this now can be a list of nodes to use. NRHUGE will be
set across all the nodes from the list.
Change-Id: I084829edde3c416e7fc6b7b6abe369cc0631fcd7
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5042
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This is done so user can easily filter out parts of the output which
are not relevant for additional processing (greping, seding, etc.).
Change-Id: I67eafd1fbeb723224cca6b0026e149cb7f85bfe9
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5152
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This is done in order to improve the grepability of the output. Sort
is based on the BDF address.
Hugepages statistics are intact.
Change-Id: I121e9a46bfb1550380f233c1d5c06cefb9d4e7a9
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5022
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Each nvme BDF can have multiple namespaces, hence multiple block
devices, return them all.
Change-Id: I93fe8acf5b1904f05514445eb3e970ef4254caed
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5170
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
lsblk will include all the potential holders|virtual devices given
block device may be part of. This is relevant in case a nvme|virtio
device is part of the lvm|md setup.
Change-Id: I32198ddcf71746c3cbad4a96496a1762a5c201cd
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5167
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
If they are not, there's no point to continue setup on FreeBSD.
Change-Id: I6d94712f1507c0597fbd569ff73c52f262ab9df9
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4658
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In case contigmem is loaded, but the kernel environment hasn't been
properly updated with hw.contigmem.* options, kenv will fail,
breaking the check to determine if contigmem should be unloaded.
Avoid the above scenario by determining first if given option is
actually set in kernel's environment.
Change-Id: I4118b7622fd876b28f0c07d081c583f5782f0357
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4656
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Some 4.18.x kernels are shipped with a broken uio driver due to the
following change:
git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=9421e45f5
Try to detect if we are running against a faulty uio driver and if so
fallback to igb_uio driver if present. The priority of picking up the
drivers has not changed.
Note that this commit may be deemed as not needed since from CI
perspective the https://review.spdk.io/gerrit/c/spdk/spdk/+/4342 may
be simply enough.
Change-Id: I9b12511c203c0be0e8f3f462c9c96babde52dc6e
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4343
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Recent refactors removed the behavior introduced with a6edaa9600.
Bring it back and don't try to rebind the VMD devices if they were
not explicitliy allowed in the setup.
Also, shuffle the code a bit and put verification pieces under one
block where $mode is being determined.
Change-Id: Ie2cc41e402f20147b98ab288d623ac76a4472839
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4398
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Error log as below:
06:27:53 # nvme_namespace_revert
06:27:53 # /home/storage/workspace/opal-end-to-end-reservation-autotest/spdk/scripts/setup.sh
/home/storage/workspace/opal-end-to-end-reservation-autotest/spdk/scripts/setup.sh: line 109: /sys/bus/pci/devices/0000:80:04.2/driver/remove_id: No such file or directory
home/storage/workspace/opal-end-to-end-reservation-autotest/spdk/scripts/setup.sh: line 110: /sys/bus/pci/devices/0000:80:04.2/driver/unbind: No such file or directory
06:27:54 # trap - ERR
Change-Id: I302e58143f0e165b17a9b5edd14121e6282f55a7
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4094
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michalx.berger@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Most distros ship these drivers as modules, however, some, like
clearlinux, have them compiled directly in the kernel (e.g. nvme).
In case modalias lookup fails, have a fallback prepared just in case.
Change-Id: Ib9da8ff69edbb0c4ce427aa089fa7d04b399a20b
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4047
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The previous approach didn't take into the account systems where root
partition is representend in a virtual form of /dev/root (see
clearlinux). In case the device was bound to virtio this would fool
setup.sh into thinking nothing is mounted and proceed with unbinding
the entire rootfs.
Fix this by checking the maj:min numbers of the device instead of its
name against the mountinfo list.
Change-Id: I0feb2584869f6bb72df3e9a4e619620240cfce3b
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4046
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Make sure that setup.sh waits for block devices during the tests.
This is to make sure that the underlying controllers are ready to
be talked to and minimize the amount of their flakiness.
Change-Id: Id09445de7ac7ccf4c9f679ee2b6c4bdd14c89733
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4029
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifeadc3fac3f5d09143dd13ef3d510de9cecffb8a
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4032
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
There are some devices for which nvme driver takes a long time to
finalize the unbind stage. With that in mind, each device would
add up a significant amount of time needed for setup.sh to complete.
To mitigate such a scenario, make sure the controllers are unbound
in a parallel fashion.
Examples taken from the system with 19 nvmes on board:
[root@supermicro4 spdk]# time ./scripts/setup.sh &>/dev/null
real 0m36.250s
user 0m1.024s
sys 0m1.990s
[root@supermicro4 spdk]# time ./scripts/setup.sh &>/dev/null
real 0m4.848s
user 0m0.867s
sys 0m17.605s
Also, take note that this is currently done only for the nvme
devices since other, i.e., ioatdma, seem to trigger a BUG in the
kernel when unbound in parallel. Some details here:
https://bugzilla.kernel.org/show_bug.cgi?id=209041
Change-Id: Icaeb2b2ecb306f149587bc5da73743b1519bc5d6
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3893
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This is a first patch from a series attempting to merge similar
routines used throughout setup.sh.
Change-Id: I4c71b88f7556b0e9e2f65a37b8175914022486a4
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3889
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When using vfio a warning for memlock limit is printed.
TARGET_USER is one set by caller when executing the script,
or set to user that called the script (even when using sudo).
Yet ulimit was printed for the one executing the script,
most likely root.
This patch verifies ulimit for a particular TARGET_USER.
Along with more explicit information which user the
information pertains to.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I20ad62109bc316295208e21f959ecfc263307e1c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1964
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
