Spdk/autotest.sh

296 lines
7.8 KiB
Bash
Raw Normal View History

#!/usr/bin/env bash
rootdir=$(readlink -f $(dirname $0))
# In autotest_common.sh all tests are disabled by default.
# If the configuration of tests is not provided, no tests will be carried out.
autotest: don't source the configuration file in every test script Most of our bash test scripts source autotest_common.sh to be able to use some autotest-specific functions like timing_enter(). The same test scripts allow specifying custom command line parameters without actually realizing that those parameters can be potentially picked up by autotest_common.sh as well. For example, if particular nvmf tests are run in "isolation" mode by being executed with the first param set to "iso", and there is a file named "iso" in the current dir, that file will be sourced. This could be bad. In this patch we stop sourcing or even processing $1 in autotest_common.sh. Instead, the test configuration will be sourced just once from autobuild.sh, autopackage.sh and autotest.sh. If the user wants to run particular test scripts manually, he should source an SPDK test configuration by himself - manually as well. In most cases he won't even have two, as only a few test scripts depend on SPDK_* variables. Note that we still have to setup the default values for SPDK_* variables in autotest_common.sh because some of our test scripts actually depend on them: > if [ $SPDK_TEST_RBD -eq 1 ]; then ... Because it lacks any type of quotes around SPDK_TEST_RBD, it will print the following message when that variable is unset: > /bin/bash: line 0: [: -eq: unary operator expected It doesn't trigger any error ($? == 0), but can be still a bit misleading in the script output. Change-Id: I350045d8582d66fe1ed7697d4bcbba324cb541ad Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com> Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453876 Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-09 11:22:41 +00:00
if [[ ! -f $1 ]]; then
echo "ERROR: SPDK test configuration not specified"
exit 1
fi
autotest: don't source the configuration file in every test script Most of our bash test scripts source autotest_common.sh to be able to use some autotest-specific functions like timing_enter(). The same test scripts allow specifying custom command line parameters without actually realizing that those parameters can be potentially picked up by autotest_common.sh as well. For example, if particular nvmf tests are run in "isolation" mode by being executed with the first param set to "iso", and there is a file named "iso" in the current dir, that file will be sourced. This could be bad. In this patch we stop sourcing or even processing $1 in autotest_common.sh. Instead, the test configuration will be sourced just once from autobuild.sh, autopackage.sh and autotest.sh. If the user wants to run particular test scripts manually, he should source an SPDK test configuration by himself - manually as well. In most cases he won't even have two, as only a few test scripts depend on SPDK_* variables. Note that we still have to setup the default values for SPDK_* variables in autotest_common.sh because some of our test scripts actually depend on them: > if [ $SPDK_TEST_RBD -eq 1 ]; then ... Because it lacks any type of quotes around SPDK_TEST_RBD, it will print the following message when that variable is unset: > /bin/bash: line 0: [: -eq: unary operator expected It doesn't trigger any error ($? == 0), but can be still a bit misleading in the script output. Change-Id: I350045d8582d66fe1ed7697d4bcbba324cb541ad Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com> Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453876 Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-09 11:22:41 +00:00
source "$1"
source "$rootdir/test/common/autotest_common.sh"
source "$rootdir/test/nvmf/common.sh"
if [ $EUID -ne 0 ]; then
echo "$0 must be run as root"
exit 1
fi
if [ $(uname -s) = Linux ]; then
# set core_pattern to a known value to avoid ABRT, systemd-coredump, etc.
echo "core" > /proc/sys/kernel/core_pattern
# make sure nbd (network block device) driver is loaded if it is available
# this ensures that when tests need to use nbd, it will be fully initialized
modprobe nbd || true
fi
trap "process_core; autotest_cleanup; exit 1" SIGINT SIGTERM EXIT
timing_enter autotest
create_test_list
src=$(readlink -f $(dirname $0))
out=$PWD
cd $src
./scripts/setup.sh status
freebsd_update_contigmem_mod
if hash lcov; then
# setup output dir for unittest.sh
export UT_COVERAGE=$out/ut_coverage
export LCOV_OPTS="
--rc lcov_branch_coverage=1
--rc lcov_function_coverage=1
--rc genhtml_branch_coverage=1
--rc genhtml_function_coverage=1
--rc genhtml_legend=1
--rc geninfo_all_blocks=1
"
export LCOV="lcov $LCOV_OPTS --no-external"
# Print lcov version to log
$LCOV -v
# zero out coverage data
$LCOV -q -c -i -t "Baseline" -d $src -o cov_base.info
fi
# Make sure the disks are clean (no leftover partition tables)
timing_enter cleanup
# Remove old domain socket pathname just in case
rm -f /var/tmp/spdk*.sock
# Load the kernel driver
./scripts/setup.sh reset
# Let the kernel discover any filesystems or partitions
sleep 10
if [ $(uname -s) = Linux ]; then
# OCSSD devices drivers don't support IO issues by kernel so
# detect OCSSD devices and blacklist them (unbind from any driver).
# If test scripts want to use this device it needs to do this explicitly.
#
# If some OCSSD device is bound to other driver than nvme we won't be able to
# discover if it is OCSSD or not so load the kernel driver first.
while IFS= read -r -d '' dev
do
# Send Open Channel 2.0 Geometry opcode "0xe2" - not supported by NVMe device.
if nvme admin-passthru $dev --namespace-id=1 --data-len=4096 --opcode=0xe2 --read >/dev/null; then
bdf="$(basename $(readlink -e /sys/class/nvme/${dev#/dev/}/device))"
echo "INFO: blacklisting OCSSD device: $dev ($bdf)"
PCI_BLACKLIST+=" $bdf"
OCSSD_PCI_DEVICES+=" $bdf"
fi
done < <(find /dev -maxdepth 1 -regex '/dev/nvme[0-9]+' -print0)
export OCSSD_PCI_DEVICES
# Now, bind blacklisted devices to pci-stub module. This will prevent
# automatic grabbing these devices when we add device/vendor ID to
# proper driver.
if [[ -n "$PCI_BLACKLIST" ]]; then
PCI_WHITELIST="$PCI_BLACKLIST" \
PCI_BLACKLIST="" \
DRIVER_OVERRIDE="pci-stub" \
./scripts/setup.sh
# Export our blacklist so it will take effect during next setup.sh
export PCI_BLACKLIST
fi
fi
# Delete all leftover lvols and gpt partitions
# Matches both /dev/nvmeXnY on Linux and /dev/nvmeXnsY on BSD
# Filter out nvme with partitions - the "p*" suffix
for dev in $(ls /dev/nvme*n* | grep -v p || true); do
dd if=/dev/zero of="$dev" bs=1M count=1
done
sync
timing_exit cleanup
# set up huge pages
timing_enter afterboot
./scripts/setup.sh
timing_exit afterboot
timing_enter nvmf_setup
rdma_device_init
timing_exit nvmf_setup
if [[ $SPDK_TEST_CRYPTO -eq 1 || $SPDK_TEST_REDUCE -eq 1 ]]; then
if grep -q '#define SPDK_CONFIG_IGB_UIO_DRIVER 1' $rootdir/include/spdk/config.h; then
./scripts/qat_setup.sh igb_uio
else
./scripts/qat_setup.sh
fi
fi
# Revert existing OPAL to factory settings that may have been left from earlier failed tests.
# This ensures we won't hit any unexpected failures due to NVMe SSDs being locked.
# Disable this for now as we don't have opal test running
# opal_revert_cleanup
#####################
# Unit Tests
#####################
if [ $SPDK_TEST_UNITTEST -eq 1 ]; then
timing_enter unittest
run_test suite ./test/unit/unittest.sh
report_test_completion "unittest"
timing_exit unittest
fi
if [ $SPDK_RUN_FUNCTIONAL_TEST -eq 1 ]; then
timing_enter lib
run_test suite test/env/env.sh
run_test suite test/rpc_client/rpc_client.sh
run_test suite ./test/json_config/json_config.sh
run_test suite test/json_config/alias_rpc/alias_rpc.sh
run_test suite test/spdkcli/tcp.sh
if [ $SPDK_TEST_BLOCKDEV -eq 1 ]; then
run_test suite test/bdev/blockdev.sh
run_test suite test/bdev/bdev_raid.sh
fi
if [ $SPDK_TEST_JSON -eq 1 ]; then
run_test suite test/config_converter/test_converter.sh
fi
if [ $SPDK_TEST_EVENT -eq 1 ]; then
run_test suite test/event/event.sh
fi
if [ $SPDK_TEST_NVME -eq 1 ]; then
run_test suite test/nvme/nvme.sh
if [[ $SPDK_TEST_NVME_CLI -eq 1 ]]; then
run_test suite test/nvme/spdk_nvme_cli.sh
fi
if [[ $SPDK_TEST_NVME_CUSE -eq 1 ]]; then
run_test suite test/nvme/spdk_nvme_cli_cuse.sh
fi
# Only test hotplug without ASAN enabled. Since if it is
# enabled, it catches SEGV earlier than our handler which
# breaks the hotplug logic.
if [ $SPDK_RUN_ASAN -eq 0 ]; then
run_test suite test/nvme/hotplug.sh intel
fi
fi
if [ $SPDK_TEST_IOAT -eq 1 ]; then
run_test suite test/ioat/ioat.sh
fi
timing_exit lib
if [ $SPDK_TEST_ISCSI -eq 1 ]; then
run_test suite ./test/iscsi_tgt/iscsi_tgt.sh posix
run_test suite ./test/spdkcli/iscsi.sh
# Run raid spdkcli test under iSCSI since blockdev tests run on systems that can't run spdkcli yet
run_test suite test/spdkcli/raid.sh
fi
if [ $SPDK_TEST_VPP -eq 1 ]; then
run_test suite ./test/iscsi_tgt/iscsi_tgt.sh vpp
fi
if [ $SPDK_TEST_BLOBFS -eq 1 ]; then
run_test suite ./test/blobfs/rocksdb/rocksdb.sh
run_test suite ./test/blobstore/blobstore.sh
run_test suite ./test/blobfs/blobfs.sh
fi
if [ $SPDK_TEST_NVMF -eq 1 ]; then
run_test suite ./test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
run_test suite ./test/spdkcli/nvmf.sh
fi
if [ $SPDK_TEST_VHOST -eq 1 ]; then
run_test suite ./test/vhost/vhost.sh
report_test_completion "vhost"
fi
if [ $SPDK_TEST_LVOL -eq 1 ]; then
timing_enter lvol
run_test suite ./test/lvol/lvol.sh --test-cases=all
test/lvol: start rewriting python tests to bash There are multiple things wrong with current python tests: * they don't stop the execution on error * the output makes it difficult to understand what really happened inside the test * there is no easy way to reproduce a failure if there is one (besides running the same test script again) * they currently suffer from intermittent failures and there's no-one there to fix them * they stand out from the rest of spdk tests, which are written in bash So we rewrite those tests to bash. They will use rpc.py daemon to send RPC commands, so they won't take any more time to run than python tests. The tests are going to be split them into a few different categories: * clones * snapshots * thin provisioning * tasting * renaming * resizing * all the dumb ones - construct, destruct, etc Each file is a standalone test script, with common utility functions located in test/lvol/common.sh. Each file tests a single, specific feature, but under multiple conditions. Each test case is implemented as a separate function, so if you touch only one lvol feature, you can run only one test script, and if e.g. only a later test case notoriously breaks, you can comment out all the previous test case invocations (up to ~10 lines) and focus only on that failing one. The new tests don't correspond 1:1 to the old python ones - they now cover more. Whenever there was a negative test to check if creating lvs on inexistent bdev failed, we'll now also create a dummy bdev beforehand, so that lvs will have more opportunity to do something it should not. Some other test cases were squashed. A few negative tests required a lot of setup just to try doing something illegal and see if spdk crashed. We'll now do those illegal operations in a single test case, giving lvol lib more opportunity to break. Even if illegal operation did not cause any segfault, is the lvolstore/lvol still usable? E.g. if we try to create an lvol on an lvs that doesn't have enough free clusters and it fails as expected, will it be still possible to create a valid lvol afterwards? Besides sending various RPC commands and checking their return code, we'll also parse and compare various fields in JSON RPC output from get_lvol_stores or get_bdevs RPC. We'll use inline jq calls for that. Whenever something's off, it will be clear which RPC returned invalid values and what were the expected values even without having detailed error prints. The tests are designed to be as easy as possible to debug whenever something goes wrong. This patch removes one test case from python tests and adds a corresponding test into the new test/lvol/lvol2.sh file. The script will be renamed to just lvol.sh after the existing lvol.sh (which starts all python tests) is finally removed. As for the bash script itself - each test case is run through a run_test() function which verifies there were no lvolstores, lvols, or bdevs left after the test case has finished. Inside the particular tests we will still check if the lvolstore removal at the end was successful, but that's because we want to make sure it's gone e.g even before we remove the underlying lvs' base bdev. Change-Id: Iaa2bb656233e1c9f0c35093f190ac26c39e78623 Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com> Signed-off-by: Pawel Kaminski <pawelx.kaminski@intel.com> Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459517 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Reviewed-by: Karol Latecki <karol.latecki@intel.com> Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
2019-06-10 13:22:11 +00:00
run_test suite ./test/lvol/lvol2.sh
run_test suite ./test/blobstore/blob_io_wait/blob_io_wait.sh
report_test_completion "lvol"
timing_exit lvol
fi
if [ $SPDK_TEST_VHOST_INIT -eq 1 ]; then
timing_enter vhost_initiator
run_test suite ./test/vhost/initiator/blockdev.sh
run_test suite ./test/spdkcli/virtio.sh
run_test suite ./test/vhost/shared/shared.sh
run_test suite ./test/vhost/fuzz/fuzz.sh
report_test_completion "vhost_initiator"
timing_exit vhost_initiator
fi
if [ $SPDK_TEST_PMDK -eq 1 ]; then
run_test suite ./test/pmem/pmem.sh -x
run_test suite ./test/spdkcli/pmem.sh
fi
if [ $SPDK_TEST_RBD -eq 1 ]; then
run_test suite ./test/spdkcli/rbd.sh
fi
if [ $SPDK_TEST_OCF -eq 1 ]; then
run_test suite ./test/ocf/ocf.sh
fi
if [ $SPDK_TEST_FTL -eq 1 ]; then
run_test suite ./test/ftl/ftl.sh
fi
if [ $SPDK_TEST_VMD -eq 1 ]; then
run_test suite ./test/vmd/vmd.sh
fi
if [ $SPDK_TEST_REDUCE -eq 1 ]; then
run_test suite ./test/compress/compress.sh
fi
if [ $SPDK_TEST_OPAL -eq 1 ]; then
run_test suite ./test/nvme/nvme_opal.sh
fi
fi
timing_enter cleanup
autotest_cleanup
timing_exit cleanup
timing_exit autotest
chmod a+r $output_dir/timing.txt
trap - SIGINT SIGTERM EXIT
# catch any stray core files
process_core
if hash lcov; then
# generate coverage data and combine with baseline
$LCOV -q -c -d $src -t "$(hostname)" -o cov_test.info
$LCOV -q -a cov_base.info -a cov_test.info -o $out/cov_total.info
$LCOV -q -r $out/cov_total.info '*/dpdk/*' -o $out/cov_total.info
$LCOV -q -r $out/cov_total.info '/usr/*' -o $out/cov_total.info
git clean -f "*.gcda"
rm -f cov_base.info cov_test.info OLD_STDOUT OLD_STDERR
fi