test/lvol: start rewriting python tests to bash

There are multiple things wrong with current python tests:
 * they don't stop the execution on error
 * the output makes it difficult to understand what really
   happened inside the test
 * there is no easy way to reproduce a failure if there
   is one (besides running the same test script again)
 * they currently suffer from intermittent failures and
   there's no-one there to fix them
 * they stand out from the rest of spdk tests, which are
   written in bash

So we rewrite those tests to bash. They will use rpc.py
daemon to send RPC commands, so they won't take any more
time to run than python tests.

The tests are going to be split them into a few different
categories:
 * clones
 * snapshots
 * thin provisioning
 * tasting
 * renaming
 * resizing
 * all the dumb ones - construct, destruct, etc

Each file is a standalone test script, with common utility
functions located in test/lvol/common.sh. Each file tests
a single, specific feature, but under multiple conditions.
Each test case is implemented as a separate function, so
if you touch only one lvol feature, you can run only one
test script, and if e.g. only a later test case notoriously
breaks, you can comment out all the previous test case
invocations (up to ~10 lines) and focus only on that
failing one.

The new tests don't correspond 1:1 to the old python ones
- they now cover more. Whenever there was a negative test
to check if creating lvs on inexistent bdev failed, we'll
now also create a dummy bdev beforehand, so that lvs will
have more opportunity to do something it should not.
Some other test cases were squashed. A few negative tests
required a lot of setup just to try doing something
illegal and see if spdk crashed. We'll now do those illegal
operations in a single test case, giving lvol lib more
opportunity to break. Even if illegal operation did not
cause any segfault, is the lvolstore/lvol still usable?
E.g. if we try to create an lvol on an lvs that doesn't
have enough free clusters and it fails as expected, will
it be still possible to create a valid lvol afterwards?

Besides sending various RPC commands and checking their
return code, we'll also parse and compare various fields
in JSON RPC output from get_lvol_stores or get_bdevs RPC.
We'll use inline jq calls for that. Whenever something's
off, it will be clear which RPC returned invalid values
and what were the expected values even without having
detailed error prints.

The tests are designed to be as easy as possible to debug
whenever something goes wrong.

This patch removes one test case from python tests and
adds a corresponding test into the new test/lvol/lvol2.sh
file. The script will be renamed to just lvol.sh after
the existing lvol.sh (which starts all python tests) is
finally removed.

As for the bash script itself - each test case is run
through a run_test() function which verifies there were
no lvolstores, lvols, or bdevs left after the test case
has finished. Inside the particular tests we will still
check if the lvolstore removal at the end was successful,
but that's because we want to make sure it's gone e.g even
before we remove the underlying lvs' base bdev.

Change-Id: Iaa2bb656233e1c9f0c35093f190ac26c39e78623
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Signed-off-by: Pawel Kaminski <pawelx.kaminski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459517
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>

autotest.sh

@@ -226,6 +226,7 @@ if [ $SPDK_RUN_FUNCTIONAL_TEST -eq 1 ]; then
 	if [ $SPDK_TEST_LVOL -eq 1 ]; then
 		timing_enter lvol
 		run_test suite ./test/lvol/lvol.sh --test-cases=all
+		run_test suite ./test/lvol/lvol2.sh
 		run_test suite ./test/blobstore/blob_io_wait/blob_io_wait.sh
 		report_test_completion "lvol"
 		timing_exit lvol

test/lvol/basic.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+testdir=$(readlink -f $(dirname $0))
+rootdir=$(readlink -f $testdir/../..)
+source $rootdir/test/common/autotest_common.sh
+source $rootdir/test/lvol/common.sh
+
+# create empty lvol store and verify its parameters
+function test_construct_lvs() {
+	# create an lvol store
+	malloc_name=$(rpc_cmd bdev_malloc_create $MALLOC_SIZE_MB $MALLOC_BS)
+	lvs_uuid=$(rpc_cmd bdev_lvol_create_lvstore "$malloc_name" lvs_test)
+	lvs=$(rpc_cmd bdev_lvol_get_lvstores -u "$lvs_uuid")
+
+	# verify it's there
+	[ "$(jq -r '.[0].uuid' <<< "$lvs")" = "$lvs_uuid" ]
+	[ "$(jq -r '.[0].name' <<< "$lvs")" = "lvs_test" ]
+	[ "$(jq -r '.[0].base_bdev' <<< "$lvs")" = "$malloc_name" ]
+
+	# verify some of its parameters
+	cluster_size=$(jq -r '.[0].cluster_size' <<< "$lvs")
+	[ "$cluster_size" = "$LVS_DEFAULT_CLUSTER_SIZE" ]
+	total_clusters=$(jq -r '.[0].total_data_clusters' <<< "$lvs")
+	[ "$(jq -r '.[0].free_clusters' <<< "$lvs")" = "$total_clusters" ]
+	[ "$(( total_clusters * cluster_size ))" = "$LVS_DEFAULT_CAPACITY" ]
+
+	# remove it and verify it's gone
+	rpc_cmd bdev_lvol_delete_lvstore -u "$lvs_uuid"
+	! rpc_cmd bdev_lvol_get_lvstores -u "$lvs_uuid"
+	rpc_cmd bdev_malloc_delete "$malloc_name"
+}
+
+$rootdir/app/spdk_tgt/spdk_tgt &
+spdk_pid=$!
+trap 'killprocess "$spdk_pid"; exit 1' SIGINT SIGTERM EXIT
+waitforlisten $spdk_pid
+
+run_lvol_test test_construct_lvs
+
+trap - SIGINT SIGTERM EXIT
+killprocess $spdk_pid

test/lvol/common.sh

@@ -0,0 +1,20 @@
+MALLOC_SIZE_MB=128
+MALLOC_BS=512
+LVS_DEFAULT_CLUSTER_SIZE_MB=4
+LVS_DEFAULT_CLUSTER_SIZE=$(( LVS_DEFAULT_CLUSTER_SIZE_MB * 1024 * 1024 ))
+# reserve some MBs for lvolstore metadata
+LVS_DEFAULT_CAPACITY_MB=$(( MALLOC_SIZE_MB - LVS_DEFAULT_CLUSTER_SIZE_MB ))
+LVS_DEFAULT_CAPACITY=$(( LVS_DEFAULT_CAPACITY_MB * 1024 * 1024 ))
+
+function rpc_cmd() {
+	$rootdir/scripts/rpc.py $@
+}
+
+function run_lvol_test() {
+	run_test suite $@
+
+	leftover_bdevs=$(rpc_cmd bdev_get_bdevs)
+	[ "$(jq length <<< "$leftover_bdevs")" == "0" ]
+	leftover_lvs=$(rpc_cmd bdev_lvol_get_lvstores)
+	[ "$(jq length <<< "$leftover_lvs")" == "0" ]
+}

test/lvol/lvol.sh

@@ -21,7 +21,6 @@ function usage() {
     echo "    --block-size          Block size for this bdev"
     echo "-x                        set -x for script debug"
     echo "    --test-cases=         List test cases which will be run:
-                                    1: 'construct_lvs_positive',
                                     50: 'construct_logical_volume_positive',
                                     51: 'construct_multi_logical_volumes_positive',
                                     52: 'bdev_lvol_create_using_name_positive',

test/lvol/lvol2.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+testdir=$(readlink -f $(dirname $0))
+rootdir=$(readlink -f $testdir/../..)
+source $rootdir/test/common/autotest_common.sh
+
+timing_enter lvol
+
+timing_enter basic
+run_test suite test/lvol/basic.sh
+timing_exit basic
+
+timing_exit lvol

test/lvol/test_cases.py

@@ -110,8 +110,6 @@ def test_counter():
 def case_message(func):
     def inner(*args, **kwargs):
         test_name = {
-            # bdev_lvol_create_lvstore  - positive tests
-            1: 'construct_lvs_positive',
             # bdev_lvol_create - positive tests
             50: 'construct_logical_volume_positive',
             51: 'construct_multi_logical_volumes_positive',
@@ -314,36 +312,6 @@ class TestCases(object):
         lvs = self.c.bdev_lvol_get_lvstores(lvs_name)[0]
         return int(int(lvs['cluster_size']) / MEGABYTE)
 
-    # positive tests
-    @case_message
-    def test_case1(self):
-        """
-        construct_lvs_positive
-
-        Positive test for constructing a new lvol store.
-        Call bdev_lvol_create_lvstore with correct base bdev name.
-        """
-        # Create malloc bdev
-        base_name = self.c.bdev_malloc_create(self.total_size,
-                                              self.block_size)
-        # Construct_lvol_store on correct, exisitng malloc bdev
-        uuid_store = self.c.bdev_lvol_create_lvstore(base_name,
-                                                     self.lvs_name)
-        # Check correct uuid values in response bdev_lvol_get_lvstores command
-        fail_count = self.c.check_bdev_lvol_get_lvstores(base_name, uuid_store,
-                                                         self.cluster_size)
-        self.c.bdev_lvol_delete_lvstore(uuid_store)
-        self.c.bdev_malloc_delete(base_name)
-        if self.c.check_bdev_lvol_get_lvstores("", "", "") == 1:
-            fail_count += 1
-
-        # Expected result
-        # - call successful, return code = 0, uuid printed to stdout
-        # - bdev_lvol_get_lvstores: backend used for bdev_lvol_create_lvstore has uuid
-        #   field set with the same uuid as returned from RPC call
-        # - no other operation fails
-        return fail_count
-
     @case_message
     def test_case50(self):
         """