29784f35cd
bdevperf application is utilized in multiple test scenarios and most prominently in SPDK performance reports. Similar to perf and fio_plugins, it is used to measure performance. It is intended to represent the expected behavior of users application that use SPDK bdev layer. Applications under --enable-tests are intended for specific test scenarios and user interaction is usually via a test scripts. Meanwhile --enable-examples are used more broadly to teach and show usage of SPDK libraries. As an added benefit with this patch, its possible to compile bdevperf without need to satisfy additional requirements to compile all the test applications. Change-Id: I9aaec1f9d729a1cdee89b5fdc365d61c19b03f82 Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com> Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14558 Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com> Reviewed-by: Paul Luse <paul.e.luse@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Mellanox Build Bot
104 lines
3.5 KiB
Markdown
104 lines
3.5 KiB
Markdown
# NVMe-oF Multipath HOWTO {#nvmf_multipath_howto}
|
|
|
|
This HOWTO provides step-by-step instructions for setting-up a simple SPDK deployment and testing multipath.
|
|
It demonstrates configuring path preferences with Asymmetric Namespace Access (ANA), as well as round-robin
|
|
path load balancing.
|
|
|
|
## Build SPDK on both the initiator and target servers
|
|
|
|
Clone the repo:
|
|
~~~{.sh}
|
|
git clone https://github.com/spdk/spdk --recursive
|
|
~~~
|
|
|
|
Configure and build SPDK:
|
|
~~~{.sh}
|
|
cd spdk/
|
|
./configure
|
|
make -j16
|
|
~~~
|
|
|
|
## Setup hugepages
|
|
|
|
This should be run once on each server (and after reboots):
|
|
~~~{.sh}
|
|
cd spdk/
|
|
./scripts/setup.sh
|
|
~~~
|
|
|
|
## On target: start and configure SPDK
|
|
|
|
Start the target in the background and configure it:
|
|
~~~{.sh}
|
|
cd spdk/
|
|
./build/bin/nvmf_tgt -m 0x3 &
|
|
./scripts/rpc.py nvmf_create_transport -t tcp -o -u 8192
|
|
~~~
|
|
|
|
Create a subsystem, with `-r` to enable ANA reporting feature:
|
|
~~~{.sh}
|
|
./scripts/rpc.py nvmf_create_subsystem nqn.2022-02.io.spdk:cnode0 -a -s SPDK00000000000001 -r
|
|
~~~
|
|
|
|
Create and add a malloc block device:
|
|
~~~{.sh}
|
|
./scripts/rpc.py bdev_malloc_create 64 512 -b Malloc0
|
|
./scripts/rpc.py nvmf_subsystem_add_ns nqn.2022-02.io.spdk:cnode0 Malloc0
|
|
~~~
|
|
|
|
Add two listeners, each with a different `IP:port` pair:
|
|
~~~{.sh}
|
|
./scripts/rpc.py nvmf_subsystem_add_listener -t tcp -a 172.17.1.13 -s 4420 nqn.2022-02.io.spdk:cnode0
|
|
./scripts/rpc.py nvmf_subsystem_add_listener -t tcp -a 172.18.1.13 -s 5520 nqn.2022-02.io.spdk:cnode0
|
|
~~~
|
|
|
|
## On initiator: start and configure bdevperf
|
|
|
|
Launch the bdevperf process in the background:
|
|
~~~{.sh}
|
|
cd spdk/
|
|
./build/examples/bdevperf -m 0x4 -z -r /tmp/bdevperf.sock -q 128 -o 4096 -w verify -t 90 &> bdevperf.log &
|
|
~~~
|
|
|
|
Configure bdevperf and add two paths:
|
|
~~~{.sh}
|
|
./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_set_options -r -1
|
|
./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_attach_controller -b Nvme0 -t tcp -a 172.17.1.13 -s 4420 -f ipv4 -n nqn.2022-02.io.spdk:cnode0 -l -1 -o 10
|
|
./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_attach_controller -b Nvme0 -t tcp -a 172.18.1.13 -s 5520 -f ipv4 -n nqn.2022-02.io.spdk:cnode0 -x multipath -l -1 -o 10
|
|
~~~
|
|
|
|
## Launch a bdevperf test
|
|
|
|
Connect to the RPC socket of the bdevperf process and start the test:
|
|
~~~{.sh}
|
|
PYTHONPATH=$PYTHONPATH:/root/src/spdk/python ./examples/bdev/bdevperf/bdevperf.py -t 1 -s /tmp/bdevperf.sock perform_tests
|
|
~~~
|
|
|
|
The RPC command will return, leaving the test to run for 90 seconds in the background. On the target server,
|
|
observe that only the first path (port) is receiving packets by checking the queues with `ss -t`.
|
|
|
|
You can view the paths available to the initiator with:
|
|
~~~{.sh}
|
|
./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_get_io_paths -n Nvme0n1
|
|
~~~
|
|
|
|
## Switching paths
|
|
|
|
This can be done on the target server by setting the first path's ANA to `non_optimized`:
|
|
~~~{.sh}
|
|
./scripts/rpc.py nvmf_subsystem_listener_set_ana_state nqn.2022-02.io.spdk:cnode0 -t tcp -a 172.17.1.13 -s 4420 -n non_optimized
|
|
~~~
|
|
|
|
Use `ss -t` to verify that the traffic has switched to the second path.
|
|
|
|
## Use round-robin (active_active) path load balancing
|
|
|
|
First, ensure the ANA for both paths is configured as `optimized` on the target. Then, change the
|
|
multipath policy on the initiator to `active_active` (multipath policy is per bdev, so
|
|
`bdev_nvme_set_multipath_policy` must be called after `bdev_nvme_attach_controller`):
|
|
~~~{.sh}
|
|
./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_set_multipath_policy -b Nvme0n1 -p active_active
|
|
~~~
|
|
|
|
Observe with `ss -t` that both connections are receiving traffic (queues build up).
|