From 4a74580112c65a630aa22ce41740d0341a5d1a8b Mon Sep 17 00:00:00 2001 From: Jim Harris Date: Wed, 15 Mar 2017 14:47:17 -0700 Subject: [PATCH] bdev: add a getting started guide For now, this contains the documentation for configuring block devices in configuration files. This file can then be used a common reference for other getting started guides - iscsi, vhost and nvmf targets. Signed-off-by: Jim Harris Change-Id: Ie6f6c0b3f36dd3fdf418b904462c81a1696b9694 --- doc/Doxyfile | 8 +-- doc/bdev/getting_started.md | 89 ++++++++++++++++++++++++++++++++ doc/bdev/index.md | 3 ++ doc/index.md | 1 + doc/iscsi/getting_started.md | 99 +++--------------------------------- 5 files changed, 104 insertions(+), 96 deletions(-) create mode 100644 doc/bdev/getting_started.md create mode 100644 doc/bdev/index.md diff --git a/doc/Doxyfile b/doc/Doxyfile index c4fc9a274..383756447 100644 --- a/doc/Doxyfile +++ b/doc/Doxyfile @@ -762,17 +762,19 @@ INPUT = ../include/spdk \ index.md \ directory_structure.md \ porting.md \ + bdev/index.md \ + bdev/getting_started.md \ event/index.md \ ioat/index.md \ + iscsi/index.md \ + iscsi/getting_started.md \ nvme/index.md \ nvme/async_completion.md \ nvme/fabrics.md \ nvme/initialization.md \ nvme/io_submission.md \ nvmf/index.md \ - nvmf/getting_started.md \ - iscsi/index.md \ - iscsi/getting_started.md + nvmf/getting_started.md # This tag can be used to specify the character encoding of the source files # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses diff --git a/doc/bdev/getting_started.md b/doc/bdev/getting_started.md new file mode 100644 index 000000000..649e646c6 --- /dev/null +++ b/doc/bdev/getting_started.md @@ -0,0 +1,89 @@ +# SPDK bdev Getting Started Guide {#bdev_getting_started} + +Block storage in SPDK applications is provided by the SPDK bdev layer. SPDK bdev consists of: + +* a driver module API for implementing bdev drivers +* an application API for enumerating and claiming SPDK block devices and performance operations +(read, write, unmap, etc.) on those devices +* bdev drivers for NVMe, malloc (ramdisk), Linux AIO and Ceph RBD +* configuration via SPDK configuration files or JSON RPC + +# Configuring block devices {#bdev_config} + +SPDK block devices are typically configured via an SPDK configuration file. These block devices +can then be associated with higher level abstractions such as iSCSI target nodes, NVMe-oF namespaces +or vhost-scsi controllers. This section will describe how to configure block devices for the +SPDK bdev drivers included with SPDK. + +## NVMe + +The SPDK nvme bdev driver provides SPDK block layer access to NVMe SSDs via the SPDK userspace +NVMe driver. The nvme bdev driver binds only to devices explicitly specified. These devices +can be either locally attached SSDs or remote NVMe subsystems via NVMe-oF. + +~~~ +[Nvme] + # NVMe Device Whitelist + # Users may specify which NVMe devices to claim by their transport id. + # See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format. + # The devices will be assigned names in the format nY, where YourName is the + # name specified at the end of the TransportId line and Y is the namespace id, which starts at 1. + TransportID "trtype:PCIe traddr:0000:00:00.0" Nvme0 + TransportID "trtype:RDMA subnqn:nqn.2016-06.io.spdk:cnode1 traddr:192.168.100.1 trsvcid:4420" Nvme1 +~~~ + +This exports block devices for all namespaces attached to the two controllers. Block devices +for namespaces attached to the first controller will be in the format Nvme0nY, where Y is +the namespace ID. Most NVMe SSDs have a single namespace with ID=1. Block devices attached to +the second controller will be in the format Nvme1nY. + +## Malloc + +The SPDK malloc bdev driver allocates a buffer of memory in userspace as the target for block I/O +operations. This effectively serves as a userspace ramdisk target. + +Configuration file syntax: +~~~ +[Malloc] + NumberOfLuns 4 + LunSizeInMB 64 +~~~ + +This exports 4 malloc block devices, named Malloc0 through Malloc3. Each malloc block device will +be 64MB in size. + +## Linux AIO + +The SPDK aio bdev driver provides SPDK block layer access to Linux kernel block devices via Linux AIO. +Note that O_DIRECT is used and thus bypasses the Linux page cache. This mode is probably as close to +a typical kernel based target as a user space target can get without using a user-space driver. + +Configuration file syntax: + +~~~ +[AIO] + # normal file or block device + AIO /dev/sdb + AIO /dev/sdc +~~~ + +This exports 2 aio block devices, named AIO0 and AIO1. + +## Ceph RBD + +The SPDK rbd bdev driver provides SPDK block layer access to Ceph RADOS block devices (RBD). Ceph +RBD devices are accessed via librbd and librados libraries to access the RADOS block device +exported by Ceph. + +Configuration file syntax: + +~~~ +[Ceph] + # The format of provided rbd info should be: Ceph rbd_pool_name rbd_name size. + # In the following example, rbd is the name of rbd_pool; foo is the name of + # rbd device exported by Ceph; value 512 represents the configured block size + # for this rbd, the block size should be a multiple of 512. + Ceph rbd foo 512 +~~~ + +This exports 1 rbd block device, named Ceph0. diff --git a/doc/bdev/index.md b/doc/bdev/index.md new file mode 100644 index 000000000..2873baa5b --- /dev/null +++ b/doc/bdev/index.md @@ -0,0 +1,3 @@ +# Block Device Abstraction Layer {#bdev} + +- @ref bdev_getting_started diff --git a/doc/index.md b/doc/index.md index 804284be2..06f1b16f1 100644 --- a/doc/index.md +++ b/doc/index.md @@ -24,3 +24,4 @@ which avoids kernel context switches and eliminates interrupt handling overhead. - @ref nvmf - @ref ioat - @ref iscsi +- @ref bdev diff --git a/doc/iscsi/getting_started.md b/doc/iscsi/getting_started.md index 29e089c0a..18142fddc 100644 --- a/doc/iscsi/getting_started.md +++ b/doc/iscsi/getting_started.md @@ -21,107 +21,20 @@ TCP ports to use as iSCSI portals; general iSCSI parameters; initiator names and access to iSCSI target nodes; number and types of storage backends to export over iSCSI LUNs; iSCSI target node mappings between portal groups, initiator groups, and LUNs. -The SPDK iSCSI target supports several different types of storage backends. These backends will create -SCSI LUNs which can be exported via iSCSI target nodes. +Each LUN in an iSCSI target node is associated with an SPDK block device. See @ref bdev_getting_started +for details on configuring SPDK block devices. The block device to LUN mappings are specified in the +configuration file as: -The storage backends can be configured in the iscsi.conf configuration file to specify the number or -size of LUNs, block device names (for exporting in-kernel block devices), or other parameters. - -Currently there are 3 types of storage backends supported by the iSCSI target: - -## Malloc - -Configuration file syntax: -~~~ -[Malloc] - NumberOfLuns 4 - LunSizeInMB 64 -~~~ - -Other TargetNode parameters go here (TargetName, Mapping, etc.): -~~~ +~~~~ [TargetNodeX] LUN0 Malloc0 -~~~ + LUN1 Nvme0n1 +~~~~ This exports a malloc'd target. The disk is a RAM disk that is a chunk of memory allocated by iscsi in user space. It will use offload engine to do the copy job instead of memcpy if the system has enough DMA channels. -## Block Device - -AIO devices are accessed using Linux AIO. O_DIRECT is used and thus unaligned writes will be double buffered. -This option also bypasses the Linux page cache. This mode is probably as close to a typical kernel based -target as a user space target can get without using a user-space driver. - -Configuration file syntax: - -~~~ -[AIO] - #normal file or block device - AIO /dev/sdb -~~~ - -Other TargetNode parameters go here (TargetName, Mapping, etc.): -~~~ -[TargetNodeX] - LUN0 AIO0 -~~~ - -## Ceph RBD - -Ceph RBD devices are accessed via librbd and librados libraries to access the RADOS block device -exported by Ceph. - -Configuration file syntax: - -~~~ -[Ceph] - # The format of provided rbd info should be: Ceph rbd_pool_name rbd_name size. - # In the following example, rbd is the name of rbd_pool; foo is the name of - # rbd device exported by Ceph; value 512 represents the configured block size - # for this rbd, the block size should be a multiple of 512. - Ceph rbd foo 512 -~~~ - -Other TargetNode parameters go here (TargetName, Mapping, etc.): -~~~ -[TargetNodeX] - LUN0 Ceph0 -~~~ - -## NVMe - -The SPDK NVMe driver by default binds to all NVMe controllers which are not bound to the kernel-mode -nvme driver. Users can choose to bind to fewer controllers by setting the NumControllers parameter. -Then the NVMe backend controls NVMe controller(s) directly from userspace and completely bypasses -the kernel to avoid interrupts and context switching. - -~~~ -[Nvme] - # NVMe Device Whitelist - # Users may specify which NVMe devices to claim by their transport id. - # See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format. - # The second argument is the assigned name, which can be referenced from - # other sections in the configuration file. For NVMe devices, a namespace - # is automatically appended to each name in the format nY, where - # Y is the NSID (starts at 1). - TransportID "trtype:PCIe traddr:0000:00:00.0" Nvme0 - TransportID "trtype:PCIe traddr:0000:01:00.0" Nvme1 - - # The number of attempts per I/O when an I/O fails. Do not include - # this key to get the default behavior. - NvmeRetryCount 4 - -[TargetNodeX] - # other TargetNode parameters go here (TargetName, Mapping, etc.) - # nvme with the following format: NvmeXnY, where X = the controller ID - # and Y = the namespace ID - # Note: NVMe namespace IDs always start at 1, not 0 - and most - # controllers have only 1 namespace. - LUN0 Nvme0n1 -~~~ - You should make a copy of the example configuration file, modify it to suit your environment, and then run the iscsi_tgt application and pass it the configuration file using the -c option. Right now, the target requires elevated privileges (root) to run.