bdev: add a getting started guide
For now, this contains the documentation for configuring block devices in configuration files. This file can then be used a common reference for other getting started guides - iscsi, vhost and nvmf targets. Signed-off-by: Jim Harris <james.r.harris@intel.com> Change-Id: Ie6f6c0b3f36dd3fdf418b904462c81a1696b9694
This commit is contained in:
Jim Harris
parent
29a3efdd26
commit
4a74580112
@ -762,17 +762,19 @@ INPUT = ../include/spdk \
|
||||
index.md \
|
||||
directory_structure.md \
|
||||
porting.md \
|
||||
bdev/index.md \
|
||||
bdev/getting_started.md \
|
||||
event/index.md \
|
||||
ioat/index.md \
|
||||
iscsi/index.md \
|
||||
iscsi/getting_started.md \
|
||||
nvme/index.md \
|
||||
nvme/async_completion.md \
|
||||
nvme/fabrics.md \
|
||||
nvme/initialization.md \
|
||||
nvme/io_submission.md \
|
||||
nvmf/index.md \
|
||||
nvmf/getting_started.md \
|
||||
iscsi/index.md \
|
||||
iscsi/getting_started.md
|
||||
nvmf/getting_started.md
|
||||
|
||||
# This tag can be used to specify the character encoding of the source files
|
||||
# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
|
||||
|
||||
89
doc/bdev/getting_started.md
Normal file
89
doc/bdev/getting_started.md
Normal file
@ -0,0 +1,89 @@
|
||||
# SPDK bdev Getting Started Guide {#bdev_getting_started}
|
||||
|
||||
Block storage in SPDK applications is provided by the SPDK bdev layer. SPDK bdev consists of:
|
||||
|
||||
* a driver module API for implementing bdev drivers
|
||||
* an application API for enumerating and claiming SPDK block devices and performance operations
|
||||
(read, write, unmap, etc.) on those devices
|
||||
* bdev drivers for NVMe, malloc (ramdisk), Linux AIO and Ceph RBD
|
||||
* configuration via SPDK configuration files or JSON RPC
|
||||
|
||||
# Configuring block devices {#bdev_config}
|
||||
|
||||
SPDK block devices are typically configured via an SPDK configuration file. These block devices
|
||||
can then be associated with higher level abstractions such as iSCSI target nodes, NVMe-oF namespaces
|
||||
or vhost-scsi controllers. This section will describe how to configure block devices for the
|
||||
SPDK bdev drivers included with SPDK.
|
||||
|
||||
## NVMe
|
||||
|
||||
The SPDK nvme bdev driver provides SPDK block layer access to NVMe SSDs via the SPDK userspace
|
||||
NVMe driver. The nvme bdev driver binds only to devices explicitly specified. These devices
|
||||
can be either locally attached SSDs or remote NVMe subsystems via NVMe-oF.
|
||||
|
||||
~~~
|
||||
[Nvme]
|
||||
# NVMe Device Whitelist
|
||||
# Users may specify which NVMe devices to claim by their transport id.
|
||||
# See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format.
|
||||
# The devices will be assigned names in the format <YourName>nY, where YourName is the
|
||||
# name specified at the end of the TransportId line and Y is the namespace id, which starts at 1.
|
||||
TransportID "trtype:PCIe traddr:0000:00:00.0" Nvme0
|
||||
TransportID "trtype:RDMA subnqn:nqn.2016-06.io.spdk:cnode1 traddr:192.168.100.1 trsvcid:4420" Nvme1
|
||||
~~~
|
||||
|
||||
This exports block devices for all namespaces attached to the two controllers. Block devices
|
||||
for namespaces attached to the first controller will be in the format Nvme0nY, where Y is
|
||||
the namespace ID. Most NVMe SSDs have a single namespace with ID=1. Block devices attached to
|
||||
the second controller will be in the format Nvme1nY.
|
||||
|
||||
## Malloc
|
||||
|
||||
The SPDK malloc bdev driver allocates a buffer of memory in userspace as the target for block I/O
|
||||
operations. This effectively serves as a userspace ramdisk target.
|
||||
|
||||
Configuration file syntax:
|
||||
~~~
|
||||
[Malloc]
|
||||
NumberOfLuns 4
|
||||
LunSizeInMB 64
|
||||
~~~
|
||||
|
||||
This exports 4 malloc block devices, named Malloc0 through Malloc3. Each malloc block device will
|
||||
be 64MB in size.
|
||||
|
||||
## Linux AIO
|
||||
|
||||
The SPDK aio bdev driver provides SPDK block layer access to Linux kernel block devices via Linux AIO.
|
||||
Note that O_DIRECT is used and thus bypasses the Linux page cache. This mode is probably as close to
|
||||
a typical kernel based target as a user space target can get without using a user-space driver.
|
||||
|
||||
Configuration file syntax:
|
||||
|
||||
~~~
|
||||
[AIO]
|
||||
# normal file or block device
|
||||
AIO /dev/sdb
|
||||
AIO /dev/sdc
|
||||
~~~
|
||||
|
||||
This exports 2 aio block devices, named AIO0 and AIO1.
|
||||
|
||||
## Ceph RBD
|
||||
|
||||
The SPDK rbd bdev driver provides SPDK block layer access to Ceph RADOS block devices (RBD). Ceph
|
||||
RBD devices are accessed via librbd and librados libraries to access the RADOS block device
|
||||
exported by Ceph.
|
||||
|
||||
Configuration file syntax:
|
||||
|
||||
~~~
|
||||
[Ceph]
|
||||
# The format of provided rbd info should be: Ceph rbd_pool_name rbd_name size.
|
||||
# In the following example, rbd is the name of rbd_pool; foo is the name of
|
||||
# rbd device exported by Ceph; value 512 represents the configured block size
|
||||
# for this rbd, the block size should be a multiple of 512.
|
||||
Ceph rbd foo 512
|
||||
~~~
|
||||
|
||||
This exports 1 rbd block device, named Ceph0.
|
||||
3
doc/bdev/index.md
Normal file
3
doc/bdev/index.md
Normal file
@ -0,0 +1,3 @@
|
||||
# Block Device Abstraction Layer {#bdev}
|
||||
|
||||
- @ref bdev_getting_started
|
||||
@ -24,3 +24,4 @@ which avoids kernel context switches and eliminates interrupt handling overhead.
|
||||
- @ref nvmf
|
||||
- @ref ioat
|
||||
- @ref iscsi
|
||||
- @ref bdev
|
||||
|
||||
@ -21,107 +21,20 @@ TCP ports to use as iSCSI portals; general iSCSI parameters; initiator names and
|
||||
access to iSCSI target nodes; number and types of storage backends to export over iSCSI LUNs; iSCSI
|
||||
target node mappings between portal groups, initiator groups, and LUNs.
|
||||
|
||||
The SPDK iSCSI target supports several different types of storage backends. These backends will create
|
||||
SCSI LUNs which can be exported via iSCSI target nodes.
|
||||
Each LUN in an iSCSI target node is associated with an SPDK block device. See @ref bdev_getting_started
|
||||
for details on configuring SPDK block devices. The block device to LUN mappings are specified in the
|
||||
configuration file as:
|
||||
|
||||
The storage backends can be configured in the iscsi.conf configuration file to specify the number or
|
||||
size of LUNs, block device names (for exporting in-kernel block devices), or other parameters.
|
||||
|
||||
Currently there are 3 types of storage backends supported by the iSCSI target:
|
||||
|
||||
## Malloc
|
||||
|
||||
Configuration file syntax:
|
||||
~~~
|
||||
[Malloc]
|
||||
NumberOfLuns 4
|
||||
LunSizeInMB 64
|
||||
~~~
|
||||
|
||||
Other TargetNode parameters go here (TargetName, Mapping, etc.):
|
||||
~~~
|
||||
~~~~
|
||||
[TargetNodeX]
|
||||
LUN0 Malloc0
|
||||
~~~
|
||||
LUN1 Nvme0n1
|
||||
~~~~
|
||||
|
||||
This exports a malloc'd target. The disk is a RAM disk that is a chunk of memory allocated by iscsi in
|
||||
user space. It will use offload engine to do the copy job instead of memcpy if the system has enough DMA
|
||||
channels.
|
||||
|
||||
## Block Device
|
||||
|
||||
AIO devices are accessed using Linux AIO. O_DIRECT is used and thus unaligned writes will be double buffered.
|
||||
This option also bypasses the Linux page cache. This mode is probably as close to a typical kernel based
|
||||
target as a user space target can get without using a user-space driver.
|
||||
|
||||
Configuration file syntax:
|
||||
|
||||
~~~
|
||||
[AIO]
|
||||
#normal file or block device
|
||||
AIO /dev/sdb
|
||||
~~~
|
||||
|
||||
Other TargetNode parameters go here (TargetName, Mapping, etc.):
|
||||
~~~
|
||||
[TargetNodeX]
|
||||
LUN0 AIO0
|
||||
~~~
|
||||
|
||||
## Ceph RBD
|
||||
|
||||
Ceph RBD devices are accessed via librbd and librados libraries to access the RADOS block device
|
||||
exported by Ceph.
|
||||
|
||||
Configuration file syntax:
|
||||
|
||||
~~~
|
||||
[Ceph]
|
||||
# The format of provided rbd info should be: Ceph rbd_pool_name rbd_name size.
|
||||
# In the following example, rbd is the name of rbd_pool; foo is the name of
|
||||
# rbd device exported by Ceph; value 512 represents the configured block size
|
||||
# for this rbd, the block size should be a multiple of 512.
|
||||
Ceph rbd foo 512
|
||||
~~~
|
||||
|
||||
Other TargetNode parameters go here (TargetName, Mapping, etc.):
|
||||
~~~
|
||||
[TargetNodeX]
|
||||
LUN0 Ceph0
|
||||
~~~
|
||||
|
||||
## NVMe
|
||||
|
||||
The SPDK NVMe driver by default binds to all NVMe controllers which are not bound to the kernel-mode
|
||||
nvme driver. Users can choose to bind to fewer controllers by setting the NumControllers parameter.
|
||||
Then the NVMe backend controls NVMe controller(s) directly from userspace and completely bypasses
|
||||
the kernel to avoid interrupts and context switching.
|
||||
|
||||
~~~
|
||||
[Nvme]
|
||||
# NVMe Device Whitelist
|
||||
# Users may specify which NVMe devices to claim by their transport id.
|
||||
# See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format.
|
||||
# The second argument is the assigned name, which can be referenced from
|
||||
# other sections in the configuration file. For NVMe devices, a namespace
|
||||
# is automatically appended to each name in the format <YourName>nY, where
|
||||
# Y is the NSID (starts at 1).
|
||||
TransportID "trtype:PCIe traddr:0000:00:00.0" Nvme0
|
||||
TransportID "trtype:PCIe traddr:0000:01:00.0" Nvme1
|
||||
|
||||
# The number of attempts per I/O when an I/O fails. Do not include
|
||||
# this key to get the default behavior.
|
||||
NvmeRetryCount 4
|
||||
|
||||
[TargetNodeX]
|
||||
# other TargetNode parameters go here (TargetName, Mapping, etc.)
|
||||
# nvme with the following format: NvmeXnY, where X = the controller ID
|
||||
# and Y = the namespace ID
|
||||
# Note: NVMe namespace IDs always start at 1, not 0 - and most
|
||||
# controllers have only 1 namespace.
|
||||
LUN0 Nvme0n1
|
||||
~~~
|
||||
|
||||
You should make a copy of the example configuration file, modify it to suit your environment, and
|
||||
then run the iscsi_tgt application and pass it the configuration file using the -c option. Right now,
|
||||
the target requires elevated privileges (root) to run.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user