diff --git a/CHANGELOG.md b/CHANGELOG.md index e42a64227..904cf1cb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -112,6 +112,10 @@ applications when a qpair is failed. This list of functions includes: These functions now return -ENXIO when the qpair or controller on which they operate is failed. +EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux +kernel for controller as well as for namespace and process ioctl requests as usual +from linux environment. + ### nvmf The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` diff --git a/doc/bdev.md b/doc/bdev.md index 8c2af3d7b..2552d6f5b 100644 --- a/doc/bdev.md +++ b/doc/bdev.md @@ -411,6 +411,23 @@ To remove an NVMe controller use the bdev_nvme_detach_controller command. This command will remove NVMe bdev named Nvme0. +## NVMe bdev character device {#bdev_config_nvme_cuse} + +This feature is considered as experimental. + +Example commands + +`rpc.py bdev_nvme_cuse_register -n Nvme0 -p spdk/nvme0` + +This command will register /dev/spdk/nvme0 character device associated with Nvme0 +controller. If there are namespaces created on Nvme0 controller, for each namespace +device /dev/spdk/nvme0nX is created. + +Cuse devices are removed from system, when NVMe controller is detached or unregistered +with command: + +`rpc.py bdev_nvme_cuse_unregister -n Nvme0` + # Logical volumes {#bdev_ug_logical_volumes} The Logical Volumes library is a flexible storage space management system. It allows diff --git a/doc/img/nvme_cuse.svg b/doc/img/nvme_cuse.svg new file mode 100644 index 000000000..bed843402 --- /dev/null +++ b/doc/img/nvme_cuse.svg @@ -0,0 +1,124 @@ + + + NVMe CUSE + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + NVMe CUSE + + + + + + /dev/spdk/nvme0 + /dev/spdk/nvme0n1 + + + + + + + + + + + + + + + + + + + + + + + + io_msg queue + CUSE threads + SPDK threads + + + + + + + + + + + + + + + + + NVMe + CUSE ctrlr + CUSE ns + ioctl pthread + ioctl pthread + + + + + + Kernel + CUSE + io poller + + + + + + + + io execute + fn(arg) + nvme_io_msg send() + nvme_io_msg send() + spdk_nvme_io_msg process() + + + diff --git a/doc/jsonrpc.md b/doc/jsonrpc.md index c01dcd8f3..c81a26b3f 100644 --- a/doc/jsonrpc.md +++ b/doc/jsonrpc.md @@ -1627,6 +1627,80 @@ Example response: } ~~~ +## bdev_nvme_cuse_register {#rpc_bdev_nvme_cuse_register} + +Register CUSE device on NVMe controller. +This feature is considered as experimental. + +### Parameters + +Name | Optional | Type | Description +----------------------- | -------- | ----------- | ----------- +name | Required | string | Name of the NVMe controller +dev_path | Required | string | Path to the CUSE controller device, e.g. spdk/nvme0 + +### Example + +Example request: + +~~~ +{ + "params": { + "dev_path": "spdk/nvme0", + "name": "Nvme0" + }, + "jsonrpc": "2.0", + "method": "bdev_nvme_cuse_register", + "id": 1 +} +~~~ + +Example response: + +~~~ +{ + "jsonrpc": "2.0", + "id": 1, + "result": true +} +~~~ + +## bdev_nvme_cuse_unregister {#rpc_bdev_nvme_cuse_unregister} + +Unregister CUSE device on NVMe controller. +This feature is considered as experimental. + +### Parameters + +Name | Optional | Type | Description +----------------------- | -------- | ----------- | ----------- +name | Required | string | Name of the NVMe controller + +### Example + +Example request: + +~~~ +{ + "params": { + "name": "Nvme0" + }, + "jsonrpc": "2.0", + "method": "bdev_nvme_cuse_unregister", + "id": 1 +} +~~~ + +Example response: + +~~~ +{ + "jsonrpc": "2.0", + "id": 1, + "result": true +} +~~~ + ## bdev_rbd_create {#rpc_bdev_rbd_create} Create @ref bdev_config_rbd bdev diff --git a/doc/nvme.md b/doc/nvme.md index c07b7be6d..90b5da0db 100644 --- a/doc/nvme.md +++ b/doc/nvme.md @@ -9,6 +9,7 @@ * @ref nvme_fabrics_host * @ref nvme_multi_process * @ref nvme_hotplug +* @ref nvme_cuse # Introduction {#nvme_intro} @@ -266,3 +267,31 @@ This means I/O in flight during a hot remove will complete with an appropriate e code and will not crash the application. @sa spdk_nvme_probe + +# NVMe Character Devices {#nvme_cuse} + +This feature is considered as experimental. + +![NVMe character devices processing diagram](nvme_cuse.svg) + +For each controller as well as namespace, character devices are created in the +locations: +~~~{.sh} + /dev/'dev_path' + /dev/'dev_path'nY + ... +~~~ + +Requests from CUSE are handled by pthreads when controller and namespaces are created. +Those pass the I/O or admin commands via a ring to a thread that processes them using +spdk_nvme_io_msg_process(). + +Ioctls that request information attained when attaching NVMe controller receive an +immediate response, without passing them through the ring. + +This interface reserves one qpair for sending down the I/O for each controller. + +## Enabling cuse support for NVMe + +Cuse support is disabled by default. To enable support for NVMe devices SPDK +must be compiled with "./configure --with-nvme-cuse".