From b94155c85344d04976e672f9ae689e7f0112bd79 Mon Sep 17 00:00:00 2001 From: Stephen Bates Date: Wed, 28 Feb 2018 07:13:32 -0700 Subject: [PATCH] docs: Add documentation for peer-2-peer In 35a331a9 we added a new API that enables peer-2-peer (P2P) copies between NVMe SSDs using Controller Memory Buffers (CMBs) that support DMA operations. Add documentation for the API, the example application (cmb_copy) and for P2P setup and operation. Note that this new API is currently marked experimental. Change-Id: Ifeedb86d5b72b3aa7a6803b87c4d4709c54108f8 Signed-off-by: Stephen Bates Reviewed-on: https://review.gerrithub.io/401961 Tested-by: SPDK Automated Test System Reviewed-by: Daniel Verkamp Reviewed-by: Jim Harris --- doc/Doxyfile | 1 + doc/index.md | 4 +++ doc/peer_2_peer.md | 62 +++++++++++++++++++++++++++++++++++++++++++++ include/spdk/nvme.h | 16 ++++++++++-- 4 files changed, 81 insertions(+), 2 deletions(-) create mode 100644 doc/peer_2_peer.md diff --git a/doc/Doxyfile b/doc/Doxyfile index e1bd84930..638fb521f 100644 --- a/doc/Doxyfile +++ b/doc/Doxyfile @@ -803,6 +803,7 @@ INPUT = ../include/spdk \ nvme-cli.md \ nvmf.md \ nvmf_tgt_pg.md \ + peer_2_peer.md \ ssd_internals.md \ userspace.md \ vagrant.md \ diff --git a/doc/index.md b/doc/index.md index 70ff76af3..20541f671 100644 --- a/doc/index.md +++ b/doc/index.md @@ -35,6 +35,10 @@ - @ref event - @ref blob +# Miscellaneous {#misc} + +- @ref peer_2_peer + # Modules {#modules} - @ref nvme diff --git a/doc/peer_2_peer.md b/doc/peer_2_peer.md new file mode 100644 index 000000000..3be7d8592 --- /dev/null +++ b/doc/peer_2_peer.md @@ -0,0 +1,62 @@ +# Peer-2-Peer DMAs {#peer_2_peer} + +Please note that the functionality discussed in this document is +currently tagged as experimental. + +# In this document {#p2p_toc} + +* @ref p2p_overview +* @ref p2p_nvme_api +* @ref p2p_cmb_copy +* @ref p2p_issues + +# Overview {#p2p_overview} + +Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI +End Point (EP) to another without using a system memory buffer. The +most obvious example of this from an SPDK perspective is using a NVMe +Controller Memory Buffer (CMB) to enable direct copies of data between +two NVMe SSDs. + +In this section of documentation we outline how to perform P2P +operations in SPDK and outline some of the issues that can occur when +performing P2P operations. + +# The P2P API for NVMe {#p2p_nvme_api} + +The functions that provide access to the NVMe CMBs for P2P +capabilities are given in the table below. + +Key Functions | Description +------------------------------------------- | ----------- +spdk_nvme_ctrlr_alloc_cmb_io_buffer() | @copybrief spdk_nvme_ctrlr_alloc_cmb_io_buffer() +spdk_nvme_ctrlr_free_cmb_io_buffer() | @copybrief spdk_nvme_ctrlr_free_cmb_io_buffer() + +# cmb_copy: An example P2P Application {#p2p_cmb_copy} + +Run the cmb_copy example application. + +~~~{.sh} +./examples/nvme/cmb_copy -r -1-0-1 -w -1-0-1 -c +~~~ +This should copy a single LBA (LBA 0) from namespace 1 on the read +NVMe SSD to LBA 0 on namespace 1 on the write SSD using the CMB as the +DMA buffer. + +# Issues with P2P {#p2p_issues} + +* In some systems when performing peer-2-peer DMAs between PCIe EPs + that are directly connected to the Root Complex (RC) the DMA may + fail or the performance may not be great. Basically your milage may + vary. It is recommended that you use a PCIe switch (such as those + provided by Broadcom or Microsemi) as that is know to provide good + performance. +* Even with a PCIe switch there may be occasions where peer-2-peer + DMAs fail to work. This is probaby due to PCIe Access Control + Services (ACS) being enabled by the BIOS and/or OS. You can disable + ACS using setpci or via out of tree kernel patches that can be found + on the internet. +* In more complex topologies involving several switches it may be + possible to construct multiple paths between EPs. This could lead to + TLP ordering problems. If you are working in these environments be + careful! diff --git a/include/spdk/nvme.h b/include/spdk/nvme.h index 1189afe50..12e5f525b 100644 --- a/include/spdk/nvme.h +++ b/include/spdk/nvme.h @@ -917,21 +917,33 @@ int spdk_nvme_ctrlr_update_firmware(struct spdk_nvme_ctrlr *ctrlr, void *payload struct spdk_nvme_status *completion_status); /** - * \brief Allocate an I/O buffer from the controller memory buffer. + * \brief Allocate an I/O buffer from the controller memory buffer (Experimental). * * \param ctrlr Controller from which to allocate memory buffer. * \param size Size of buffer to allocate in bytes. * * \return Pointer to controller memory buffer allocation, or NULL if allocation was not possible. + * + * This function allocates registered memory which belongs to the + * Controller Memory Buffer (CMB) of the specified NVMe + * controller. Note that the CMB has to support the WDS and RDS + * capabilities for the allocation to be successful. Also, due to + * vtophys contraints the CMB must be at least 4MiB in size. Free + * memory allocated with this function using + * spdk_nvme_ctrlr_free_cmb_io_buffer(). */ void *spdk_nvme_ctrlr_alloc_cmb_io_buffer(struct spdk_nvme_ctrlr *ctrlr, size_t size); /** - * \brief Free a controller memory I/O buffer. + * \brief Free a controller memory I/O buffer (Experimental). * * \param ctrlr Controller from which the buffer was allocated. * \param buf Buffer previously allocated by spdk_nvme_ctrlr_alloc_cmb_io_buffer(). * \param size Size of buf in bytes. + * + * Note this function is currently a NOP which is not a good thing and + * is one reason why this and spdk_nvme_ctrlr_alloc_cmb_io_buffer() + * are currently marked as experimental. */ void spdk_nvme_ctrlr_free_cmb_io_buffer(struct spdk_nvme_ctrlr *ctrlr, void *buf, size_t size);