Commit Graph

215 Commits

Author SHA1 Message Date
Ben Walker
6b1e4e732d Drop libpciaccess and switch to DPDK PCI
This patch also drops support for automatically unbinding
devices from the kernel - run scripts/setup.sh first.

Our generic pci interface is now hidden behind include/spdk/pci.h
and implemented in lib/util/pci.c. We no longer wrap the calls
in nvme_impl.h or ioat_impl.h. The implementation now only uses
DPDK and the libpciaccess dependency has been removed. If using
a version of DPDK earlier than 16.07, enumerating devices
by class code isn't available and only Intel SSDs will be
discovered. DPDK 16.07 adds enumeration by class code and all
NVMe devices will be correctly discovered.

Change-Id: I0e8bac36b5ca57df604a2b310c47342c67dc9f3c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-04 15:59:00 -07:00
Ben Walker
f4140ad023 nvme: Change the deallocate interface to generic dsm
Provide a convenience wrapper for general purpose dataset
management commands. The previous wrapper for deallocate
was difficult to use correctly and only for deallocate.

Note that the name is "dataset_management" as opposed to
"data_set_management" to match the NVMe specification.
It's questionable whether "dataset" is valid English, but
it is best to match the specification.

Change-Id: Ifc03d66dbabeabe8146968cf8a09f7ac3446ad68
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-04 14:36:09 -07:00
Ben Walker
db69cad95c nvme: Handle inactive but allocated namespaces
Namespaces can be allocated but inactive, which causes
the identify namespace command to fail. Handle this
case so that attaching to the controller does not fail.

Change-Id: I9d692f8e7841a9315a737b0a5e44d9b4e4484a13
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-03 13:55:56 -07:00
Ben Walker
b0e349a804 nvme: Use log library instead of nvme_printf
Change-Id: Ic9b2db9bff3a914b3e5021695287157f1e076f9f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-09-28 10:15:55 -07:00
GangCao
372942e569 nvme: move global request_mempool allocation into nvme_impl
The user no longer needs to create the request pool.

Change-Id: I83bb8948143d4cc961d232f9f30df3106d5e0eab
Signed-off-by: GangCao <gang.cao@intel.com>
2016-09-27 14:25:39 -07:00
GangCao
b9eedbf403 nvme: use nvme_malloc for calloc to share ctrlr->ioq among processes
Change-Id: I480fe43b8ac59c285999eedc2e4c2323c0401a8b
Signed-off-by: GangCao <gang.cao@intel.com>
2016-09-26 16:05:02 -07:00
GangCao
7faf9554dd nvme: correct the error message for mapping bar operation
Change-Id: Ie0def66b0d395937cfd1e2fec09841f02005e5ed
Signed-off-by: GangCao <gang.cao@intel.com>
2016-09-22 15:42:09 -07:00
Daniel Verkamp
5e9d859327 nvme: alloc buffer internally for non-I/O requests
Rather than forcing the NVMe library user to pass a specially-allocated
block of memory (e.g. rte_malloc() in the case of the default
nvme_impl.h), just make the NVMe library allocate a suitable buffer
itself and copy to/from the user buffer as needed.

The fast path I/O functions still require special rte_malloc()
allocations, since we don't want to add an allocation and copy to the
I/O critical path.

Change-Id: I7fe88c0ba60c859a33bbe95b7713f423c6bf1ea8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-09-13 12:47:46 -07:00
GangCao
20953e1ccd nvme: add the APIs to use the shared memory zone
Change-Id: I7faca95a15d320f3e2940c112b91d05a69797c90
Signed-off-by: GangCao <gang.cao@intel.com>
2016-09-06 13:18:16 -07:00
GangCao
dcd19bdb23 nvme: add the shared mutex init function for g_spdk_nvme_driver
Change-Id: Ib2a89beffb58004fdfd5a308feb6de2307dd5b81
Signed-off-by: GangCao <gang.cao@intel.com>
2016-08-26 14:34:21 -07:00
Changpeng Liu
d7120a3ee8 nvme: Fix max I/O size enforcement when no stripe size is specified
Intel DC P3*** NVMe devices specify a desired stripe size, which was
used for splitting I/O. Not all devices, however, specify a desired
stripe size (such as the Intel DC D3*** line), and for only these
devices there was a logic mistake that overwrote the maximum I/O
size with a 2MB default. This patch corrects that error.

Change-Id: I94b72a3a3dd1dfa18bd638daf7e01a592eb6ed17
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-08-25 08:51:16 -07:00
GangCao
c4afe2804b nvme: change global pointer g_nvme_driver to g_spdk_nvme_driver
Change-Id: I6fcd2d61ab896ef52f430a5cce1f2fd44b809725
Signed-off-by: GangCao <gang.cao@intel.com>
2016-08-24 15:46:09 -07:00
Daniel Verkamp
fd1ba17c16 nvme: match devices by class code with DPDK PCI
DPDK 16.07 introduced a new PCI ID field for matching by class code
instead of vendor/device ID. Use it to match all NVMe devices instead of
explicitly listing vendor and device ID pairs.

Change-Id: Ib2a5cc6833bf2b793d37d77caab97207f365df8f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-24 09:47:23 -07:00
Ben Walker
0606eaad1a No longer wrap assert()
assert is part of the C standard library and is available
on any platform we'd consider porting to. Don't put a
wrapper around it.

Change-Id: I0acfdd6a8a269d6c37df38fb7ddf4f1227630223
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-08-19 10:53:06 -07:00
Ben Walker
888014289c nvme: No longer abstract away pthread calls
pthreads are widely supported and are available on any
platform we currently foresee porting to. Use that API
instead of attempting to abstract it away to simplify
the code.

Change-Id: I822f9c10910020719e94cce6fca4e1600a2d9f2a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-08-19 10:53:06 -07:00
Tsuyoshi Uchida
0d2330c2df nvme: Add some status codes specified in NVM Express 1.2b (#37) 2016-08-19 08:52:31 -07:00
Changpeng Liu
dcf49aa018 nvme: disable the controller from generating INTx# interrupts
Since we bind the NVMe device to UIO driver to protect against native
NVMe driver, but for Admin queue, there are still INTx interrupts
exist, as all the completion for Admin queue will be processed in
user space, so we don't need INTx anymore.

Change-Id: Ife5b3e410ae95690ed0f3f9a2f2dfaf55a7797b5
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-08-17 09:08:33 -07:00
Tsuyoshi Uchida
81976ebd55 nvme: modify not to retry IOs on reset (#33)
When a controller reset is finished, all outstanding IOs and all queued
IOs which submitted before the reset are returned to the caller.
2016-08-15 10:01:02 -07:00
Daniel Verkamp
dd2e6164c9 nvme: replace rte_zmalloc() with rte_malloc() + memset
rte_zmalloc() is broken and does not actually return zeroed memory on at
least DPDK 16.07 on FreeBSD, so do it ourselves.

Change-Id: If8da93ead0b3911c8bca24aa27ed90dc00b8a9a4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-10 16:12:59 -07:00
GangCao
b724e2e5a6 nvme: change global g_nvme_driver object to a pointer
This is a step towards enabling sharing SPDK NVMe
device access from multiple processes using DPDK's
multi-process framework.

Change-Id: I57d5eec158b42addc1036bd2583596471a467a95
Signed-off-by: GangCao <gang.cao@intel.com>
2016-08-04 15:45:35 -07:00
Daniel Verkamp
336ba0d09c build: replace USE_PCIACCESS with config.h define
Drop the special-case preprocessor definition for PCI access library now
that config.h is available with an equivalent SPDK_CONFIG_PCIACCESS
define.

Change-Id: I4891d0f2fd7d3eea51b767df9e594555b36265ea
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-07-20 09:53:47 -07:00
Daniel Verkamp
77cb78fadf nvme: add enumeration of Identify command types
Change-Id: I2f8dc3de48c2e5dc06ccc9d007373e361bffcf63
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-07-19 13:31:06 -07:00
Daniel Verkamp
0cb9522781 build: include spdk.common.mk in lib Makefiles
Explicitly include spdk.common.mk at the top of all lib Makefiles so
that CONFIG options and other predefined variables are set.

Change-Id: I1e560c294fe8242602e45191a280f4295533ae44
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-07-18 16:17:40 -07:00
Ben Walker
c65e726acd nvme: Limit max completions in a single poll
For a single poll of the completion queue, if the user
submits I/O from within their completion callback and their
completion callback is particularly slow to execute, the loop
could potentially continue forever. To support this, we
need to limit the number of completions we'll process
in one batch.

Change-Id: If6bae47e52b36347dbe5622ace68c866ee88a0b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-07 17:05:17 -07:00
Daniel Verkamp
a464f1394e nvme: add API to get CAP and VS registers
This is intended to be used for examples/nvme/identify and similar
diagnostic utilities.

Change-Id: Ib2f941e9af7a3fb7555865ef253742e30ccad2b5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-07-07 15:32:20 -07:00
Daniel Verkamp
0e93df5c34 nvme: combine CAP register into one 64-bit union
Previously, we used cap_lo and cap_hi to represent the 32-bit halves of
the full CAP register.  However, it is simpler to keep them in a single
64-bit structure, and is no less efficient on 64-bit platforms.

Also name the NSSRS field from NVMe 1.2, which was previously reserved.

Change-Id: I1d5d9b0dccbb12373b4aed3db29c883881d43223
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-07-07 13:52:16 -07:00
Changpeng Liu
90c15f6863 nvme: remove duplicated header file include
Change-Id: I96b84ec19f8394da8311c452789e962b0255395b
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-07-07 11:17:43 +08:00
Daniel Verkamp
19fec6bb9c nvme: add remove callback to spdk_nvme_probe()
This will allow removal notifications to be propagated to the library
user (e.g. for hotplug).

The callback is currently unused, but this at least prepares the API for
the future hotplug support.

Based on a patch by Dave Jiang <dave.jiang@intel.com>

Change-Id: I20b1c2dbf5e084e0b45a7e51205aba4514ee9a95
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-06-28 14:21:58 -07:00
Daniel Verkamp
e2d3cc6502 nvme: replace rte_memcpy with specialized function
Use the knowledge that both the source and destination of
nvme_copy_command() are aligned to emit the aligned variants of the
SSE2/AVX mov instructions.

Change-Id: I0a7e32a3bb10b9a1920cd85691b79fa7172eecb3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-06-28 14:17:21 -07:00
Ben Walker
6b10df3576 nvme: Update nvme_spec.h to 1.2.1
This moves some definitions from nvmf_spec.h to
nvme_spec.h based on the latest publication.

Change-Id: I51b0abd16f7d034696239894aea5089f8ac70c40
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-06-24 09:18:14 -07:00
Daniel Verkamp
ae17467ecf nvme: check BAR map return code before using addr
Swap the order of checks in the failure check - if rc is not 0, addr may
be garbage.

Change-Id: I110710efd00397c777d59ac8b219ba3cc2156596
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-06-23 16:48:08 -07:00
GangCao
c65210d034 nvme: return error immediately when enabling controller failed
Change-Id: Id9cf6873cc831bd5099df49db95ef5073badf461
Signed-off-by: GangCao <gang.cao@intel.com>
2016-06-21 16:37:15 -04:00
Ziye Yang
7991eb1957 Create a utility and enable WRR arbitration mechanism
Change-Id: I2194863ae812ece72c17b78e003ccf7895b8a812
Signed-off-by: GangCao <gang.cao@intel.com>
2016-06-21 16:35:04 -07:00
Minfei Huang
f0b20026d7 nvme: Return negative errno for failure (#24)
The conventional rule for returning errno is negative, hence there is no
need to modify caller's code to adjust this NVMe library.

Signed-off-by: Minfei Huang <mnghuan@gmail.com>
Signed-off-by: Minfei Huang <minfei.hmf@alibaba-inc.com>
2016-06-14 15:19:10 -07:00
Daniel Verkamp
4957d2642a nvme: return NSID from spdk_nvme_ctrlr_create_ns()
Previously, there was no way to determine what namespace ID was assigned
when a namespace was created via the NVMe library interface.

Also drop the incorrect comment about calling
spdk_nvme_ctrlr_process_admin_completions(), since
spdk_nvme_ctrlr_create_ns() checks the admin queue internally.

Change-Id: If90a6e9fc773aefa220ebbf6effc2d033c9f20cc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-18 17:06:09 -07:00
Daniel Verkamp
790b99e863 nvme: reorder qprio in spdk_nvme_qpair for packing
This lets spdk_nvme_qpair fit in 128 bytes exactly.

Change-Id: I7c42582f22ece72a7f1d651468e63d4fe05babd6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-17 14:17:20 -07:00
Daniel Verkamp
59ae518add nvme: set qpair fields to NULL during destruct
Avoid potential double free cases.

This fixes a clang warning during scan-build.

Change-Id: I487d6fcd485d1f8ebb96b6f8cb54511628461f39
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-16 10:52:51 -07:00
Daniel Verkamp
dfc4b030b1 nvme: assert that outstanding_tr request is valid
All trackers in outstanding_tr should have a non-NULL request.  Add an
assert to verify this.

Fixes a clang warning during scan-build.

Change-Id: I0ac4d2bad17449f684808cbb98777627d890b65b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-16 10:45:40 -07:00
Daniel Verkamp
4a36aabd0c nvme_qpair: add sanity-check asserts to print_command
Document the requirement that qpair and cmd can't be NULL.

This placates clang, which previously generated a warning
during scan-build.

Change-Id: Ic2d5e808faee0028c890ce1312444fb3dc95f223
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-16 10:39:35 -07:00
Daniel Verkamp
cd8e9833f9 nvme: remove unused CMB_SQ_SUPPORTED flag
The user can determine whether submission queues will be placed in the
controller memory buffer by checking the controller options use_cmb_sqs
flag in the attach callback.

Change-Id: I8a925ef99a48665a0e2ffaa90d9ff2b79b90b2fa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-12 13:20:10 -07:00
Changpeng Liu
ca3d1c5b45 spdk: add controller memory buffer support in driver
The D3700/D3600 series support Controller Memory Buffer(CMB) feature,
CMB is available for holding submission queues, for those controllers
which can support submission queues in CMB, user can set the option
whether to enable it or not.

Change-Id: I8b0dc9e28dd6f5bb01bee99a532087212c04e492
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-05-13 08:14:10 +08:00
Cunyin Chang
f2168e1d73 nvme: Add firmware upgrade interface and unit test suite
Change-Id: If66e5f97f6793df0388629fab7c3d0e9f9d5eb67
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2016-05-11 10:18:33 -07:00
Daniel Verkamp
cd48a01fcb build: wrap $(CURDIR) relative paths in $(abspath)
Resolve relative paths before using them to clean up command lines.

This should also help shorten the overall command line length that gets
embedded in the binary and used when locating the executable from a
coredump.

Change-Id: Ibff9849ede198bb04313496c8b7131485ffaf14f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-09 13:56:07 -07:00
Daniel Verkamp
3a94688d8e build: add spdk.lib.mk for common library rules
Change-Id: I0547554c010b0eaa6e98c843e7d9eb78c90f76d5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-09 11:24:58 -07:00
Cunyin Chang
10f0f4b0e9 spdk: Add Intel specific log page.
This patch add support for Intel specific log pages :
marketing description page.

Change-Id: I87bccb2af286279598c9dd3c870094b384a0d2f7
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2016-05-05 13:39:14 +08:00
Daniel Verkamp
8eafb6546c doc: add Doxygen file marker to all public headers
For existing \file markers, move them to the top of the header and tweak
the wording for consistency.

Change-Id: Icce748effe4dbe97d79a8c87d31caf0ee5797058
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-05-04 11:17:17 -07:00
Changpeng Liu
784182ed47 SPDK: Add end-to-end data protection support in driver
For those controllers which can support end-to-end data protection
feature, add the support in the driver layer.

Change-Id: Ifac3dd89dec9860773c850416a6116113a6ce22a
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-04-28 15:46:13 -07:00
Liang Yan
b2db4f94ec nvme/test: Add hw sgl request unit test
Change-Id: I1364a6634b9be25ff647eee56c91e312ae3d004e
Signed-off-by: Liang Yan <liangx.yan@intel.com>
2016-04-28 14:26:24 +08:00
Li Peng
33b46ee9a8 nvme: fix typos
Signed-off-by: Li Peng <lip@dtdream.com>
2016-04-26 15:49:42 +08:00
Daniel Verkamp
e175af8879 nvme: add workaround for VFIO initialization issue
When using DPDK PCI support via VFIO, the PCI device is reset
immediately before calling the PCI driver's init function.  In some
cases, the device seems to not be ready to handle MMIO accesses right
away.  Until the cause of this issue is fully understood, add a 500 ms
sleep as a workaround.

Change-Id: Ic893080a6f34d57eee80df3e6aa68c220c08df3e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-04-14 14:24:27 -07:00
Daniel Verkamp
cf368b5e5b nvme: add sanity check to nvme_free_request()
All children must be removed from a parent request before the parent is
freed.

Change-Id: I073ff0e9c5bcdd6181d90b918bfe4cce054f6c0b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-04-08 16:51:15 -07:00
Daniel Verkamp
73de52e9ae nvme: add sanity checks to remove_child
Change-Id: Id212700459901c3f7bb6cd2f63ba1b88a7417ac2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-04-08 16:51:15 -07:00
Daniel Verkamp
2cf675bb29 nvme: rename remove_child_request for consistency
Rename nvme_remove_child_request() to nvme_request_remove_child() and
move it next to nvme_request_add_child() to make the symmetry clear.

Change-Id: I78747c44ab3db1a656b33555a45f634dc5a55b31
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-04-08 16:51:15 -07:00
Daniel Verkamp
1f19be6515 nvme: destroy I/O queues before shutdown
The NVMe specification recommends destroying all I/O submission and
completion queues before setting CC.SHN.

Change-Id: Iad71dd3fe03d897858034f3ca6ee02e0c55cc2b0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-04-08 16:47:49 -07:00
Daniel Verkamp
2f7d4bee02 nvme: do not set EN=0 during shutdown
The NVMe specification recommends that orderly shutdown should just
write CC.SHN while the controller is still enabled rather than writing
CC.EN = 0 first.

This also allows removal of the now-unused nvme_ctrlr_disable() and
nvme_ctrlr_wait_for_ready() functions.

Change-Id: I4702ffda153f218ebb8ed92f0e36144b7ceded93
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-04-08 16:47:43 -07:00
Daniel Verkamp
87a3244f97 nvme: handle CC.EN = 0 && CSTS.RDY = 1 on startup
This can happen if the controller is still resetting as the SPDK NVMe
driver takes control.

Change-Id: I263ae8f2e7b271e0448450557452a115c90c4fb6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-04-08 16:47:25 -07:00
Ziye Yang
20f92ad45a SPDK: Handle the memory leak issue for nvme_request free
This patch is used to handle the memory leak issue when
a parent nvme_request is free. In our current code,
we did not free the nvme_request allocated by the children
in the exceptional case.

Change-Id: Iabd1f1c3594af60c38e74e3d96c14f78d1aa1aed
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2016-04-08 09:40:07 -07:00
Ziye Yang
deb90a93de SPDK: add nvme_remove_child_request helper function
This patch is used to add a nvme_request remove child
helpler function

Change-Id: I1e5bb228d53333ca3601f4ae30fcd801ea39e532
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2016-04-08 09:36:57 -07:00
Daniel Verkamp
2c4476560b nvme: simplify SGL length calculation
Drop the "data_transfered" variable and just update length, since length
was not used otherwise after this point in the loop.

Change-Id: Icd2991e4e85de7e8c951ba14c441434e871ea4ef
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-30 10:45:08 -07:00
Daniel Verkamp
3cec6c846b nvme: clarify single SGL descriptor case
If only one descriptor is needed, there is a special case in the spec for
SGL1 using the Data Block descriptor type.  Add a comment to make it
clear what is going on.

Also tweak the SGL1 setup to copy from the first SGL descriptor list
element instead of relying on the last value from the loop above, since
that could be easily broken by accident.

Change-Id: I49ef97fe5bf18d2bf1d86b4310a7d3abdfd03e57
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-30 10:45:08 -07:00
Daniel Verkamp
1524f2935c nvme: remove unnecessary cast
tr->u.sgl is already a struct spdk_nvme_sgl_descriptor pointer.

Change-Id: Ie2c8c052fc28e6369d1d095b8d566acae47975d1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-30 10:45:08 -07:00
Daniel Verkamp
bf6477ccab nvme: remove dead code for zero-length SGL request
_nvme_qpair_build_hw_sgl_request() will only be called for payload_size
!= 0, so every SGL will have at least one segment.  Drop the 'else' that
was handling nseg == 0, and add an assert to document the payload_size
requirement.

Change-Id: I48e2a862a7657ba85605c0d35c0b65dfac072167
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-30 10:45:08 -07:00
Daniel Verkamp
69f9796482 nvme: move assert out of SGL loop
The assert is checking a variable that is invariant within the loop, so
move the assert up to the top of the function.

Change-Id: Iee7eea1736bc7f953665feb390c3d6340dbeffbc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-30 10:44:34 -07:00
Daniel Verkamp
149ee30ee8 doc/nvme: move pages to separate text files
This makes it easier to find the larger doc comments that produce separate
pages.

It also allows removing the lib/nvme directory from the Doxyfile, so
only the public API headers are used to generate documentation.

Change-Id: I8c46edb8067a91dda5b23fb0864efd3dd8aaeba5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-29 10:49:06 -07:00
Daniel Verkamp
7d74bd7072 nvme: remove duplicated doc comment
spdk_nvme_qpair_process_completions() is already documented in
spdk/nvme.h, so merge the doc comment from nvme_qpair.c into the public
header.

Change-Id: Id7722d99d209852ee64286e0a3fa127b863e10aa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-29 10:27:55 -07:00
Daniel Verkamp
90095a79fe nvme: enforce minimum and maximum I/O queues
Don't allow the user to request more than the valid maximum number of
I/O queues (65535) or 0 I/O queues, since this can't be encoded.

Change-Id: I2d6e0bba03476085842bad683b273cdf9d6e6d5e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-28 15:34:29 -07:00
Daniel Verkamp
4671dbd53f nvme: add compile-time assert for SGL alignment
Per the NVMe spec, SGL segments must be Qword (8-byte) aligned.  Add a
static assert to make sure this is true for the sgl member of struct
nvme_tracker (assuming the whole nvme_tracker is at least 8-byte aligned).

Change-Id: I827aa40b56de648d83f524a4f1e79c3202b676be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-23 10:39:30 -07:00
Cunyin Chang
db3fda2e68 nvme: handle NULL ioq array in nvme_ctrlr_destruct()
Fix a potential segmentation fault issue.

Change-Id: I39d2cd1850265ca0dfa987995011563cadeb5bb5
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2016-03-23 09:37:02 -07:00
Cunyin Chang
eae688576a spdk: Add nvme format interface and unit tests.
Change-Id: Ie0506debf547a5fc011e116421387a5adb7abf0e
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2016-03-17 13:37:20 -07:00
Daniel Verkamp
eb555b139e nvme: add return code to nvme_qpair_submit_request
If the controller is failed, attempting to submit additional I/O is
futile - it will be immediately failed using the completion callback,
which can result in infinite recursion if the application code resubmits
I/Os on failure.

Instead, provide a way for request submission to indicate failure, and
use it to exit early if the controller is failed; this can only happen
when a reset failed (timed out).

If a request is submitted directly by the user when the controller has
failed, we can return an error code directly.  For the case where I/O
was queued and is being resubmitted after a reset, we still need to call
the completion handler via _nvme_fail_request_ctrlr_failed().

Change-Id: I9e144328d524b25db2acf48e923b584746e8d0b6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-16 12:16:47 -07:00
Daniel Verkamp
4ad99808f2 nvme: allow user to override controller defaults
Provide a new structure, spdk_nvme_ctrlr_opts, to let the user modify
the default controller initialization options during probe/attach.

Currently, only the number of queue pairs can be modified in this way;
other options will be added later.

Change-Id: Ie27b9429291d93a9353c0d820f0ad467d3b0e7cb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-16 08:14:15 -07:00
Daniel Verkamp
1dd7473078 nvme: create tracker pool in a single allocation
Replace the previous code that allocated each tracker individually with
one large allocation per queue pair.

struct nvme_tracker is now explicitly padded to reach exactly 4096 bytes
to allow normal array indexing to work correctly while maintaining the
alignment requirement that ensures each tracker's PRP list does not
cross a page boundary.

This also allows removal of the act_tr array, since the tr array can be
indexed directly now, and each tracker can store its own active state.

Change-Id: Ia7c51735b96594d12f7f478cefcc4aedc84207ad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-15 09:20:25 -07:00
Daniel Verkamp
3272320c73 nvme: make I/O queue allocation explicit
The previous method for registering I/O queues did not allow the user
to specify queue priority for weighted round robin arbitration, and it
limited the application to one queue per controller per thread.

Change the API to require explicit allocation of each queue for each
controller using the new function spdk_nvme_ctrlr_alloc_io_qpair().

Each function that submits a command on an I/O queue now takes an
explicit qpair parameter rather than implicitly using the thread-local
queue.

This also allows the application to allocate different numbers of
threads per controller; previously, the number of queues was capped at
the smallest value supported by any attached controller.

Weighted round robin arbitration is not supported yet; additional
changes to the controller startup process are required to enable
alternate arbitration methods.

Change-Id: Ia33be1050a6953bc5a3cca9284aefcd95b01116e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-14 16:00:54 -07:00
Daniel Verkamp
9f67a07fdc nvme: use full nvme_tracker space for PRP list
Since the nvme_tracker struct was extended to allow space for 253 SGL
descriptors at 16 bytes each, we can use the same amount of space in the
other branch of the union to store 506 PRP list entries at 8 bytes each.

This increases the maximum supported I/O size for PRP-only devices from
128 KB to slightly under 2 MB.

Change-Id: I2b9905be41343ff360b4cdaccca87ea6f753e89c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-14 08:25:57 -07:00
Daniel Verkamp
68ef53d128 nvme: make nvme_tracker fit in a single page
Also add a compile-time assert to make sure this doesn't accidentally
break again in the future.

Change-Id: I4d18cfbf21392291e1bdd76eff055429009d28d6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-14 08:25:57 -07:00
Daniel Verkamp
cc0b900e29 nvme: don't set interrupt vector in create_io_cq
nvme_ctrlr_cmd_create_io_cq() was using the queue ID as the
IV (Interrupt Vector) field in the Create I/O Completion Queue command.

Since the SPDK NVMe driver does not enable interrupts, this is
misleading at best.

Change-Id: I3ea53701fdb9f21d9dc8d8fe20ccf2833b76cfbf
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-14 08:04:44 -07:00
Cunyin Chang
30089f8939 nvme manage: Add interface for format progress indicator.
This patch just implement the feature of format progress indicator.
the NVMe available does't support FPI currently.

Change-Id: Ie937591fb1720d8a062354322aabcc95ff14b2d3
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2016-03-10 09:02:06 +08:00
Daniel Verkamp
17005b5756 nvme: remove nvme_request::timeout field
This field is write-only in the current code; the NVMe library does
not track timeouts on requests.

Change-Id: I50e53bb3c299bf16912c48be8aad3eec829154af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-09 16:56:05 -07:00
Changpeng Liu
2374599671 spdk: fix wrong length parameter for large IO when uses hardware SGL
When I/O spans a stripe boundary, the driver splits the request into
multiple requests, so for 1 segment memory larger than the stripe
size, we also need to split the segment memory.

Change-Id: I22ea5734d7066865a57a3c90fe18d5f76f373f1d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-03-09 15:22:08 -07:00
Changpeng Liu
eb9ef5cc2b nvme: Add SGL support in NVMe driver
For those NVMe controllers which can support SGL feature in
firmware, we will use SGL for scattered payloads.

Change-Id: If688e6494ed62e8cba1d55fc6372c6e162cc09c3
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-03-04 09:36:40 -07:00
Cunyin Chang
9841610855 spdk: Add namespace management interface and unit tests.
Change-Id: I9d203bf7532d50b1f8c8ca50c50df09ded8b5256
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2016-03-04 09:52:30 +08:00
Daniel Verkamp
e20639540e nvme_spec: declare CSTS with the correct type
nvme_spec.h already has a structure with the correct bitfields for the
CSTS register, so use it in struct spdk_nvme_registers.

Change-Id: Id0663aee2611fb5195f9012a3176799e32701bb0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-02 08:52:35 -07:00
Daniel Verkamp
345d9a4da3 nvme: add spdk_ prefix to nvme_qpair_process_completions()
This will be exposed in the public API.  This rename is in a separate
commit to ease review.

Change-Id: I1b7fef36f85265db27935ac4d22ceef3c7282502
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-01 14:42:04 -07:00
Daniel Verkamp
df26ab0583 nvme: add priority field to qpair
Set up the infrastructure for creating I/O submission queues with
variable queue priority (QPRIO in Create I/O SQ command).

Currently, this is unused, since we always use the default arbitration
method (round robin), but it will allow reinitializing submission queues
with the correct priority once weighted round robin is supported.

Change-Id: I425003879e624cfcc9687bdc495b5c1726b5a8af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-01 14:42:03 -07:00
Daniel Verkamp
0b2848ffb8 nvme: remove nvme_ctrlr_post_failed_request declaration
This function no longer exists and was not part of the public API.

Change-Id: I94fd066b63e812367687d11bc00aa11ab88d4671
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-01 14:42:03 -07:00
Daniel Verkamp
89bf6d5ce1 nvme: add error checking for internal ctrlr_cmds
Many of the internal controller initialization functions did not check
for allocation failure; add return codes and check them where
applicable.

Change-Id: Id1b33bb06fca84035369d8b7ecd4c36b8ba7134c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-01 14:42:03 -07:00
Daniel Verkamp
5ee7a5df37 nvme: add spdk_nvme_ns_is_active() function
This function returns true if the namespace is active or false if it is
inactive (e.g. no namespace has been attached to the specified namespace
ID yet).

Also use the new function to add checks in the examples and tests where
applicable.

Change-Id: I35465b315ae1a1677c5a82191ad9b1da1c216d50
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-03-01 14:40:44 -07:00
Daniel Verkamp
6eb18e2f3d nvme: rename nvme_qpair to spdk_nvme_qpair
Prepare for qpair to be exposed as part of the public API.

Change-Id: Ia63e863e95554adceeade20c829f12fe346375d5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-29 10:11:35 -07:00
Daniel Verkamp
20abbe8abe nvme: perform resets in parallel during attach
When multiple NVMe controllers are being initialized during
spdk_nvme_probe(), we can overlap the hardware resets of all controllers
to improve startup time.

Rewrite the initialization sequence as a polling function,
nvme_ctrlr_process_init(), that maintains a per-controller state machine
to determine which initialization step is underway.  Each step also has
a timeout to ensure the process will terminate if the hardware is hung.

Currently, only the hardware reset (toggling of CC.EN and waiting for
CSTS.RDY) is done in parallel; the rest of initialization is done
sequentially in nvme_ctrlr_start() as before.  These steps could also be
parallelized in a similar framework if measurements indicate that they
take a significant amount of time.

Change-Id: I02ce5863f1b5c13ad65ccd8be571085528d98bd5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-25 13:25:59 -07:00
Daniel Verkamp
a62b194f1b nvme: add timestamp counter interface to nvme_impl
Change-Id: Ic652163e4f5944c1516eaf58615f7eabcbe34a7a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-24 21:17:12 -07:00
Daniel Verkamp
d82473cd7d nvme: remove cc.en check in ctrlr_wait_for_ready
This check was dead code, since both places that called
nvme_ctrlr_wait_for_ready() could only ever have cc.en = 1.

Remove the original nvme_ctrlr_wait_for_ready() wrapper and rename
_nvme_ctrlr_wait_for_ready() without the underscore to replace it.

Change-Id: I6c9aa6a5b93606fb89d168c23f6735fcf3a84eaa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-24 21:17:11 -07:00
Daniel Verkamp
a15573c47c nvme: don't reset already-disabled controllers
In nvme_ctrlr_hw_reset(), if we encounter a controller whose CC.EN bit
is already 0 (controller is disabled), the previous code would enable
the controller just so that it could be disabled to get a full reset
(transition from CC.EN = 1 to CC.EN = 0).  However, it is a safe
assumption that if CC.EN is already 0, the controller has just been
reset, so we don't need to reset it again.

This saves a significant amount of time (2+ seconds per controller with
Intel SSD DC P3700) during initialization for devices that were disabled
on startup.

Change-Id: I552b1f0f185a84a8a0ce57a93b012d9d5fe096f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-23 16:53:09 -07:00
Daniel Verkamp
5dafa4940f nvme.h: include stdbool.h and stdint.h
Explicitly include system headers for types that are used in public
headers.

These were being pulled in by example code, so SPDK itself would build,
but other apps that did not include stdbool.h would fail to compile when
including spdk/nvme.h.

Also include nvme.h first in nvme_internal.h so this case gets tested
during normal compilation.

Change-Id: I8ed0fc4e0dcf71551738c461b4b825cc2ee1d233
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-22 09:57:11 -07:00
Daniel Verkamp
093d5479f9 spdk_nvme_ns_cmd_reservation_report: fix request allocation
payload here is a pointer to the buffer, not a struct nvme_payload.

Use nvme_allocate_request_contig() and pass the length in bytes rather
than dwords.

Change-Id: Idbbb3614b1d69148fe041d26e0c148bd9ce53724
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-19 15:24:09 -07:00
Daniel Verkamp
639fee5f13 nvme: remove initialization of nvme_reqest::parent
The parent field is in the cache line of nvme_request that is only
supposed to be accessed for split (child) I/Os.  All accesses to parent
are done from child-specific calls now, so it does not need to be
initialized in the common case of a non-split I/O.
nvme_request_add_child() will set parent when splitting occurs.

Change-Id: Ib86c16ba1ea2ce32f62079831101da2a099047af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-18 15:15:36 -07:00
Daniel Verkamp
7c34c2005d nvme: store SGL callback in nvme_payload
Allows simplification of _nvme_qpair_build_sgl_request(), which does not
need to know whether a request is a child or not.

This also removes a read of req->parent for non-split I/Os; the parent
field is in the section of nvme_request that is only intended to be
initialized for split I/Os, which should be detected by looking at
num_children.

Additionally, this fixes a potential problem if requests were nested
more than one level deep (e.g. req->parent was not the original user
request).

Change-Id: I3ea1dc134bbb1e3b8c6b5a479f5d760bd97ea848
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-18 15:07:18 -07:00
Andrey Kuzmin
5bb66add46 Include rte_config.h before other DPDK headers
Building the tip of the spdk master against the dpdk-2.2.0 fails with
inappropriate RTE_CACHE_LINE_SIZE error. The simple reversal of the RTE
include file order below fixed it for me.

Change-Id: I8782b7ee21d7f185e6e678f874fbdab9403117a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-16 14:44:41 -07:00
Ziye Yang
accf229829 SPDK: remove the duplicated code in ioat/nvme_impl.h
This patch is used to remove the duplicated code.
As we found the structure in "ifdef and else" are same.

Change-Id: I1717ce3dcc14134ac59c165d801e5e811b987be5
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2016-02-16 14:40:12 -07:00
Daniel Verkamp
6ce73aa6e7 nvme: add spdk_ prefix
Shorten commonly-used names:
controller -> ctrlr
namespace -> ns

Change-Id: I64f0ce7c65385bab0283f8a8341a3447792b3312
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-10 11:27:45 -07:00
Daniel Verkamp
ad35d6cd86 nvme_spec: add spdk_ prefixes
Use shorter names for commonly-used objects:
namespace -> ns
controller -> ctrlr
command -> cmd
completion -> cpl

Change-Id: I97d192546b35a6aeb76ad3a709f65631502cde71
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-02-09 11:06:48 -07:00