Rename all functions with a spdk_ prefix, and provide enough of an API
to avoid apps needing to #include <pciaccess.h>.
The opaque type used in the public API for a PCI device is now
struct spdk_pci_device *.
Change-Id: I1e7a09bbc5328c624bec8cf5c8a69ab0ea8e8254
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The new probing API will find all NVMe devices on the system and ask the
caller whether to attach to each one. The caller will then receive a
callback once each controller has finished initializing and has been
attached to the driver.
This will enable cleanup of the PCI abstraction layer (enabling us to
use DPDK PCI functionality) as well as allowing future work on parallel
NVMe controller startup and PCIe hotplug support.
Change-Id: I3cdde7bfab0bc0bea1993dd549b9b0e8d36db9be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch adds Intel NVMe device list and overrides the
supported log pages according to the quirk list.
In particular, the READ_CMD_LATENCY and WRITE_CMD_LATENCY pages are
supported on Intel DC P3x00 devices despite not being listed in the
Intel vendor-specific log page directory.
Change-Id: I3a2b6a5fa142c6e9c93567df65e85980bd3c7cc0
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Also add a space between Copyright and (c).
The copyright year can be determined using git metadata.
Also remove the duplicated "All rights reserved." - every instance of
this line already has a corresponding "All rights reserved" immediately
below it, except for examples/ioat/kperf/kmod/dma_perf.c, where I have
added it manually.
Performed using this command:
git ls-files | xargs sed -i -e 's/Copyright(c) \(.*\) Intel Corporation. All rights reserved./Copyright (c) Intel Corporation./'
Change-Id: I3779f404966800709024eb1eb66a50068af2716c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
1 Add new API nvme_ctrlr_is_feature_supported().
2 Add unit test for new API.
Change-Id: Ia6d8710755c3b13984fca9d56700efe043be1402
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
This cleans up the I/O splitting code somewhat.
It also moves the SGL payload function pointers up into the hot cache
section of struct nvme_request without pushing the other important
members past the cacheline boundary (because payload is now a union).
Change-Id: I14a5c24f579d57bb84d845147d03aa53bb4bb209
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Support for the Force Unit Access and Limited Retry
bits on reads and writes.
Change-Id: I9860848358377d63a967a4ba6ee9c061faf284d4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Request allocation may fail, so we need a way to indicate failure to the
caller.
Change-Id: I278c3f42e4d2fa1902bb0ab33ad3bf7c7007fd0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
1 Add supported log pages data structure.
2 Bulid up supported log pages when NVME start.
3 Provide unified API for getting log pages.
3 Unit test suit optimization base on above modification.
Change-Id: I03cdb93f5c94e6897510d7f19bc7d9f4e70f9222
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
nvme_ctrlr_process_io_completions() and
nvme_ctrlr_process_admin_completions() now return the number of
completions processed.
This also adds the possibility of returning an error from the
process_*_completions functions (currently unused, but this at least
gets the API ready in case error conditions are added later).
Change-Id: I1b32ee4f2f3c1c474d646fa2d6b8b7bbb769785f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make sure allocation succeed in the unit tests.
Change-Id: I1c9dd64c256d529f2e40be49448d56359969a6f5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The normal CU_ASSERT_FATAL macro calls a function that is not marked as
noreturn, so static analyzers (e.g. scan-buid) can't figure out that
fatal asserts are really fatal.
Add a wrapper macro that calls abort(), which the analyzer can determine
does not return.
Change-Id: I0c087bf9c8d3c272bf88120caa70e87dab6d9546
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The deallocate command's num_ranges have changed to unit16_t.
Update the nvme_ns_cmd_deallocate unit test for the change.
Change-Id: I43b8637bbb953b0e56c39998e1e6682a54304a8f
Signed-off-by: Liang Yan <liangx.yan@intel.com>
nvme_qpair_submit_tracker() and nvme_qpair_manual_complete_request() are
only used from within nvme_qpair.c, so they can be static.
nvme_qpair_submit_tracker() is moved up to avoid needing a declaration
(no other code change).
nvme_ctrlr_hw_reset() is only used from within nvme_ctrlr.c, so it can
be static.
Change-Id: I9a7953d7baaec76e875dd535daf557ea24bef801
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Most devices today support far fewer than 1024, but this is a
more reasonable default upper limit than the spec-defined 64K.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia8a6d80c3a5aa181f27c8354758c6ca468013d92
nvme_ctrlr_process_io_completions() now takes a second parameter,
max_completions, to let the user limit the number of I/Os completed on
each poll.
If there are many I/Os waiting to be completed, the
nvme_ctrlr_process_io_completions() function could run for a long time
before returning control to the user, so the max_completions parameter
lets the user have more control of latency.
Change-Id: I3173059d94ec1cc5dbb636fc0ffd3dc09f3bfe4b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Set SPDK_ROOT_DIR explicitly in each Makefile so that make from a
subdirectory will work (assuming all dependencies from the upper
directory have already been built). This allows partial rebuilds of the
source tree, as well as building the unit tests without requiring DPDK.
Change-Id: I3f65b805d490b40ff5ec53cceb61df542ce814f1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This catches functions with empty parameter lists, which are not the
same as void parameter lists in C, as well as other (less-likely)
K&R-style parameter lists.
Change-Id: I18b09e01a60b3669d1cc5d8d96eec20cd872c497
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This helps weed out functions that should be static, functions that are
not declared in public header files, and .c files that don't include
their .h interface headers.
Change-Id: Ie39f83ad4b320847e4a938bd1d4d0b4fa21c2ffa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Replace {0} initializers with {}, since it is equivalent and simpler
(and avoids the warning).
Change-Id: Id2014a352e2c627598cd3b4f1ca04dba35f2dc3a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fix all of the uses of __thread so they are at the beginning (similar to
e.g. static).
Don't actually enable -Wold-style-declaration, since clang doesn't
understand that.
Change-Id: I0dcbb758143eab90fc978334c8f256c6602cc4cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename the nvme_free_request macro to nvme_dealloc_request to match
nvme_alloc_request and add a wrapper function to nvme.c so that the
macro contents are only expanded once.
The DPDK nvme_impl.h uses rte_mempool_put(), which generates a large
amount of code inline. Moving this macro expansion to a wrapper
function avoids inlining it in the multiple places nvme_free_request()
gets called, most of which are error handling cases that are not in the
hot I/O path.
Change-Id: I64ea9c39ba47e26672eee8d5058f1489e07eee5b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The I/O splitting tests don't pass the I/Os through the normal
completion path, so we need to free the children ourselves.
Fixes all Valgrind warnings for nvme_ns_cmd_ut.
Change-Id: Iaf7d9f7f4cab71428a0123ee30d0f6042001e423
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_alloc_request() does not zero out the request (this is
intentional, since the real implementation uses a mempool where requests
get reused). Add nvme_allocate_request() wrapper that initializes the
request correctly. This fixes all uses of uninitialized memory caught
by Valgrind in nvme_qpair_ut. This test was also failing in practice on
FreeBSD due to non-zero buffers returned from malloc().
Change-Id: I2d6ea29289bd4887aaa9120fba6bce10088e6917
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The default version of nvme_impl.h was cleaned up to release the pthread
mutex attr on failure cases, but the unit test version did not receive
this update.
Change-Id: I899b7dc809393dc8e6fd24ed98e1d0a61ecf1c95
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Check the LBA and block count fields of the I/Os generated by the
splitting function to ensure they were split correctly.
Change-Id: I84abb1ac462fb7423d51a1be384fd1be68dfecae
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Instead of repeating the check for conditions that must be fulfilled to
continue the tests, just use CU_ASSERT_FATAL, which will abort the test
and return.
Change-Id: If617b286a587d9efb1ce57b90061634ed5bc7ae8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The current implementation of nvme_assert in the unit test nvme_impl.h
just prints the message and continues.
We should not be triggering assert conditions, even in the unit test
code, so make nvme_assert actually call assert(). This lets us catch
mistakes in the unit tests more easily.
Also fix the two unit tests that currently trigger an assert:
- The I/O splitting test in nvme_ns_cmd_ut was passing an invalid
combination of NULL payload with non-zero lba_count.
- The ctrlr_cmd test was passing an invalid number of entries to
nvme_ctrlr_cmd_get_error_page(). This case should probably not be an
assert but rather an error code. However, the function does not
return a status code currently, so it is not trivial to make that
change. For now, just drop the asserting test case and the code added
to the test to work around it.
While we're here, fix the macros in the unit test nvme_impl.h so they
are usable in single-line conditionals without braces - that is the
whole point of the do { ... } while (0) pattern, so there should be no
trailing semicolon.
Change-Id: Iad503c5c5d19a426d48c80d9a7d6da12ff2c982a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Cover more functions and conditions for nvme_ctrlr_cmd.
Remove unnecessary lines in nvme_ctrlr_ut.
Update nvme_qpair_ut.
Remove unnecessary header file.
Change-Id: I8c5a75573b26210ca57711b366acd55ab96614c0
Signed-off-by: Liang Yan <liangx.yan@intel.com>
Initialize the full nvme_namespace structure in prepare_for_test() so
that e.g. ns->id is not used uninitialized.
Also check for request allocation failure - if the request is NULL, we
can't run the rest of the tests that dereference request without
crashing.
Change-Id: I3010ca3e81f153a4d0201498a14a963c2b9e960d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
GCC generates a series of 64-bit MOV instructions for the memcpy() into
the submission queue. We can do better with 128-bit SSE2 instructions.
DPDK already has a memcpy implementation that is optimized for small
inline copies, so use it instead of memcpy.
Change-Id: I5f09259b4d5cb089ace4a8ea6d2078c03fee84f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use {} consistently rather than {0} when initializing structs as part of
definitions.
Fixes warnings about initializing subobjects when compiling with clang.
Change-Id: I507fdf47e7c41455033d6c3d5edc563ec4667dee
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_ctrlr_cmd_set_async_event_config() is called from within nvme_ctrlr
but was never stubbed out in its unit test.
Change-Id: I44002540e74ee010f21d0cf2d089d1dc51217a2f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This prevents warnings about unused phys_addr variable in the unit
tests.
Change-Id: I022483735ba92eb112e541e6de37dc9a8c5d2b8e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Replace unnecessary allocation with a stack variable.
Clears up a potential NULL pointer dereference indicated by scan-build.
Change-Id: I81a7591729b0f1630bab9ce378716bd2369d6b85
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
act_tr is supposed to be allocated with the system allocator
(malloc/calloc) rather than nvme_malloc.
Additionally, the size was incorrect - act_tr is an array of pointers,
not a single struct nvme_tracker.
Change-Id: I2c66b4891d11d0a8a32e3740c27fcfb42b1db2d7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_dump_command is totally unused aside from the unit test.
nvme_dump_completion was used in qpair, but it can be replaced with the
equivalent nvme_qpair_print_completion.
Also added the missing nvme_completion fields to nvme_qpair_print_completion
that had been printed by nvme_dump_command.
Change-Id: Ia5ee66f3553df06febe8f465d42e49a84c555dd2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make nvme_pcicfg_read32 set the referenced variable and make
nvme_pcicfg_write32 appear to read it so that the compiler doesn't warn
about unused/uninitialized data.
Change-Id: I4f06c0cca2fc11a8c6c5a60543c1c50a2f6a412d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Replace a bunch of allocations with stack variables.
Remove many unnecessary variables.
The actual tests are unchanged (and still not testing very much), but
this removes a lot of cruft that should make it easier to see what the
tests are actually doing.
Change-Id: I2c4810391cbf4d8edde88d5bc1c0dddcba3ae175
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
