This will allow replacing these _nvme_fail_request_bad_vtophys() calls
with the correct error later. vtophys is not actually used within the
SGL request builder, so this is the wrong error.
Change-Id: Ibc2a3b029a8abad1d563b9df200325d7d64498da
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
No code change, just moved into a function for readability.
Change-Id: I883443c06d961c6dbeffed1a6fb153177e6e3fcd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This cleans up the I/O splitting code somewhat.
It also moves the SGL payload function pointers up into the hot cache
section of struct nvme_request without pushing the other important
members past the cacheline boundary (because payload is now a union).
Change-Id: I14a5c24f579d57bb84d845147d03aa53bb4bb209
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For the purpose to support different types of input scattered payloads,
such as iovs or scattered list, we define common method in the NVMe
driver, users should implement their own functions to iterate each
segment memory.
Change-Id: Id2765747296a66997518281af0db04888ffc4b53
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
nvme_ctrlr_process_io_completions() and
nvme_ctrlr_process_admin_completions() now return the number of
completions processed.
This also adds the possibility of returning an error from the
process_*_completions functions (currently unused, but this at least
gets the API ready in case error conditions are added later).
Change-Id: I1b32ee4f2f3c1c474d646fa2d6b8b7bbb769785f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_qpair_submit_tracker() and nvme_qpair_manual_complete_request() are
only used from within nvme_qpair.c, so they can be static.
nvme_qpair_submit_tracker() is moved up to avoid needing a declaration
(no other code change).
nvme_ctrlr_hw_reset() is only used from within nvme_ctrlr.c, so it can
be static.
Change-Id: I9a7953d7baaec76e875dd535daf557ea24bef801
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
NVMe doesn't require the specific 64-bit MMIO ordering on 32-bit
platforms performed in spdk_mmio_read_8(), but it doesn't hurt.
We have to pick one of the two possible orderings, so pick the one
required by I/OAT.
Change-Id: I2b909d64d0c077b797d0f64a11d78d1ecc55eec7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_ctrlr_process_io_completions() now takes a second parameter,
max_completions, to let the user limit the number of I/Os completed on
each poll.
If there are many I/Os waiting to be completed, the
nvme_ctrlr_process_io_completions() function could run for a long time
before returning control to the user, so the max_completions parameter
lets the user have more control of latency.
Change-Id: I3173059d94ec1cc5dbb636fc0ffd3dc09f3bfe4b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
GCC generates a series of 64-bit MOV instructions for the memcpy() into
the submission queue. We can do better with 128-bit SSE2 instructions.
DPDK already has a memcpy implementation that is optimized for small
inline copies, so use it instead of memcpy.
Change-Id: I5f09259b4d5cb089ace4a8ea6d2078c03fee84f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
No change in behavior, just a simplification.
We already have a check for retry, so pull the cb_fn check out and put
it under the !retry branch.
This makes it clearer that requests that are going to be retried will
not get their callbacks called.
Change-Id: I70c7067e550c7fca78b0441b5474833f73863315
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Since it is empty function. It can be deleted.
Add nvme_assert to fix issue reported by scan-build.
Change-Id: Ia0e8f656e1dac0da7ec72f8404469ea1b0dcb40e
Signed-off-by: Liang Yan <liangx.yan@intel.com>
In nvme_qpair_complete_tracker, make sure we got a valid request in the
tracker that is being completed.
This should never occur in practice, but safeguard against it in case of
programmer error. Fixes a scan-build warning about potential NULL
dereference.
Change-Id: Id82af604d2a5ed5de0aeccf3affa1900f6712ebe
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Document the assumptions made by qpair_construct using asserts.
These values can't actually be 0 in practice due to the way they are
derived, but scan-build can't see that. It is also useful to have these
asserts in case of future modifications.
Change-Id: I546c057f5cbe7ccc62acd90b595e423cd450d86a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_dump_command is totally unused aside from the unit test.
nvme_dump_completion was used in qpair, but it can be replaced with the
equivalent nvme_qpair_print_completion.
Also added the missing nvme_completion fields to nvme_qpair_print_completion
that had been printed by nvme_dump_command.
Change-Id: Ia5ee66f3553df06febe8f465d42e49a84c555dd2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
