Unmapped blocks on some older nvme devices will read zero even if the
device does not explicitly define the unmap behavior.
Change-Id: Ia825a406cbd01f89192c300cfe35013fb4aed715
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/372006
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
For pcie, this just equals the number of SGLs we can fit
into the per-tracker memory.
For rdma, this is just set to 1 for now since nvme_rdma.c
does not support multiple SGEs yet. Once that support is
added, this will change to use MSDBD (Maximum SGL Data Block
Descriptors) instead from the controller identify data.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I34a4c546b5ff46918a296a73ed8cbcc6c9879d5a
Reviewed-on: https://review.gerrithub.io/372358
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add a new struct spdk_nvme_io_qpair_opts to allow the user to override
controller options on a per-I/O qpair basis.
Existing callers with qprio == 0 can be updated to:
... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);
Callers that need to specify a non-default qprio should be updated to:
struct spdk_nvme_io_qpair_opts opts;
spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts));
opts.qprio = SPDK_NVME_QPRIO_...;
... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts));
Change-Id: I8ac3ea369535cfde759abbe75e1d974b6450a800
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/369676
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The VirtualBox emulated NVMe device will intermittently
hang on the first read/write command after an I/O
qpair has been allocated. The frequency of the hang
diminishes if a delay is added after allocating the I/O
qpair - until it disappears completely with a 100us delay.
So add a quirk to insert this delay.
Note - the 100us delay was tested by running
the hello_world example app 50000 times.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I237e31b1b8a1a1e28262851ae0a21cd7345f0f1a
A 100us is so small that applying the quirk to the specific
SSDs that require the delay is more trouble than it is worth.
So remove the quirk and always wait 100us before re-enabling
the NVMe SSD during initialization.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id6a8cc6e35d103fffdf135580301fc3e5b27e722
Queue aborts that would exceed the abort command limit
in software as a convenience for the user.
Change-Id: I8c1f0380984cc6c0cdb453db961939a7f571b336
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
It has been discovered that some devices require
a very small delay before writing CC.EN to 1 after
CSTS.RDY goes to 0.
Change-Id: I73d31726d17ebf5bbec7ee528e2f98fcd05234dd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The actual uses of intrinsics are already guarded by feature-specific
ifdefs in nvme_pcie_copy_command(), but the header itself should also
only be included when it will actually be needed.
Change-Id: Ife65d6432b8dfd9d9db80fe4e385ab76491874c0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For namespaces with end-to-end protection information, metadata size
of exactly 8 bytes, and extended LBA configured, the NVMe driver would
calculate the size of the data block incorrectly. The NVMe spec has a
special provision for this specific case (8-byte metadata only) and
PRACT = 1 that requires that the host does not send the metadata as part
of the host memory buffer.
To fix this, clean up the calculation of the per-block data transfer
size by adding a new extended_lba_size field in the namespace, which
represents the total size of data to be transferred per block based on
the namespace's configured metadata size and whether it transfers
metadata as part of the data buffer. Then add the special case for
PRACT = 1 and PI configured and extended LBA in the R/W helper
functions.
Change-Id: I0b383a58c773cac06e6c018858b57129064c6059
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These were repeated a few different places, so pull them into a common
header file.
Change-Id: Id807fa2cfec0de2e0363aeb081510fb801781985
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The PCIe transport initializes the quirks directly, so the generic hook
to get PCI ID is no longer necessary. This path was dead code.
Change-Id: I25bdaa598db53e4312a264d9d8356d1b416696e5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The logic to fail queue pairs when the controller is failed should be
handled in the generic code, not in the individual transports.
This also allows nvme_qpair_fail() to be private to nvme_qpair.c.
Change-Id: I6194576dceb35073b9af8847e59314900028637c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Since nvme_ns_cmd.c now walks the SGL, some of the test code
needs to also be updated to initialize and return correct values
such as ctrlr->flags and sge_length.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I521213695def35d0897aabf57a0638a6c347632e
Now that the hotplug code is isolated in nvme_pcie.c, it can call the
PCIe transport attach function directly.
Change-Id: I2df3b9168473b537cc9b13367e06d3d3b6fa22be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The spdk_nvme_qpair::num_entries value is never used in the common code,
so move it to the individual transport qpairs to make it clear that it
is a transport-specific implementation detail.
Change-Id: I5c8f0de4fcd808912ba6d248cf5cee816079fd32
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The definition of SPDK_UNREACHABLE uses the build-time DEBUG definition,
which is not available in the public API.
Change-Id: I1862c99fa5c85ccd3483f94e9c35de531da57f3c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This function initializes the members of an existing
qpair struct. It doesn't construct one from scratch.
Change-Id: I0b9afac1ad25cfb217efd146702f693c74f5f697
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Make the qpair construct functions private to the transports - it
doesn't need to be called from generic code.
Change-Id: I5f730a4bcf60ce231fe27bc8f4c3c39cb647dd2d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a transport callback to return the maximum queue size, and enforce
it in the generic nvme_ctrlr layer.
This allows the user to tell what io_queue_size was actually selected by
the transport via the ctrlr_opts returned during attach_cb.
Change-Id: I8a51332cc01c6655e2a3a171bb92877fe48ea267
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This better describes what the field controls (it does not affect the
admin queue size).
Change-Id: I851ae46fb4ed0fce819af07ae235824e0fc817e6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The other simplifications to probe_info and trid made the
trtype argument redundant.
Change-Id: Ie7bea4e2204e690dc4909eeacd065e0722b53272
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The probe_info was reduced to just containing a
transport_id, so remove probe_info entirely.
Change-Id: Ica9a22d126cd14e282decd3eea1a0afe0460f099
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Use the standard quirk mechanism to specify which devices
need software assisted striping.
Change-Id: Id8156876a90b4caf9d687637e14c7ad4a66ceda6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Scanning the transport may result in both new
devices and removed devices, so pass the callback
for both operations.
Change-Id: I6f73dbe6fd7cf61575c354b43f8ae3e2a01e2965
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Simplify the arguments to nvme_transport_ctrlr_scan to take
a transport id that identifies the discovery service (or
NULL to scan PCIe).
Further, separate scan into two functions - scan and attach.
Scan is for scanning an entire bus, attach is for a specific
device.
Change-Id: I464f351a02a04bc5a45096dcf5dc8fc5ac489041
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is a small step toward making discovery more like
scanning a local PCI bus.
Change-Id: Ie7149ad060f2eeb56939b1241187bdf09681f2aa
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
It's not the whole transport - it's just an enum for the
type of transport.
Change-Id: Ia435a21792f221ddf50ddf4f0923c6152622eccb
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This version of multi-process support needs to have DPDK 16.11 builtin.
Change-Id: I3352944516f327800b4bd640347afc6127d82ed4
Signed-off-by: GangCao <gang.cao@intel.com>
Allow the host NQN to be overriden when connecting to NVMe over Fabrics
controllers.
Change-Id: I8fcf2e89ae7d9722677e834f76a8fe805c52f91b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Each transport should handle its own qpair cleanup internally.
Change-Id: I7dd737be820ea6bad686f4aad7d74044fad58a47
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Let the transport access the controller options during
ctrlr_construct().
Change-Id: I83590c111e75c843685dd9315f0f08416168356d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the NVMe over Fabrics spec definitions for TRTYPE rather than the
internal library transport type.
Change-Id: Idead559a8f8d95274fc580d10e82033822e6eda8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The SPDK_TRACELOG macro depends on a CONFIG setting (DEBUG), so it
should not be part of the public API.
Create a new include/spdk_internal directory for headers that should
only be used within SPDK, not exported for public use.
Change-Id: I39b90ce57da3270e735ba32210c4b3a3468c460b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Considering the process can be terminated in the cases like ctrl+c,
kill command or memory fault, the ref is tracked in the per process
structure spdk_nvme_controller_process and whenever there is other
process attaches or detaches the controller, a scan will be issued
to cleanup those unexpectedly exited processes.
Change-Id: Ib4f974f567a865748d42da4ead49edd383dfc752
Signed-off-by: GangCao <gang.cao@intel.com>
Function pointers will not work for the DPDK multi-process model (they
can have different addresses in different processes), so define a
transport enum and dispatch functions that switch on the transport type
instead.
Change-Id: Ic16866786eba5e523ce533e56e7a5c92672eb2a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
spdk_nvme_probe() will now provide a struct spdk_nvme_probe_info to the
probe and attach callbacks in place of the PCI device pointer.
This struct contains the useful information that could be retrieved from
the PCI device during probe.
The goal of this change is to allow expansion of the probe information
in the future when other transports (specifically, NVMe over Fabrics)
are added that do not necessarily use PCI addressing or device IDs.
Change-Id: I59a2a9e874e248ce5fa1d7f4b57c8056962ff3cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the new public PCI ID structure in the NVMe library to replace the
previously private struct pci_id.
Change-Id: I267d343917f60bdae949a824bc0fe67457cbbc0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a field to struct spdk_nvme_ctrlr_opts that allows the user to
specify a keep alive timeout, and add automatic submission of Keep Alive
commands to spdk_nvme_ctrlr_process_admin_completions().
Change-Id: Ib282299a571d8edc59c7933418751bc3a6c98b40
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the quirks mechanism generic in preparation for quirks for devices
from other vendors.
Change-Id: Ic003b020a38f1b966021db30e3f2bce9cf6a1a0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a transport function to get the max data transfer size to break the
dependency on NVME_MAX_XFER_SIZE.
Change-Id: I846d12878bdd8b80903ca1b1b49b3bb8e2be98bb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Move the PCIe-specific admin queue setup to nvme_pcie_ctrlr_enable.
Change-Id: Ic3f5625fa804f719040ba86b7fc3bf82fcc057c0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The value of CAP should not change during the lifetime of a controller,
so read it once during ctrlr_construct and store it in the ctrlr.
Change-Id: I089d4141b4e0c9aae6c53abf9bb0ef6577dabe0b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than embedding adminq directly in the spdk_nvme_ctrlr structure,
change it to a pointer to a spdk_nvme_qpair. This is necessary to allow
the transport to extend the qpair structure.
Change-Id: I041685d5037088cf56d046fe99bf204edcfc57b1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This requires a couple of related changes:
- I/O queue IDs are now allocated by using a bit array of free queue IDs
instead of keeping an array of pre-initialized qpair structures.
- The "create I/O qpair" function has been split into two: one to create
the queue pair at startup, and one to reinitialize an existing qpair
structure after a reset.
Change-Id: I4ff3bf79b40130044428516f233b07c839d1b548
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the transport ctrlr_construct callback responsible for allocating
its own controller.
Change-Id: I5102ee233df23e27349410ed063cde8bfdce4c67
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These are specific to local NVMe PCIe devices, so move them out of the
generic NVMe code into the PCIe transport.
Change-Id: Iea2056a4c438b7d3a303b4b5e977ce7aa9e58c05
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow factoring out PCIe-specific code into a swappable
transport so that NVMe over Fabrics host support can be added.
Change-Id: I4df74dd268d655e3b36e8d6114ebe7d79a24844d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than forcing the NVMe library user to pass a specially-allocated
block of memory (e.g. rte_malloc() in the case of the default
nvme_impl.h), just make the NVMe library allocate a suitable buffer
itself and copy to/from the user buffer as needed.
The fast path I/O functions still require special rte_malloc()
allocations, since we don't want to add an allocation and copy to the
I/O critical path.
Change-Id: I7fe88c0ba60c859a33bbe95b7713f423c6bf1ea8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
pthreads are widely supported and are available on any
platform we currently foresee porting to. Use that API
instead of attempting to abstract it away to simplify
the code.
Change-Id: I822f9c10910020719e94cce6fca4e1600a2d9f2a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is a step towards enabling sharing SPDK NVMe
device access from multiple processes using DPDK's
multi-process framework.
Change-Id: I57d5eec158b42addc1036bd2583596471a467a95
Signed-off-by: GangCao <gang.cao@intel.com>
Previously, we used cap_lo and cap_hi to represent the 32-bit halves of
the full CAP register. However, it is simpler to keep them in a single
64-bit structure, and is no less efficient on 64-bit platforms.
Also name the NSSRS field from NVMe 1.2, which was previously reserved.
Change-Id: I1d5d9b0dccbb12373b4aed3db29c883881d43223
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This lets spdk_nvme_qpair fit in 128 bytes exactly.
Change-Id: I7c42582f22ece72a7f1d651468e63d4fe05babd6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The user can determine whether submission queues will be placed in the
controller memory buffer by checking the controller options use_cmb_sqs
flag in the attach callback.
Change-Id: I8a925ef99a48665a0e2ffaa90d9ff2b79b90b2fa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The D3700/D3600 series support Controller Memory Buffer(CMB) feature,
CMB is available for holding submission queues, for those controllers
which can support submission queues in CMB, user can set the option
whether to enable it or not.
Change-Id: I8b0dc9e28dd6f5bb01bee99a532087212c04e492
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
For those controllers which can support end-to-end data protection
feature, add the support in the driver layer.
Change-Id: Ifac3dd89dec9860773c850416a6116113a6ce22a
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Rename nvme_remove_child_request() to nvme_request_remove_child() and
move it next to nvme_request_add_child() to make the symmetry clear.
Change-Id: I78747c44ab3db1a656b33555a45f634dc5a55b31
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch is used to add a nvme_request remove child
helpler function
Change-Id: I1e5bb228d53333ca3601f4ae30fcd801ea39e532
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Per the NVMe spec, SGL segments must be Qword (8-byte) aligned. Add a
static assert to make sure this is true for the sgl member of struct
nvme_tracker (assuming the whole nvme_tracker is at least 8-byte aligned).
Change-Id: I827aa40b56de648d83f524a4f1e79c3202b676be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If the controller is failed, attempting to submit additional I/O is
futile - it will be immediately failed using the completion callback,
which can result in infinite recursion if the application code resubmits
I/Os on failure.
Instead, provide a way for request submission to indicate failure, and
use it to exit early if the controller is failed; this can only happen
when a reset failed (timed out).
If a request is submitted directly by the user when the controller has
failed, we can return an error code directly. For the case where I/O
was queued and is being resubmitted after a reset, we still need to call
the completion handler via _nvme_fail_request_ctrlr_failed().
Change-Id: I9e144328d524b25db2acf48e923b584746e8d0b6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Provide a new structure, spdk_nvme_ctrlr_opts, to let the user modify
the default controller initialization options during probe/attach.
Currently, only the number of queue pairs can be modified in this way;
other options will be added later.
Change-Id: Ie27b9429291d93a9353c0d820f0ad467d3b0e7cb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
