Make the quirks mechanism generic in preparation for quirks for devices
from other vendors.
Change-Id: Ic003b020a38f1b966021db30e3f2bce9cf6a1a0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a transport function to get the max data transfer size to break the
dependency on NVME_MAX_XFER_SIZE.
Change-Id: I846d12878bdd8b80903ca1b1b49b3bb8e2be98bb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Move the PCIe-specific admin queue setup to nvme_pcie_ctrlr_enable.
Change-Id: Ic3f5625fa804f719040ba86b7fc3bf82fcc057c0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The value of CAP should not change during the lifetime of a controller,
so read it once during ctrlr_construct and store it in the ctrlr.
Change-Id: I089d4141b4e0c9aae6c53abf9bb0ef6577dabe0b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than embedding adminq directly in the spdk_nvme_ctrlr structure,
change it to a pointer to a spdk_nvme_qpair. This is necessary to allow
the transport to extend the qpair structure.
Change-Id: I041685d5037088cf56d046fe99bf204edcfc57b1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This requires a couple of related changes:
- I/O queue IDs are now allocated by using a bit array of free queue IDs
instead of keeping an array of pre-initialized qpair structures.
- The "create I/O qpair" function has been split into two: one to create
the queue pair at startup, and one to reinitialize an existing qpair
structure after a reset.
Change-Id: I4ff3bf79b40130044428516f233b07c839d1b548
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the transport ctrlr_construct callback responsible for allocating
its own controller.
Change-Id: I5102ee233df23e27349410ed063cde8bfdce4c67
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These are specific to local NVMe PCIe devices, so move them out of the
generic NVMe code into the PCIe transport.
Change-Id: Iea2056a4c438b7d3a303b4b5e977ce7aa9e58c05
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow factoring out PCIe-specific code into a swappable
transport so that NVMe over Fabrics host support can be added.
Change-Id: I4df74dd268d655e3b36e8d6114ebe7d79a24844d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For the nvme readv/writev APIs, the PRP checking logic was
incorrectly failing single SGE payloads that were larger
than 4KB. This patch adds a test case for this scenario,
and fixes the PRP checking logic.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6357d620599666046d2cb74d7923dac1f75418c5
Use standard GCC style atomic operations instead of
the DPDK calls. The DPDK calls end up translating
to the gcc standard inline calls in the generic
case anyway.
Change-Id: I0ea760c4e23c3660b082a803bbc174de7250f365
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Remove #includes for all DPDK headers that weren't
necessary.
Change-Id: Ib02522e0f04e64a1c98afceb7508cc0e8d931a9d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This was only used for debugging. Everywhere else
used the spdk_memzone abstraction.
Change-Id: I8a828ea3c7abccb66c8a027cb13de43c560ff7a1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This converts some, but not all, usage of rte_mempool
to spdk_mempool. The remaining rte_mempools use features
we elected not to expose through spdk_mempool such as
constructors, so that will need to be revisited.
Change-Id: I6528809a864ab466b8d19431789bf0f976b648b6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Use the env library to perform all memory allocations
that previously called DPDK directly.
Change-Id: I6d33e85bde99796e0c85277d6d4880521c34f10d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Enforce exactly one trailing \n, and fix all of the existing cases.
Change-Id: I6218e4700e90aeb647eaee78089530c79993c8c8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows users to swap their PCI library from
libpciaccess/dpdk to another mechanism using the standard
method for swapping out the env library.
Change-Id: Ib2248f8b43754a540de2ec01897e571f0302b667
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This allows users to swap out SPDK's third party
libraries for an implementation based on their own
framework.
Change-Id: Ia0b7384ce5e31acba5ad0d7002dec9e95b759c52
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The new env library will wrap all third-party library
calls and be easily swappable with alternate implementations
at build time. For now, it's just the memory library
renamed.
Change-Id: I26a70933289f8137107208ba75f7520fd7a33da0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This patch also drops support for automatically unbinding
devices from the kernel - run scripts/setup.sh first.
Our generic pci interface is now hidden behind include/spdk/pci.h
and implemented in lib/util/pci.c. We no longer wrap the calls
in nvme_impl.h or ioat_impl.h. The implementation now only uses
DPDK and the libpciaccess dependency has been removed. If using
a version of DPDK earlier than 16.07, enumerating devices
by class code isn't available and only Intel SSDs will be
discovered. DPDK 16.07 adds enumeration by class code and all
NVMe devices will be correctly discovered.
Change-Id: I0e8bac36b5ca57df604a2b310c47342c67dc9f3c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Provide a convenience wrapper for general purpose dataset
management commands. The previous wrapper for deallocate
was difficult to use correctly and only for deallocate.
Note that the name is "dataset_management" as opposed to
"data_set_management" to match the NVMe specification.
It's questionable whether "dataset" is valid English, but
it is best to match the specification.
Change-Id: Ifc03d66dbabeabe8146968cf8a09f7ac3446ad68
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
If posix_memalign() fails, it may not have updated buf, so set it to
NULL explicitly.
Change-Id: I756bdc59ec1e31987ad3e6754eec4e2194b95074
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than forcing the NVMe library user to pass a specially-allocated
block of memory (e.g. rte_malloc() in the case of the default
nvme_impl.h), just make the NVMe library allocate a suitable buffer
itself and copy to/from the user buffer as needed.
The fast path I/O functions still require special rte_malloc()
allocations, since we don't want to add an allocation and copy to the
I/O critical path.
Change-Id: I7fe88c0ba60c859a33bbe95b7713f423c6bf1ea8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
assert is part of the C standard library and is available
on any platform we'd consider porting to. Don't put a
wrapper around it.
Change-Id: I0acfdd6a8a269d6c37df38fb7ddf4f1227630223
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
pthreads are widely supported and are available on any
platform we currently foresee porting to. Use that API
instead of attempting to abstract it away to simplify
the code.
Change-Id: I822f9c10910020719e94cce6fca4e1600a2d9f2a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Add slightly better statistics to get a range of values instead of just
the average.
Change-Id: I159994dce38412755afdd8980030c407125957e9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is a step towards enabling sharing SPDK NVMe
device access from multiple processes using DPDK's
multi-process framework.
Change-Id: I57d5eec158b42addc1036bd2583596471a467a95
Signed-off-by: GangCao <gang.cao@intel.com>
Previously, we used cap_lo and cap_hi to represent the 32-bit halves of
the full CAP register. However, it is simpler to keep them in a single
64-bit structure, and is no less efficient on 64-bit platforms.
Also name the NSSRS field from NVMe 1.2, which was previously reserved.
Change-Id: I1d5d9b0dccbb12373b4aed3db29c883881d43223
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow removal notifications to be propagated to the library
user (e.g. for hotplug).
The callback is currently unused, but this at least prepares the API for
the future hotplug support.
Based on a patch by Dave Jiang <dave.jiang@intel.com>
Change-Id: I20b1c2dbf5e084e0b45a7e51205aba4514ee9a95
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the knowledge that both the source and destination of
nvme_copy_command() are aligned to emit the aligned variants of the
SSE2/AVX mov instructions.
Change-Id: I0a7e32a3bb10b9a1920cd85691b79fa7172eecb3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Match the expected alignment for nvme_alloc_request() and nvme_malloc()
to allow optimized memory copy code to work even in the unit tests.
Change-Id: I546692a6df9615a12a8209618fb6159a9c9e426b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This moves some definitions from nvmf_spec.h to
nvme_spec.h based on the latest publication.
Change-Id: I51b0abd16f7d034696239894aea5089f8ac70c40
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Simplify the build rules so that common libraries are always linked.
Also fix up a couple of -lpciaccess instances that should have been
changed to $(PCIACCESS_LIB).
Change-Id: I4c50fa3aa59cae013d3385e38fbb830794299f6e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
NVMe opcodes contain a two-bit field that encodes the expected data
direction for each command. Add an enum and a function to extract these
bits.
Change-Id: Ie214319f121cf0899c6aa5663866f2988b128dd2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use SPDK_CU_ASSERT_FATAL instead of CU_ASSERT_FATAL so static analyzers
recognize that g_request cannot be NULL in the following lines.
Change-Id: Ie7ab3bd34a177bea0d565441014e8db12be8bb01
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The D3700/D3600 series support Controller Memory Buffer(CMB) feature,
CMB is available for holding submission queues, for those controllers
which can support submission queues in CMB, user can set the option
whether to enable it or not.
Change-Id: I8b0dc9e28dd6f5bb01bee99a532087212c04e492
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Resolve relative paths before using them to clean up command lines.
This should also help shorten the overall command line length that gets
embedded in the binary and used when locating the executable from a
coredump.
Change-Id: Ibff9849ede198bb04313496c8b7131485ffaf14f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fix copy of uninitialized data in the unit tests.
Change-Id: Ib71a6fc90edb07f7ddac3067ff0efbda7938bf17
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch add support for Intel specific log pages :
marketing description page.
Change-Id: I87bccb2af286279598c9dd3c870094b384a0d2f7
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
For those controllers which can support end-to-end data protection
feature, add the support in the driver layer.
Change-Id: Ifac3dd89dec9860773c850416a6116113a6ce22a
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Rename nvme_remove_child_request() to nvme_request_remove_child() and
move it next to nvme_request_add_child() to make the symmetry clear.
Change-Id: I78747c44ab3db1a656b33555a45f634dc5a55b31
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This can happen if the controller is still resetting as the SPDK NVMe
driver takes control.
Change-Id: I263ae8f2e7b271e0448450557452a115c90c4fb6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch is used to add a nvme_request remove child
helpler function
Change-Id: I1e5bb228d53333ca3601f4ae30fcd801ea39e532
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The next_sge callback can be called multiple times for a single I/O if
the I/O is split. Get the physical address for each element once
instead of every time the SGL is walked.
Also bake the SGL element offset into the physical address.
Change-Id: I5d83426a234ffe9d13b89621077dbbb1561181c9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Also use uint32_t consistently for counting SGL elements.
Change-Id: Ibeffffdc5e0eccd11681703cd333bfb64319e164
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows us to test elements with particular (mis-)alignments without
depending on the internal behavior of the DPDK allocator.
Change-Id: I006358b82ae3a677353333954ba88d1bcfe289f6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
There's no reason to use the system-defined iovec structure; we only use
it for our internal scatter-gather callbacks. Define our own structure
so it can be extended.
Change-Id: Ifc055adfdae8cddd53e7f5d638a24f1e812b1ac6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This helps identify which iteration of the test failed.
Change-Id: I9186de08ea8217348b8ee61b0ecba4e9d7ab986e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If the controller is failed, attempting to submit additional I/O is
futile - it will be immediately failed using the completion callback,
which can result in infinite recursion if the application code resubmits
I/Os on failure.
Instead, provide a way for request submission to indicate failure, and
use it to exit early if the controller is failed; this can only happen
when a reset failed (timed out).
If a request is submitted directly by the user when the controller has
failed, we can return an error code directly. For the case where I/O
was queued and is being resubmitted after a reset, we still need to call
the completion handler via _nvme_fail_request_ctrlr_failed().
Change-Id: I9e144328d524b25db2acf48e923b584746e8d0b6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This test is only intended to validate functionality, not actual
performance, so one second is plenty.
Change-Id: I2ff198c035226b50a113f9ff189c1abbd0fd1c17
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Provide a new structure, spdk_nvme_ctrlr_opts, to let the user modify
the default controller initialization options during probe/attach.
Currently, only the number of queue pairs can be modified in this way;
other options will be added later.
Change-Id: Ie27b9429291d93a9353c0d820f0ad467d3b0e7cb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Replace the previous code that allocated each tracker individually with
one large allocation per queue pair.
struct nvme_tracker is now explicitly padded to reach exactly 4096 bytes
to allow normal array indexing to work correctly while maintaining the
alignment requirement that ensures each tracker's PRP list does not
cross a page boundary.
This also allows removal of the act_tr array, since the tr array can be
indexed directly now, and each tracker can store its own active state.
Change-Id: Ia7c51735b96594d12f7f478cefcc4aedc84207ad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The previous method for registering I/O queues did not allow the user
to specify queue priority for weighted round robin arbitration, and it
limited the application to one queue per controller per thread.
Change the API to require explicit allocation of each queue for each
controller using the new function spdk_nvme_ctrlr_alloc_io_qpair().
Each function that submits a command on an I/O queue now takes an
explicit qpair parameter rather than implicitly using the thread-local
queue.
This also allows the application to allocate different numbers of
threads per controller; previously, the number of queues was capped at
the smallest value supported by any attached controller.
Weighted round robin arbitration is not supported yet; additional
changes to the controller startup process are required to enable
alternate arbitration methods.
Change-Id: Ia33be1050a6953bc5a3cca9284aefcd95b01116e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This field is write-only in the current code; the NVMe library does
not track timeouts on requests.
Change-Id: I50e53bb3c299bf16912c48be8aad3eec829154af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
When I/O spans a stripe boundary, the driver splits the request into
multiple requests, so for 1 segment memory larger than the stripe
size, we also need to split the segment memory.
Change-Id: I22ea5734d7066865a57a3c90fe18d5f76f373f1d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Remove the common code that was factored out to spdk.common.mk.
Change-Id: I2a6265fa144b2c03f3484e2aa528fd1e15d13429
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add CUnit test case to verify payload_offset value in split_test4
Change-Id: I4a9a33854295ed802709bbe4f11746d284ed8cbd
Signed-off-by: Liang Yan <liangx.yan@intel.com>
Add test case which verify invalid second phys_addr would cause
_nvme_fail_request_bad_vtophys be called.
Change-Id: Id1b62e249b7192e29334bd7cab33815722f5662d
Signed-off-by: Liang Yan <liangx.yan@intel.com>
