Also add adrfam to the NVMe bdev JSON config output.
Change-Id: I9472bda04947cffc0df9b02eba0035bac01b7d7b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/367292
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Nothing in the spec indicates that NQNs should be case insensitive, and
we have fixed this elsewhere (e.g. commit df70bc1559: "nvmf: use
case-sensitive comparison for NQNs").
Change-Id: I4a48d1c7f25ec5af9ce4d73f1bf2fa543236503a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/367106
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
PCIe transport IDs are a non-standard extension to the NVMe-oF transport
address, and they only use the transport type and address fields of the
structure. Add a special case so that the rest of the fields are
ignored for PCIe addresses. All other transport types are NVMe-oF
addresses and should compare all fields.
Change-Id: I45ed143ea1712d17c6de8082677deeefd395c8a2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/365916
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
- rename spdk_malloc_socket to spdk_dma_malloc_socket
- rename spdk_malloc to spdk_dma_malloc
- rename spdk_zmalloc to spdk_dma_zmalloc
- rename spdk_realloc to spdk_dma_realloc
- rename spdk_free to spdk_dma_free
Change-Id: I52a11b7a4243281f9c56f503e826fd7c4a1fd883
Signed-off-by: John Meneghini <johnm@netapp.com>
Reviewed-on: https://review.gerrithub.io/362604
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Unit tests will be added as part of a separate patch updating all UT for
nvme.c. Global used for timeout value so it can be easily overwritten
by the upcoming unit tests for this function.
Change-Id: I7fc15aab91601ac57c94cae266b212c0998d2495
Signed-off-by: paul luse <paul.e.luse@intel.com>
They were very close to the same already, so finish the job.
Change-Id: Ifba9e3b2d11a3e70cbfbe46f57a67552db2757ed
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The other simplifications to probe_info and trid made the
trtype argument redundant.
Change-Id: Ie7bea4e2204e690dc4909eeacd065e0722b53272
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The probe_info was reduced to just containing a
transport_id, so remove probe_info entirely.
Change-Id: Ica9a22d126cd14e282decd3eea1a0afe0460f099
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This bug was preventing multiple calls to spdk_nvme_probe() from
working, since the first call would return 0 from all of the DPDK driver
init callbacks and prevent other devices from ever being enumerated in
subsequent calls.
Reported-by: Tsuyoshi Uchida <tuchida@us.fujitsu.com>
Change-Id: I871aa170bbd03be111604eeabe3a7a7a4f40ce89
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This way, all new controllers discovered will be initialized
in parallel.
Change-Id: Iebedb3905eb2787a3708f74425afae40ca31253d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
If the first call to spdk_nvme_probe probes a device and
the driver elects not to take it, still call the probe
callback for that device on subsequence calls to
spdk_nvme_probe.
Change-Id: If06467cf6796c827a0bbfba6e36d5b91534526fc
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Move this down a level so it happens on all paths.
Change-Id: Iea9913f0e102353882466c8dea4ee39abb857520
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Scanning the transport may result in both new
devices and removed devices, so pass the callback
for both operations.
Change-Id: I6f73dbe6fd7cf61575c354b43f8ae3e2a01e2965
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Simplify the arguments to nvme_transport_ctrlr_scan to take
a transport id that identifies the discovery service (or
NULL to scan PCIe).
Further, separate scan into two functions - scan and attach.
Scan is for scanning an entire bus, attach is for a specific
device.
Change-Id: I464f351a02a04bc5a45096dcf5dc8fc5ac489041
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Instead of repeating the fields, just embed a transport_id.
Change-Id: I282704c9d59784abd5f7c93be4e47c673fcf6dde
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is a small step toward making discovery more like
scanning a local PCI bus.
Change-Id: Ie7149ad060f2eeb56939b1241187bdf09681f2aa
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
It's not the whole transport - it's just an enum for the
type of transport.
Change-Id: Ia435a21792f221ddf50ddf4f0923c6152622eccb
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The user's remove_cb should detach the NVMe controller when it can
ensure that it is no longer in use. In the interim (between remove_cb
and spdk_nvme_detach()), the controller will remain in a failed state,
so any new I/O submissions will return an error code but not crash.
examples/nvme/hotplug is not yet updated for this change, but that will
be done in a separate patch.
Change-Id: I8827ba36f9688ccb734e7871f20f11ec11e88f96
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This version of multi-process support needs to have DPDK 16.11 builtin.
Change-Id: I3352944516f327800b4bd640347afc6127d82ed4
Signed-off-by: GangCao <gang.cao@intel.com>
The discover and probe 'nqn' fields are subsystem NQNs, so name them
subnqn to be consistent with the spec and the rest of the code and to
distinguish them from host NQNs.
Change-Id: I4a80fbc1f4b037c8a4f91c8f28d2a96e47c66c47
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Let the transport access the controller options during
ctrlr_construct().
Change-Id: I83590c111e75c843685dd9315f0f08416168356d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
spdk_nvme_probe frees ctrlr when nvme_ctrlr_process_init is failed. But
ctrlr has already been freed while calling nvme_ctrlr_destruct. So
spdk_nvme_probe doen't need to free ctrlr.
Use the NVMe over Fabrics spec definitions for TRTYPE rather than the
internal library transport type.
Change-Id: Idead559a8f8d95274fc580d10e82033822e6eda8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Considering the process can be terminated in the cases like ctrl+c,
kill command or memory fault, the ref is tracked in the per process
structure spdk_nvme_controller_process and whenever there is other
process attaches or detaches the controller, a scan will be issued
to cleanup those unexpectedly exited processes.
Change-Id: Ib4f974f567a865748d42da4ead49edd383dfc752
Signed-off-by: GangCao <gang.cao@intel.com>
Function pointers will not work for the DPDK multi-process model (they
can have different addresses in different processes), so define a
transport enum and dispatch functions that switch on the transport type
instead.
Change-Id: Ic16866786eba5e523ce533e56e7a5c92672eb2a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The initialization of dev_addr was replaced with probe_info.pci_addr,
but its use in spdk_pci_addr_compare() wasn't replaced to match.
Fixes commit fcb00f3780 (nvme: expand
probe information to a struct).
Change-Id: Ic4c273d2aa0bf1f9e3e1527f3ab09d3c019158cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
spdk_nvme_probe() will now provide a struct spdk_nvme_probe_info to the
probe and attach callbacks in place of the PCI device pointer.
This struct contains the useful information that could be retrieved from
the PCI device during probe.
The goal of this change is to allow expansion of the probe information
in the future when other transports (specifically, NVMe over Fabrics)
are added that do not necessarily use PCI addressing or device IDs.
Change-Id: I59a2a9e874e248ce5fa1d7f4b57c8056962ff3cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
- Split the part that gets a PCI device's address into its own function,
spdk_pci_device_get_addr(). This is useful outside of the comparison
function and is orthogonal to comparing addresses.
- Make the comparison function take two addresses instead of a device
and an address. The more general form will be useful with addresses
that are not directly associated with a device. Because of this, also
rename the function from spdk_pci_device_compare_addr() to
spdk_pci_addr_compare().
- Return a signed value similar to strcmp() so that addresses can be
ordered, not just compared for equality.
Change-Id: Idf304454af09ea57f1e1d5dc3a39b077378cecad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the transport ctrlr_construct callback responsible for allocating
its own controller.
Change-Id: I5102ee233df23e27349410ed063cde8bfdce4c67
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow factoring out PCIe-specific code into a swappable
transport so that NVMe over Fabrics host support can be added.
Change-Id: I4df74dd268d655e3b36e8d6114ebe7d79a24844d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch also drops support for automatically unbinding
devices from the kernel - run scripts/setup.sh first.
Our generic pci interface is now hidden behind include/spdk/pci.h
and implemented in lib/util/pci.c. We no longer wrap the calls
in nvme_impl.h or ioat_impl.h. The implementation now only uses
DPDK and the libpciaccess dependency has been removed. If using
a version of DPDK earlier than 16.07, enumerating devices
by class code isn't available and only Intel SSDs will be
discovered. DPDK 16.07 adds enumeration by class code and all
NVMe devices will be correctly discovered.
Change-Id: I0e8bac36b5ca57df604a2b310c47342c67dc9f3c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Rather than forcing the NVMe library user to pass a specially-allocated
block of memory (e.g. rte_malloc() in the case of the default
nvme_impl.h), just make the NVMe library allocate a suitable buffer
itself and copy to/from the user buffer as needed.
The fast path I/O functions still require special rte_malloc()
allocations, since we don't want to add an allocation and copy to the
I/O critical path.
Change-Id: I7fe88c0ba60c859a33bbe95b7713f423c6bf1ea8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
assert is part of the C standard library and is available
on any platform we'd consider porting to. Don't put a
wrapper around it.
Change-Id: I0acfdd6a8a269d6c37df38fb7ddf4f1227630223
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
pthreads are widely supported and are available on any
platform we currently foresee porting to. Use that API
instead of attempting to abstract it away to simplify
the code.
Change-Id: I822f9c10910020719e94cce6fca4e1600a2d9f2a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is a step towards enabling sharing SPDK NVMe
device access from multiple processes using DPDK's
multi-process framework.
Change-Id: I57d5eec158b42addc1036bd2583596471a467a95
Signed-off-by: GangCao <gang.cao@intel.com>
This will allow removal notifications to be propagated to the library
user (e.g. for hotplug).
The callback is currently unused, but this at least prepares the API for
the future hotplug support.
Based on a patch by Dave Jiang <dave.jiang@intel.com>
Change-Id: I20b1c2dbf5e084e0b45a7e51205aba4514ee9a95
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For those controllers which can support end-to-end data protection
feature, add the support in the driver layer.
Change-Id: Ifac3dd89dec9860773c850416a6116113a6ce22a
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
All children must be removed from a parent request before the parent is
freed.
Change-Id: I073ff0e9c5bcdd6181d90b918bfe4cce054f6c0b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename nvme_remove_child_request() to nvme_request_remove_child() and
move it next to nvme_request_add_child() to make the symmetry clear.
Change-Id: I78747c44ab3db1a656b33555a45f634dc5a55b31
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch is used to add a nvme_request remove child
helpler function
Change-Id: I1e5bb228d53333ca3601f4ae30fcd801ea39e532
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
This makes it easier to find the larger doc comments that produce separate
pages.
It also allows removing the lib/nvme directory from the Doxyfile, so
only the public API headers are used to generate documentation.
Change-Id: I8c46edb8067a91dda5b23fb0864efd3dd8aaeba5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Provide a new structure, spdk_nvme_ctrlr_opts, to let the user modify
the default controller initialization options during probe/attach.
Currently, only the number of queue pairs can be modified in this way;
other options will be added later.
Change-Id: Ie27b9429291d93a9353c0d820f0ad467d3b0e7cb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The previous method for registering I/O queues did not allow the user
to specify queue priority for weighted round robin arbitration, and it
limited the application to one queue per controller per thread.
Change the API to require explicit allocation of each queue for each
controller using the new function spdk_nvme_ctrlr_alloc_io_qpair().
Each function that submits a command on an I/O queue now takes an
explicit qpair parameter rather than implicitly using the thread-local
queue.
This also allows the application to allocate different numbers of
threads per controller; previously, the number of queues was capped at
the smallest value supported by any attached controller.
Weighted round robin arbitration is not supported yet; additional
changes to the controller startup process are required to enable
alternate arbitration methods.
Change-Id: Ia33be1050a6953bc5a3cca9284aefcd95b01116e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This field is write-only in the current code; the NVMe library does
not track timeouts on requests.
Change-Id: I50e53bb3c299bf16912c48be8aad3eec829154af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
When multiple NVMe controllers are being initialized during
spdk_nvme_probe(), we can overlap the hardware resets of all controllers
to improve startup time.
Rewrite the initialization sequence as a polling function,
nvme_ctrlr_process_init(), that maintains a per-controller state machine
to determine which initialization step is underway. Each step also has
a timeout to ensure the process will terminate if the hardware is hung.
Currently, only the hardware reset (toggling of CC.EN and waiting for
CSTS.RDY) is done in parallel; the rest of initialization is done
sequentially in nvme_ctrlr_start() as before. These steps could also be
parallelized in a similar framework if measurements indicate that they
take a significant amount of time.
Change-Id: I02ce5863f1b5c13ad65ccd8be571085528d98bd5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The parent field is in the cache line of nvme_request that is only
supposed to be accessed for split (child) I/Os. All accesses to parent
are done from child-specific calls now, so it does not need to be
initialized in the common case of a non-split I/O.
nvme_request_add_child() will set parent when splitting occurs.
Change-Id: Ib86c16ba1ea2ce32f62079831101da2a099047af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename all functions with a spdk_ prefix, and provide enough of an API
to avoid apps needing to #include <pciaccess.h>.
The opaque type used in the public API for a PCI device is now
struct spdk_pci_device *.
Change-Id: I1e7a09bbc5328c624bec8cf5c8a69ab0ea8e8254
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The new probing API will find all NVMe devices on the system and ask the
caller whether to attach to each one. The caller will then receive a
callback once each controller has finished initializing and has been
attached to the driver.
This will enable cleanup of the PCI abstraction layer (enabling us to
use DPDK PCI functionality) as well as allowing future work on parallel
NVMe controller startup and PCIe hotplug support.
Change-Id: I3cdde7bfab0bc0bea1993dd549b9b0e8d36db9be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Also add a space between Copyright and (c).
The copyright year can be determined using git metadata.
Also remove the duplicated "All rights reserved." - every instance of
this line already has a corresponding "All rights reserved" immediately
below it, except for examples/ioat/kperf/kmod/dma_perf.c, where I have
added it manually.
Performed using this command:
git ls-files | xargs sed -i -e 's/Copyright(c) \(.*\) Intel Corporation. All rights reserved./Copyright (c) Intel Corporation./'
Change-Id: I3779f404966800709024eb1eb66a50068af2716c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This cleans up the I/O splitting code somewhat.
It also moves the SGL payload function pointers up into the hot cache
section of struct nvme_request without pushing the other important
members past the cacheline boundary (because payload is now a union).
Change-Id: I14a5c24f579d57bb84d845147d03aa53bb4bb209
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For the purpose to support different types of input scattered payloads,
such as iovs or scattered list, we define common method in the NVMe
driver, users should implement their own functions to iterate each
segment memory.
Change-Id: Id2765747296a66997518281af0db04888ffc4b53
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Most devices today support far fewer than 1024, but this is a
more reasonable default upper limit than the spec-defined 64K.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia8a6d80c3a5aa181f27c8354758c6ca468013d92
Fix all of the uses of __thread so they are at the beginning (similar to
e.g. static).
Don't actually enable -Wold-style-declaration, since clang doesn't
understand that.
Change-Id: I0dcbb758143eab90fc978334c8f256c6602cc4cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename the nvme_free_request macro to nvme_dealloc_request to match
nvme_alloc_request and add a wrapper function to nvme.c so that the
macro contents are only expanded once.
The DPDK nvme_impl.h uses rte_mempool_put(), which generates a large
amount of code inline. Moving this macro expansion to a wrapper
function avoids inlining it in the multiple places nvme_free_request()
gets called, most of which are error handling cases that are not in the
hot I/O path.
Change-Id: I64ea9c39ba47e26672eee8d5058f1489e07eee5b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is the only place nvme_request_add_child() is used, so move it
nearby and make it static to allow the compiler to inline it.
Change-Id: If4a7e17fde0b0272e1d4432c1dcedbec27c25371
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_dump_command is totally unused aside from the unit test.
nvme_dump_completion was used in qpair, but it can be replaced with the
equivalent nvme_qpair_print_completion.
Also added the missing nvme_completion fields to nvme_qpair_print_completion
that had been printed by nvme_dump_command.
Change-Id: Ia5ee66f3553df06febe8f465d42e49a84c555dd2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
