This function returns a pointer to the PCIe I/O registers for a controller
or NULL if unsupported for this transport.
Used for PCIe only, other transports return NULL.
Use with caution.
Signed-off-by: James Bergsten <jamesx.bergsten@intel.com>
Change-Id: I849f9de9ad259a65b1eef9c1237345eb7195b9bf
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452927
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This transport function is a complete nop now, so
remove it.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5cc6ac75795a3cf5311f24e2ac293fb53d4b9f8c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453487
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This will allow us to move more of the reset-related
functionality to the common layer, as part of enabling
resets for fabrics controllers.
The transport qpair_enable and qpair_fail functions
acted similarly - so those are both removed now and
replaced with this new qpair_abort_reqs function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9486630ad5b807239b0b5bcde50e8cfd313695d3
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453486
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Upcoming patches will move the actual is_enabled
logic to the common layer as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9fd4d8712280295f57134ad66f8ccbfe9736d30a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453484
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We submit AERs to all controllers - both pcie and
fabrics. But currently we only manually abort the
aers when disabling the qpair for pcie. Make this
common instead by creating a new transport function
for aborting aers.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I1e926b61b8035488cdc6e8cb4336b373732f985e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453482
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This better explains what the function is doing,
and makes the name more general so we can use it
for the adminq as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6b55761cb141a9a79cdef876be47995d8813b312
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453480
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Send the given NVM I/O command, I/O buffers, lists and all to
the NVMe controller.
This is a low level interface for submitting I/O commands directly.
This can only be used on PCIe controllers and qpairs.
This function allows a caller to submit an I/O request that is
COMPLETELY pre-defined, right down to the "physical" memory buffers.
It is intended for testing hardware, specifying exact buffer location,
alignment, and offset. It also allows for specific choice of PRP
and SGLs.
The driver sets the CID. EVERYTHING else is assumed set by the caller.
Needless to say, this is potentially extremely dangerous for both the host
(accidental/malicionus storage usage/corruption), and the device.
Thus its intent is for very specific hardware testing and environment
reproduction.
Signed-off-by: James Bergsten <jamesx.bergsten@intel.com>
Change-Id: I595fe02fe0dfa9c3ceba1ac116b6900357b02d2c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451994
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The mode of dmb oshld can guarantees cpu sequential execution,
which has less impact on performance.
Change-Id: If30b6a682a2216eecd1da039267ed4f5471afc38
Signed-off-by: h00448672 <heyang18@huawei.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446827
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is an attempt to fix device hotremove with VFIO.
A soft device hotremove request through sysfs [1] would
currently just block until the SPDK process manually
releases that device - e.g. upon an RPC request.
VFIO won't get unbound from the device untill userspace
releases all its resources. VFIO can signal a pending
hotremove request by kicking any file descriptor provided
by the userspace - and DPDK does provide such descriptor -
but SPDK does not listen on it.
DPDK does offer handy API to listen and in this patch
we make use of it inside our env/pci layer. Within
a DPDK callback we set an internal per-device hotremove
flag, which upper-layer SPDK drivers can poll with a new
env API - spdk_pci_device_is_removed().
The VFIO hotremove event will be sent to primary
processes only, so that's where we listen.
We make use of this new API in the NVMe hotplug poller,
which will process it just like any other supported
hotremove event.
Fixes#595Fixes#690
[1] # echo 1 > /sys/bus/pci/devices/<bdf>/remove
Change-Id: I03d88271c2089c740e232056d9340e5a640d442c
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448927
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This avoids a data dependent load to find which
callback to call in the completion path.
Change-Id: Ifa20790a7af3332a74bc45037e589668744af797
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450558
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The same information can be obtained by checking the req
field for NULL.
Change-Id: I9689ee0be33537fd6d3f35d8c3710fabab5e1928
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450557
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
When copying the NVMe command from the request to the actual
submission queue slot, use a non-temporal move instruction.
The submission queue slots are never read by software - only
written to. So don't pollute the CPU cache with their contents.
Change-Id: I112f721abfac03bd7b33ec9ddf783d4bf2952b42
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450193
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
There are several places where we have the tracker
pointer, yet we go find the tracker again by getting
the tr->cid and using that index to find the tracker
again in the qpair's array. That's really silly.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I54acd642a2c9821f2b95e17563904b859495081a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450308
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Logically we should complete any I/O in the completion queue and
abort the rest of the I/O after we delete the submission queue
and completion queue, so that we would not lost any complete I/O.
We alse should complete I/O and abort I/O before destroy I/O qpair
even though the ctrlr/device has been removed.
Change-Id: Ieb28ad7b4a3a7be553f70178b29ca870b5413191
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449316
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Similar to recently added nvme_pcie_qpair_ring_sq_doorbell.
Prepares for using this code for coalescing cq doorbell
writes in the future.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I565e103acf73b3d305e72a4440e1cc678c95faa0
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448871
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The phys_addr param in spdk_*malloc() is about to be
deprecated, so use a separate spdk_vtophys() call to
retrieve physical addresses.
This patch also adds error checks against SPDK_VTOPHYS_ERROR.
The error handling paths are already there to account for
spdk_*malloc() failures themselves, so reuse them in case
of vtophys failures.
Change-Id: I377636e66b8c570d013c1bb2021f04bce4e6c0ce
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/416998
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Try to group data members that are used often into the
same cache lines. We still need to find more space in the second
cache line of spdk_nvme_pcie_qpair so that the important
parts of spdk_nvme_qpair fit.
Change-Id: Ib936cb2b1acc722de7ec313d6faa3812aacde394
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447968
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Don't touch the shadow doorbells if it isn't necessary.
The flag could be combined into a bit mask with other
flags in a future patch.
Change-Id: I9ffd16468d29f0f0868cf849f7fece327eb6a294
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447967
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Avoid ringing the submission queue doorbell until the
call to spdk_nvme_qpair_process_completions().
Change-Id: I7b3cd952e5ec79109eaa1c3a50f6537d7aaea51a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447239
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This is going to get called from two places shortly.
Change-Id: I2c67e719c91887987e6e65c5c0c384bed0431409
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448311
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Move req->submit_tick assignments from specific transports to generic
qpair code.
Check whether submit_tick has been assigned before doing the actual
assignment, because a request may be submitted several times and the
original submit_tick shouldn't be covered.
Change-Id: I2de8018dc21763eb5a19bb9d48dfbdef764b036e
Signed-off-by: lorneli <lorneli@163.com>
Reviewed-on: https://review.gerrithub.io/c/444702
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Add a memory barrier for arm64 to prevent possible reordering
of tracker and cpl access,
because arm64 has less strict memory ordering behavior than x86.
Change-Id: I0a8716f7bfeffb0bbce27ee3174e214c8e4566b4
Signed-off-by: heyang <heyang18@huawei.com>
Reviewed-on: https://review.gerrithub.io/c/442964
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Existing NVMe driver uses a global list g_nvme_init_ctrlrs
to track the controllers during initialization, and internal
function will start each controller in the list one by one
until the list is empty. We introduce a probe context
and move the global list into the context, with the context
we can enable asynchronous probe API in the next patch, also
this can enable parallel probe feature.
Change-Id: I538537abe8c1a4a82fb168ca8055de42caa6e4f9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/426304
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Currently infrequent cases in request completion path are marked as
unlikely. This patch applies that to submission path.
These cases are infrequent and marked using unlikely marco:
a. The sq tail reaches the end of queue.
b. The sq tail equals to sq head. (never happen if FW runs correctly)
c. The qpair is admin queue.
Change-Id: I8b873a18615788f2efbf7c683aad710c7007a082
Signed-off-by: lorneli <lorneli@163.com>
Reviewed-on: https://review.gerrithub.io/c/443451
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Makes the code slightly more readable.
Change-Id: Iebf8fb07bceacf433d4bdad0a30419a3faab7eee
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439370
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We use those values in various places in SPDK,
so let's define them in a single place now.
Change-Id: Iad9a5745d69166a6e6032370d4e5a0e604914e45
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439369
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This follows the same trend as the mem_map APIs.
Currently, most of the spdk_vtophys() callers manually
detect physically noncontiguous buffers to split them
into multiple physically contiguous chunks. This patch
is a first step towards encapsulating most of that logic
in a single place - in spdk_vtophys() itself.
This patch doesn't change any functionality on its own,
it only extends the API.
Change-Id: I16faa9dea270c370f2a814cd399f59055b5ccc3d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438449
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
As a part of cleanup they're replaced by a device-agnostic
attach API, which is easier for us to manage.
Change-Id: I7558590e41e5c580a130a6aba7ae4f7dcff58da8
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/436478
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
With various possibilities to leak the rte_pci_device in the
primary process, we could technically construct the controller
in secondary. The nvme stack is not prepared for this and
will fail to initialize the device, but will still leak the
device object memory.
This patch adds an extra check to prevent any controller from
being constructed in secondary process.
Change-Id: I772f42b541c5db53310362b6595cebf9a30e8491
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434407
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Explicitly attaching a PCI device with spdk_pci_device_attach()
bypasses any kind of blacklists and should be only used
on a user request. Hotplug uevent is certainly not a user
request and should respect the blacklist, hence it's now
changed to call spdk_pci_enumerate() to probe new devices.
The enumeration callback will reject devices other that the
one we got hotplug request for, so no behavior is changed
in that matter.
This patch also fixes undefined behavior caused by reading
unitialized struct nvme_pcie_enum_cb;
Change-Id: I1399fbdd426152a13ed75c85a52bc7f0491ce287
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/433867
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The spdk_dma_zmalloc guarantee about physical memory contiguity
is about to be removed soon. For hardware rings that require
physical memory or IOVA contiguity we will now enforce hugepage
alignment and size restrictions to make sure they occupy only
a single hugepage.
Change-Id: Iebaf1e7b701d676be1f04a9189201c5d89dad395
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/418547
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
A few open-coded sequences equivalent to SPDK_CONTAINEROF() were
scattered around; replace them with the macro from spdk/util.h.
Change-Id: I95c6e6838902f411420573399ced7c58c2e4ef84
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/418126
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The spdk_dma_zmalloc guarantee about physical memory contiguity
is about to be removed soon. A single tracker is page size
aligned and is exactly one page big, so it is physically
contiguous, but we can't assume an array of those is physically
contiguous as well.
Change-Id: I3aa4d14dd677601c30aa2d8f15197886d6c46e58
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416840
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
There's no need to split a bufffer if it's physically
contiguous. We can now merge buffers that would be
previously split by the nvme_pcie driver and also
separate SGEs provided by the user that happen to be
physically contiguous.
Change-Id: I9c9de31d52a9dc9e384806555cb94609aff0ccf3
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/417061
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows NVMe PCIe devices to be used with
physically discontiguous I/O payload buffers.
So far this is just a dumb splitting which
doesn't check for physical contiguity. This is
improved in a subsequent patch.
Change-Id: I0ecc443149225eaa0e4156ddda78613bcf034406
Suggested-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/417060
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Fix issue #313.
For multi-process test scenarios, the secondary process may access
controller’s CSTS register in the shared ctrlr list. For this situation,
all the controllers are already in the primary shared ctrlr list, but
then each controller is added one by one in the secondary process, so the
secondary process may access CSTS before it is remapped for the BAR space.
In the rpc_config.sh test case, the spdk_nvme_ctrlr_get_regs_csts function
will be called in _nvme_pcie_hotplug_monitor function before calling
spdk_pci_nvme_device_attach. This step caused the secondary process iSCSI
Target access CSTS before it is remapped for the BAR space.
Change-Id: Ifd62c38adf8624f9877a9a2f965ca4db28839d99
Signed-off-by: Liang Yan <liang.z.yan@intel.com>
Reviewed-on: https://review.gerrithub.io/412594
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Factor qpair destruction function so that we can put common
resource release together in future.
Change-Id: I44139947820c2a384b745ae2673799f1b736369c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/412604
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This will be used in other transports as well.
Change-Id: I05026b0dfea2647d61a173379aca368ca48a2f52
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413864
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If requests are in flight at the time a timeout callback is configured,
we can't retroactively get the submission time of those requests; treat
them as not having a timeout.
Change-Id: Ic589e874a0f2c8c7f4ce352afa7c3aea33b01fae
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413863
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This is the first step toward timeout handling for other transports.
Change-Id: I386dd990f667d449e94ba4bcedaa3435743755fd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413862
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This was partially fixed in commit ddeaeeec19 ("nvme: Only check
timeouts on requests from the same process"), but the function that
calls nvme_pcie_qpair_check_timeout() was also erroneously filtering out
the admin queue. Restore the original behavior of checking all queue
types.
Change-Id: I26a44ff5eb772735d314ce7b8322ba9222675911
Fixes: 31bf5d795e ("nvme: make timeout function per process")
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/411628
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
When the shadow doorbell feature is enabled, no actual MMIO takes place
when a shadow update is sufficient; tighten the bounds of the updates to
g_thread_mmio_ctrlr in the two doorbell update locations so that we only
need to touch the thread-local variable when actual MMIO access is
required.
Change-Id: Ida974bec33f56cbb9f7d3611f483c6975ec773ab
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413856
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Rather than storing nvme_payload::type explicitly, use the SGL reset
function pointer as an indicator: if reset_sgl_fn is non-NULL, then the
payload is an SGL type; otherwise it is a contiguous buffer type.
This eliminates the one-byte type member from struct nvme_payload,
making it an even 32 bytes instead of 33, allowing the removal of the
awkward packing inside struct nvme_request.
Change-Id: If2a32437a23fe14eb5287e096ac060067296f1dd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413175
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This will simplify upcoming patches that change the way nvme_payload
stores its type.
Change-Id: Idf0a5b8dfd7d66a10f89254d2c5c54fee2968a43
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413173
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
