Amortize the writev syscall cost by using the writev_async socket API.
This allows the socket layer to batch writes into one system call
and also apply further optimizations such as posix's MSG_ZEROCOPY
when they are available. As part of doing so we remove the error
return in the socket layer writev_async implementation for sockets
that don't have a poll group.
Doing so eliminates the send queue processing.
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Change-Id: I5432ae322afaff7b96c22269fc06b75f9ae60b81
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475420
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Not inlining all host to controller operations breaks the target within
the context of fused commands. This issue was discovered when enabling
the compare-and-write fused command. Only the write command buffer was
being inlined which caused the write to jump the compare in the
transport specific state machine on the target side before our fused
command checks in the generic code.
Change-Id: I9e52ae6160e01ffd36d20429ffc8459491c729ef
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482001
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Update error handling of spdk_nvme_wait_for_completion_robust_lock to
differentiate cases when request is completed (possibly with error)
or polling was aborted by transport/device error
The function returns 0 on success, -ECANCELED if transport/device
error occurred and -EIO if the request is completed with error
Change-Id: Ibf7f3c330317af0d8f27ba9cd10d8b773f6a796b
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481529
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Update error handling of spdk_nvme_wait_for_completion_timeout to
differentiate cases when request is completed (possibly with error)
or polling was aborted by timeout or transport/device error
The function returns 0 on success, -ECANCELED if transport/device
error occurred or operation timed out and -EIO if the
request is completed with error
Change-Id: I314f40d1acaa6cfa9b88e5417b1ee2c9801bbbd6
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481528
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
For admin as well as for IO operations the ioctl should return
NVMe status code.
Change-Id: Icf8efde674b847fb03e5532f47c00b1c2cdcf328
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477492
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Purpose: With this patch,
(1)We can support using different sock implementations in
one application together.
(2)For one IP address managed by kernel, we can use different method
to listen/connect, e.g., posix, or uring. With this patch, we can
designate the specified sock implementation if impl_name is not NULL
and valid. Otherwise, spdk_sock_listen/connect will try to use the sock
implementations in the list by order if impl_name is NULL.
Without this patch, the app will always use the same type of sock implementation
if the order is fixed. For example, if we have posix and uring together,
the first one will always be uring.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: Ic49563f5025085471d356798e522ff7ab748f586
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478140
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Now that we have a more flexible function table strategy for
transports, we can get rid of some of the wrapping we were doing
to match the macro definitions exactly.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I12c868babfa7bd27dc8ed5e86d35e179f8ec984f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478874
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This new api function will enable us to work with custom transports.
This is needed to enable properly parsing and comparing custom transport
IDs that may all resolve to the same enum value.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I26aa3cb8f76f8273f564799d9b2af8041ea0d219
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478752
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
For custom transports, we should use a range outside the spec value
to identify them.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I82b29c349e143b8906f79ce2de818def116a3fe4
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478747
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The trtype should be stored as both an enum and string. This is intended to
help pave the way for pluggable NVMe-oF transports.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I6af658d7a17c405e191ff401b80ab704c65497e7
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478744
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
This new function allows specifying some of the extra
cdw10, cdw11 and cdw14 bits added in the more recent
versions of the NVMe specification.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I08a7840ad066b08fe557a2e7b974df491646978f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479737
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
This function returns information if compare
operation is supported by device.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I321e9bf6d146ac8d14ea4549cb4380735b30be6f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477925
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This patch adds first_fused_submitted field in spdk_nvme_qpair
structure which is used for postponing ringing a doorbell for
fused commands.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Ibfc43931891ebaadbafa4895c05af9f228440210
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477024
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
For nvme Opal, most of the commands are a combination
of security send and receive. There are cases that
application send the security send command and is
shutdown before sending security receive command.
In these cases, when the application restarts, an opal
command will send security send command again and will
cause command sequence error.
Thus, we do a receive first and memset the response
buffer before security send and receive.
Change-Id: Iba02c8074457919816ca576cd00ef9eee6d6dadf
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472591
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: yidong0635 <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
nvme_rdma_register_rsps returned ENOMEM for all failure cases. All of
them are not directly related to shortage of memory. Every point of
failure now sets relevant return code.
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: Ia340f6c6fd3a68d8c34acfefc2c9224ffcdcad3f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477302
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
RDMA work requests generated between two calls to NVMe RDMA QP
processing function are chained into a list and then posted together
to a queue in next call to processing function.
Batching improves performance in scenarios with deep queues and heavy
load on CPU. But it may cause latency increase on smaller
loads. Batching is configurable with RPC methods and configuration file.
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I600bce78427eb7e8ed819bbbe523ad318e2da32b
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/462585
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In some real data center deployments, 100ms is not enough. Increase
the timeout to 1 second.
Change-Id: I8195a1c1e987b7eff2d8541509f79381be32ed4b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478638
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: yidong0635 <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
For the requests which don't have children requests, SPDK may queue them to
the queued_req list due to limited resources, in the completion path, we
may resubmit them to the controller. When the controller was removed
the submission path will return -ENXIO and we will free the requests directly,
so the callback will not be trigerred for these requests. Here we added a
flag to indicate the request is from queued_req list or not, so for the failure
submission, we can triger user's callback.
Fix issue #1097
Change-Id: I901ac81733c2319e540d24baf5b8faa1c649eb35
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477754
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Use a lookup table with function pointers to build
the request depending on the payload type and SGL support.
This change helps to remove several if/else branches
Change-Id: I506e4290efc218be68fc8cfda4835b242a99aa77
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478191
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Command with cns SPDK_NVME_IDENTIFY_ACTIVE_NS_LIST is issued during
controller initialization and if the controller supports SGL,
this command will be built as a contig SGL. This leads
to a failed completion with the following status:
INVALID FIELD (00/02) sqid:0 cid:95 cdw0:0 sqhd:0004 p:1 m:0 dnr:0
The first identify command SPDK_NVME_IDENTIFY_CTRLR passed since
it was built as a PRP command - we didn't know that the controller
supported SGL at that time. Fix - do not build SGL requests
for admin qpair
Change-Id: I72ab7fe33c03e60ea9f20a9c8afd7c79c40843aa
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478320
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Purpose: To avoid the hang if there is no response
from the target.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: Ib68a9e4c1a28436af2b2ae65891de04067e3dc7d
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477121
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
To address the error message:
SPDK_ERRLOG("Unable to resubmit as many requests as we completed.\n");
Reason: The "reaped" variable is used to caculate the free slots
of rdma_reqs after calling the nvme_transport_qpair_process_completions.
And we should correctly caculate the free slots when the rdma_req is
really put.
If we caculate the slots more than we will have, we will trigger
the error print described above.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I269bdb63646eee6444d340b904882736c4cbca36
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477913
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: qun wan <qun.wan@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
mapping length is initialized with 0 and spdk_vtophys() returns
min(*mapping_length, cur_size) or 0. So length -= mapping_length has no
effect and req will be failed when nseg reaches NVME_MAX_SGL_DESCRIPTORS
Initialize mapping_length = request length
Change-Id: I9082866b7f8055d99fa6930a78335b3b0fdf9b2b
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477575
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Some version of firmware on Intel SSDs exhibit poor performance
with an IO queue size of 256 (the SPDK default). Add a quirk to
account for this - and use the same size as the Linux driver (1024)
instead.
If the user specified a controller option to explicitly set the
io_queue_size different than the default, the quirk will not take
effect. Note that users can still override the io_queue_size when
creating the qpair - this only changes the ctrlr's default
io_queue_size.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iff0f7ab9856fc18f226f7171e5109bb8fe430969
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/432227
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Previously, invalid io_flags would results in -ENOMEM being
returned to the user which was incorrect.
Change-Id: I53dd0fa8684cb36f3d124baa92244e2ed30e2527
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/476938
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
The hardware sgl format can describe large contiguous
buffers using just a single element, so it's more
efficient that a prp list even for a single memory
segment. Always use the sgl format.
Change-Id: I9c62582829f0d64dcd1babdbc48930ddb4d9e626
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475542
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
'delay_pcie_doorbel' parameter in 'spdk_nvme_io_qpair_opts' structure
was renamed to 'delay_cmd_submit' to make it suitable for every
transport. Old name is also kept for backward compatibility.
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I09ef8028133c4a3d4a5bbc5329ced1f065bcaa46
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475305
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
This patch attempts to solve naming conflict between
CUSE devices created by different SPDK instances.
Each NVMe device is enumerated by SPDK process from 0
up to 127. When process attempts to start cuse device
tries to set exclusive lock on temporary file
"/tmp/spdk_nvme_cuse_lock_<index>" and keep it until
device will be stopped.
If setting lock fails, index is incremented.
It prevents to use the same controller index from
few SPDK instances.
Change-Id: If744ac23f813bd992efb80ae2b61a1acefb5054c
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/474829
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This patch removes posibility to set cuse device path. Instead
"/dev/spdk/nvme*" path is used.
Change-Id: I7c3087772a3661eebe03fce21356c35cc8204b49
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/474598
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The nvme_transport_ctrlr_scan() may return failure while there are
multiple controllers, so the probe context's init_ctrlrs list may
not null for this case, so when free the probe context, let's ensure
there is no controller in the init_ctrlrs list. Also added a UT to
cover this case.
Fix issue #1095.
Change-Id: I4d9a10ad73cf00bbe159edd1f5b919797333feb6
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/476969
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
nvme_qpair_get_state fits more closely with the semantics in other
modules.
Change-Id: I6ea8e02abe27253d9b4d779a43ac1963be56356a
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/476920
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
The qpair state transport_qpair_is_failed is actually equivalent to
NVME_QPAIR_IS_CONNECTED in the qpair state machine.
There are a couple of places where we check against
transport_qp_is_failed and then immediately check to see if we are in
the connected state. If we are failed, or we are not in the connected
state we return the same value to the calling function.
Since the checks for transport_qpair_is_failed are not necessary, they
can be removed. As a result, there is no need to keep track of it and it
can be removed from the qpair structure.
Change-Id: I4aef5d20eb267bfd6118e5d1d088df05574d9ffd
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475802
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Now all registered producers should be stopped (unregistered) before
NVMe detach, otherwise NVMe controller cannot be safely detached.
This patch allows to stop all not unregistered io producers before
NVMe detach:
1. Callback to the "struct nvme_io_msg_producer" to stop producer
started on selected controller.
2. On nvme_io_msg_ctrlr_detach() if there's some unregistered producers,
stop all before freeing resources.
This approach also fixes issue with not to stop CUSE device when
NVMe controller is detached without unregistering producer (github
issue #1033).
Fixes#1033
Change-Id: Ia1ffef566bb745edb55c54d6786ea481a35bbefd
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/474273
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This patch uses lowlevel fuse functions to process messages to
eliminate the need to use signals to interrupt blocking read
operation in fuse_session_loop().
Fixes#1032
Change-Id: Ie9c9ea76cc135c383f5757864aa2d84ac9eb3da3
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/473233
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If the initiator dies without a disconnecting a qpair, the target can
possibly retain the state of the connection. In this case, it will
inform us that the connection is stale, and we need to try again.
Change-Id: I4d349c634aee59ce9ea4af795b07dd8649db56b3
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/473063
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
NVMe hotplug must be monitored in the primary process -
DPDK doesn't support trying to handle it in the
secondary process.
This issue was somewhat masked previously in secondary
processes, since usually it would just probe(NULL) which
meant probe all attached NVMe controllers. So in the
secondary process, we would probe just once, and create
the hotplug fd - it would never actually try to monitor
it.
But when explicitly specifying multiple trids in a
secondary process, probe would get called multiple
times. First time would be fine since it only creates
the hotplug fd. But second time would segfault since
monitoring for hotplug requires checking the DPDK-allocated
context which doesn't exist in the secondary process.
Fixes issue #1063.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2a9a91e222c206034293d90e30e3f598c8d7baa8
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475015
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is the first step in properly reconnecting after a hard power off
event.
Change-Id: I9739bffacd66ec6d9f8f1d376bf42291c84f90f2
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/473061
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This step is going to become more involved, so it's best to keep it in a
separate function entirely.
Change-Id: Iefa9860420edf28e858c4ed8aa932985c686cfd9
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/473060
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We should alert the upper layer when the qpair becomes unusable due to
qpair errors.
Change-Id: Icdee3b55a14441a60111f3bd7a44dceef93bbb09
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/474095
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Purpose: To remove the duplicated code.
Change-Id: Iab9989f9928698967533e45e7cffad4f09bde16a
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/473376
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Currently we have a mix of -1 and -EINVAL which
is confusing, especially since these types of failures
also result in the caller's callback routine getting
invoked.
While here, document this new -EFAULT return code for
all of the functions that could return it.
Fixes issue #797.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8dfbba0ec0b83db0f2ec055b15830981af1965df
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/473054
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: yidong0635 <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This is always the request pointer, so rename it for clarity.
Change-Id: Ifbda7db7787c65f0deb190a1e94f0676b2c0d99a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470530
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Users already have to poll the admin queue, so embed the io_msg
queue polling there to simplify the API.
Change-Id: I4d4d3be100be0798bee4096e0bbda96e20d2405e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472833
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Disconnecting qpairs from the admin thread during a reset led to an
inevitable race with the data thread. QP related memory is freed during
the disconnect and cannot be touched from the other threads.
The only way to fix this is to force the qpair disconnect onto the
data thread.
This requires a small change in the way that resets are handled for
pcie. Please see the code in reset.c for that change.
fixes: bb01a089
Change-Id: I8a39e444c7cbbe85fafca42ffd040e929721ce95
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472749
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
If we disconnect qpairs without taking the lock, we run the risk of
trying to double free qpair resources before they have been marked as
NULL.
For example, polling on one thread and calling
nvme_rdma_qpair_disconnect from one thread while doing an
nvme_ctrlr_reset on another thread. nvme_ctrlr_reset will call down to
nvme_rdma_qpair_disconnect on the same qpair and without any locking it
can result in trying to destroy the qpair resources multiple times.
Change-Id: I9eef6f2f92961ef8e3f8ece0e4a3d54f3434cff8
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472413
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Modifying the target_disconnect.sh test to include an example of
transport_id failover for an NVMe-oF controller.
Change-Id: I746ed737ab56c7dec6ee99e840c631ba46ee359e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472230
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Added RPC commands to register/unregister CUSE devices
to NVMe controllers:
- bdev_nvme_cuse_register
- bdev_nvme_cuse_unregister
Additionally two RPC now return CUSE device names:
- bdev_get_bdevs for namespaces
- bdev_nvme_get_controllers for controllers
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I69c4bf41ec8f78a7522894268a67dd733881712f
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472211
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
IO request handling implementation for CUSE namespaces:
- NVME_IOCTL_SUBMIT_IO,
a. SPDK_NVME_OPC_READ,
b. SPDK_NVME_OPC_WRITE,
Other ioctls related to the namespace:
- NVME_IOCTL_ID,
- BLKPBSZGET,
- BLKGETSIZE,
- BLKGETSIZE64,
Additionally NVME_IOCTL_ADMIN_CMD has been implemented
for namespace to send admin cmd to its controller.
Change-Id: Iaa2a5ee409b35c682ae8b2bd2309e8b67eb51981
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469691
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
NVMe character device implementation. This patch adds implementation
of IO producer using CUSE library. It allows to create nvme device
nodes in linux kernel for controller as well as for namespace and
process ioctl requests as usual from linux environment.
Both devices (controller and namespaces) are exposed as character
devices.
To compile NVMe CUSE module use "./configure --with-nvme-cuse".
Names for created CUSE devices can be retrieved using
spdk_nvme_cuse_get_ctrlr_name() and spdk_nvme_cuse_get_ns_name().
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I0fc9a9a1ef3c9c2b3112d07c2b4b1f8d49665ee1
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466917
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch allows to send IO requests from external module to the nvme
device.
External module should call nvme_io_msg_ctrlr_start() to start IO message
producer on the controller and enable sending messages.
nvme_io_msg_send() is used to send IO to NVMe driver thread context,
where passed function will be called. Allowing the external module to
issue IO as needed.
NVMe driver users should poll spdk_nvme_io_msg_process() to move forward,
sending IO from external module and process their completions.
Change-Id: Ie59abac69870c4e4daa50120c747f3b620395921
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471386
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This can be useful when trying to perform multipath failover at the
application level. However, the controller must be in the failed state
before calling this function.
Change-Id: I5403c0036fed5dd3600ee20592925297494ba8aa
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470699
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These will be useful helper functions for the trid modification code
that gets introduced later.
Change-Id: Ief73e3045710bf35c511794c19b4dfefb93018f1
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471780
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
While it is unlikely that a single qpair will be failed, it is important
to make it possible to reconnect a single qpair.
This function is also handy at the application layer when going through
a reconnect workflow. If we get -ENXIO from a qpair when we poll, we
will turn around and call this function. If we get -ENXIO from this
function, then we know the whole controller is failed and we need to do
a reset.
Change-Id: I6a8ea0ce27fce2f5fc0a5b3db05834acd68e6a39
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471417
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Previously, get locking range info will allocate
memory everytime. Thus calling this function multiple
times will cause memory leak.
Now, we allocate only if it is NULL. If it is not,
then we just memset to zero.
Change-Id: If048416a2056176f86206a33e2b5db210288fe4f
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472112
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Some nvme drives might take 6-7mins for
this operation. Thus, introduce async function
to avoid waiting.
Change-Id: Id48478aec653d3fb75a3c5ce75d4997284ed016c
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468916
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This will be the canonical way of informing the user that we have lost
the qpair connection somehow.
Also update all of the functions that will return -ENXIO to the user.
Change-Id: Ic6c7c2d0e07e9d3e857a3476bb6b91fb4b6454fa
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471416
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Failed is not a final state for either fabric or pcie controllers. We
have historically not allowed resets in the failed state, but we should.
Instead of checking for the failed state, we should check for the
removed state. If the controller is removed, then we cannot even attempt
a reset.
Change-Id: I2c1a3d85db84f84cd1895cbfaf16575c8b496155
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471415
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We no longer need the private function with a public wrapper.
Change-Id: I0d24dfb282461174729d3eb649c78ac27e42fc8d
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471552
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Also, adds a field to the generic qpair for future use in other
transports.
Change-Id: Ie5a66e7f5ebfec1131155fc07e3c671be814fb9b
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471414
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We shouldn't always fail the whole controller if we get a failure on an
individual qpair.
Change-Id: Id0c90af83e5231593a895be66e7a7de48939e240
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471660
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The way these two functions were separated previously represented a
pretty sserious bug when doing a controller reset.
If there were any outstanding requests in the rqpair, they would get
overwritten during the call to nvme_rdma_qpair_register_reqs and the
application would never get a completion for the higher level requests.
The only thing that we need to do in this function is assign the proper
lkeys.
Change-Id: I304c70646daf9b563cd00badba7141e5e8653aad
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471659
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
check_enabled had a couple bugs in it that made it unfriendly for enabling
I/O qpairs after a reset.
1. It was calling nvme_qpair_abort_queued_requests before setting the
enabled flag to true. For applications that submit new I/O in the
completion callback for old I/O, this means you enter an infinite loop
of submitting requests, and then immediately completing them. SO
instead, wait for the qpair to reset, then just submit those requests to
the lower layer.
2. It didn't check whether we were already in the middle of calling it,
so we could reenter function calls like
nvme_qpair_abort_queued_requests.
Also, now that we have a coherent state machine for qpairs, we can limit
the enabling to a specific state in that state machine.
Change-Id: Ie0b74819a6b16839965bced47c33dec967f725a8
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470256
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
As the todo states later on in the function, the discovery controller
should really be initialized through traditional methods, but it was
hacked in. For now, enable the admin qpair to get past the non-standard
nature of this controller.
Change-Id: I2cbf1cd47d7249ae3d12bcfc2e8d21e8fb98df7e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471779
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This function is identical between the two transports.
Change-Id: If50b781259f224eb2c21de7da14564e6ce487650
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471778
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
These will form the base of a little state machine for managing the nvme
qpair structure.
Change-Id: If6f6df38cc17221ac8fcb7d8c0d7e2e808897a99
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470534
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The driver has historically waited until we have to do a listen
before enabling the admin qpair. That is a very PCIe-centric mindset.
For fabric controllers, a lot of the early initialization operations such
as get_cc and set_cc are handled through the admin qpair so it should be
enabled before we begin the initialization process.
As a side effect of this cahnge, the internal API
nvme_ctrlr_enable_admin_qpair has been removed. It would have turned
into a one-liner.
Change-Id: Icd162657d01a85c227a3f20c295d0208e07ce44d
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471743
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
This wasn't being done in the previous case which meant that I/O qpairs
were not being moved to the connecting state when connecting for the
first time. However, to prepare the way for a coherent state machine for
nvme qpairs, we need to ensure that all qpairs go through the same
states.
Change-Id: I3cfe799a003acd926b24c107ab1461a96239c1bb
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471753
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
These functions are functionally equivalent. Just unify the way they
wait for completions so that they are completely identical and we can
merge them into a common function.
Change-Id: Id5d734b6ae613b3ac828d89853d986cdadfb211a
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471936
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Leaving these on the stack outstanding list can cause unnecessary
buildup. If we fail to post the request to ibv, then the upper layer
request will be freed immediately for reuse, but we will keep that
request in the outstanding queue at the RDMA layer.
Change-Id: Ib422dc9fcb50344ce7c01749f3e20ea9310fd5cb
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470255
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This reverts commit bc4e31d6b2.
This change was accidentally merged after it was decided to go with a
different architecture.
Change-Id: Ifc9d8b08bd1fcbc4ace8dd6fb4bd0014330916ed
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471144
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When doing a reset on an NVMe-oF target with active I/O qpairs, we need
to be able to submit fabrics commands on them in order to perform a reset.
Currently, resetting a fabric controller with any I/O qpairs active will
cause the reset to hang indefinitely.
Change-Id: Ic972a301390a4dd64adabedfe01aa4e5253e40b0
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469935
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The remove callback is a built in way of alerting the user application
that we have removed a controller. Once we fail a controller, we never
move it back out of that state so it is in essence removed.
Change-Id: Iaad6bef0994e9ddd5a424f6b83502f9191b2de49
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469637
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
My recent changes that introduced batching to queued request
resubmission also introduced a regression that can lead to reordering
requests before submitting them to the drive. This change prevents that.
We wait until inside the internal _nvme_qpair_submit_request function to
check for queued entries to avoid queueing a request that has children.
If a request that has children gets queued, when we process completions
and resubmit the parent, it will result in the children being submitted.
Since we only account for the number of requests we completed in the
last iteration, some of the child requests may be requeued out of order,
or worse, none of the child requests will end up being submitted to the
transport and they will all be queued behind previously queued requests.
Change-Id: I58e1c458c25fbf3f9f75364f05b1076b166a6212
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470890
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Making this structure available from the ctrlr allows us to call the
remove callback when the controller is failed/removed on transports
other than pcie.
Change-Id: I2c66dfef12b039c0d6daf7df83da745757818006
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469636
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This paves the way for doing multiple reconnect attempts before failing
the controller.
Change-Id: I1ff4ee6d41a5ffb47dd186d76793d670287c4783
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469934
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
By moving the contents of spdk_nvme_ctrlr_reset to a new internal
function, I am paving the way for providing two reset paths. One, which
can be used by the user as an external API function and which provides
the same legacy behavior. Specifically, that it will always fail the
ctrlr after an attempted reset, and a second, internal path, which will
be used by the qpair reconnect code which will defer failing the qpair
to the qpair code.
Change-Id: I9ec9df55c1fecc2f00476c175bcf988207c31257
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469933
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If we are to have multiple reconnect attempts, we have to control
whetehr the controller is placed in the failed state from outside the
reset function itself. This will allow us to fail the controller only
after all of our retries are exhausted.
Change-Id: Ia82e10325272f25b2b8527336dc3bc507c93b401
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469932
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Use the standard API function to fail the controller in all cases.
This patch, and the several following patches are aimed at creating a
mechanism for reporting up to the application layer that a controller is
failed and or removed. To do this, I use the reset_cb to inform the
upper layer that the controller is failed.
This also requires changes to how we handle a controller reset to
pave the way for doing optional reset retries in the libraries.
Change-Id: I06dfce08326c23472a1caa8f6efbac2fd1a720f2
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469635
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We were already passing up from each transport the number of completions
done during the transport specific call. So just use that return code
and batch all of the submissions together at one time in the generic
code.
This change and subsequent moves of code from the transport layer to the
genric layer are aimed at making reset handling at the generic NVMe
layer simpler.
Change-Id: I028aea86d76352363ffffe661deec2215bc9c450
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469757
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This will be handled by nvme_qpair_submit_request when it receives
-EAGAIN from _nvme_qpair_submit_request.
Change-Id: I5e76aae170c981df0cadaadcd5da1163c715006f
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470407
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
This patch series is aimed at preserving the order of qpair entries
when resubmitting queued requests. The hope is that we will make the API
fool proof and future proof against ever reordering any queued requests.
Change-Id: Ib20d61d3abaed637c9c305b75081947630190fd4
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470062
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
1. Log level change to info when checking support
2. Delete new lines
3. Enlarge the timeout seconds to 10min for revert
TPer as it sometimes need 6-7min for this operation.
Change-Id: I1b7e32917bd99c859f1515b07f2530669418f0db
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468915
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
The tailq and the requests all belong to the generic layer, might as
well put the queueing code there for better encapsulation.
Change-Id: Id5f08f798121b50a21044cfc61856999c50ca227
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469758
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This was being ignored, and can cause some problems when trying to reset
a defunt controller over a fabric.
Change-Id: I32c11a0e2df0e140e20f870fe0fb5b9045a567b3
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469638
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Previously we would just sit forever. preventing us from properly
attempting reconnects and timing out.
Change-Id: Id7386ab95cf75fd9ac972b44afa2719aad412f49
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469021
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This enables us to create a single file descriptor and a single event
channel to poll for completions. With that accomplished, we can easily
poll for events on the admin qpair each time we check it for
completions.
Change-Id: I8b901252510744a956bef12594d1e045715e002e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467549
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This prevents us from failing a reset and then trying to double put the
rqpair->cq which ends up causing seg faults.
Change-Id: If3e14a3d039b4b19cc587a7482157f4b23f8ee32
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469609
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
In most places, we are passing NVME_TIMEOUT_INFINITE as the
timeout_in_ms argument to nvme_ctrlr_set_state, presumably in an attempt
to specify an infinite timeout. However, nvme_ctrlr_set_state only
checked against 0 when setting the actual timeout, and we didn't have
any logic to check for overflow so we just ended up setting random
timeout_tsc values which changes the behavior of the
nvme_ctrlr_process_init function in several places.
So, change NVME_TIMEOUT_INFINITE to 0, and add some integer overflow
checking to nvme_ctrlr_set_state.
Change-Id: Ic9d0cc57ed153df30c3b20313c3742072a5f992d
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469485
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I9fb7a998f7c13ce53cba630a895e8e11cf5f4a1c
Signed-off-by: Benjamin Saunders <bsaunders@google.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467559
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
SPDK_ERRLOG lists the function name, so remove old references that
assume it doesn't and reprint the function name.
Change-Id: I69da6ca0a25bf0eda07d8dad52bcfadf964ac715
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469487
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Weighted Round Robin can be enabled for users, and users
can allocate different priority IO queues for different
purpose. For now we will enable this feature in the
NVMe driver first, following patches will enable this
feature in bdev layer.
Change-Id: I0f799236ca04eb85ef3c9f972ed63ff2718563ba
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466852
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
By splitting all cm_event handling into a single function, we can create
a single point of contact for cm_events, whether we want to process them
synchronously or asynchronously.
Change-Id: I053a850358605115362f424de55e66806a769320
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467546
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This is paving the way for additional changes to enable polling for
cm_events in the initiator.
For now, just present the same blocking API on top of the now polled
file descriptor. Later, we will change this API to be more useful.
Change-Id: I174dac028720f95c30100f6dc2ed49b5bb2a7e40
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467545
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
spdk_pci_device_claim() could create a file on the
filesystem that couldn't be deleted programatically.
It could only be overwritten - e.g. by another spdk
instance - but this didn't really work if that
another instance had less privileges and hence no
access to the previous file.
This is exactly the case we're seeing on our CI when
running SPDK as non-root. In general it's a good idea
not to leave any leftover files, so now we'll delete
the pci claim file when the spdk process exits.
spdk_pci_device_claim() used to return a file descriptor
that could be simply closed to "un-claim" the device.
It'll now return only a return code. The fd will be
stored inside spdk_pci_device and will be closed either
when user calls the newly introduced spdk_pci_device_unclaim(),
or when the device is detached.
We'll still need to clean up those files somewhere in
our test scripts (probably ./setup.sh cleanup) to
clean up after crashed processes or so - but we don't
necessarily want to run such scripts inside the autotest
whenever a non-root spdk is about to be started.
Change-Id: I797e079417bb56491013cc5b92f0f0d14f451d18
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467107
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If the CPU reorders the eventidx read before the shadow doorbell
write, it is indeterminate whether the controller will read the
updated shadow doorbell without an MMIO write. See
https://lkml.org/lkml/2018/8/14/1031 for details.
Signed-off-by: Benjamin Saunders <bsaunders@google.com>
Change-Id: I5aa08fdd5b32c7b81e8048ca6efe546318d80b5c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468188
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This reverts commit 6129e78d26.
When the initiator sends the discovery log page, if the log page
exceeds the size of its data buffer, it will break it up into
multiple log page commands with appropriate offsets. However,
supporting offsets in log pages is an optional feature in NVMe
and reported by the EDLP bit in the identify data.
This commit changed the discovery process to no longer send an
identify command prior to doing the discovery log page command,
so the values in the identify data are always 0. If the discovery
log page exceeds the size of the data buffer (4k), it will then
fail to send the second log page with an offset because it
believes the controller does not support the feature.
Revert this change to fix it. An identify should always be sent
as part of the discovery process. A test case is included in a
follow up patch the demonstrates the bug.
Reported-by: Zahra Khatami <zahra.k.khatami@oracle.com>
Reported-by: Akshay Shah <akshay.shah@oracle.com>
Change-Id: Iefd512a7521e0fea90541b3eb547671cfa816ea6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466819
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Since we use pdu->data_iovcnt to
build the iov in nvme_tcp_build_iovs, so
send out pdu has the maximal iov number
equals to: 2 + pdu->data_iovcnt,
so we change the comparison.
This makes sure that we can handle all the data
owned by one pdu.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I2b9258cc5716d706c0fa38af609726c439708768
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467207
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
This will be consistent with TCP and RDMA transport, and we will use
ctrlr->flags in nvme_ctrlr_init_cap() in next patch, the flags will
be cleared to 0 for now.
Change-Id: Ic360cd0c00d60c77452d19cdc1e7a32a5fc34df0
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466678
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Purpose: Prepare the further optimnization in the
target side whening receving pdu headers, we expect
to use zero copy.
Change-Id: Iae7f9106844736d7160d39d0af1f5941084422ec
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/465380
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Currently we *always* wait 2 seconds before starting
controller initialization during attach. This
works around an issue where some older Intel NVMe SSDs
could not handle MMIO writes too soon after a PCIe
FLR (which would be triggered when VFIO was enabled).
After further discussion with Intel experts, we know
the SSD models that exhibit this issue. So we can
quirk this so that only the older SSDs incur the extra
delay.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ieb408c24f6afd5bd5147d1c87239aa20f2d13511
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466064
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
WRITE_TO_RO_PAGE was incorrect and misleading. This
0x82 NVMe status code indicates a write to a read-only
range of LBAs. So modify the constant name and
associated usages to use WRITE_TO_RO_RANGE instead.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I993dbebb5acc2e685a0e99aa14084942ef79d659
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/465083
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
spdk_nvme_detach() will do the normal shutdown notification for
most cases, and it will take some time e.g. 2 seconds to finish
the process for PCIe based controllers. If users' environment
has several drives, each drive will call spdk_nvme_detach() one
by one, and the shutdown process may take very long time.
Since users know exactly what they would like to do for the next
step, so here we provide an option to users, users can enable it
to skip the shutdown notification process so that they can have
very quick shutdown process, and when starting next time, the
controller can be enabled again.
Change-Id: Ie7f87115d57776729fab4cdac489cae6dc13511b
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/463949
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
SPDK_NVME_DEFAULT_RETRY_COUNT is the default value for each controller, so
we can move it out from public header file, and change the value if users
provide a new one.
"NvmeRetryCount" was deprecated for a long time, so we removed the support
for this configuration option as well.
Change-Id: I187251cc1e5342abb4fce96727d06631b7c16a01
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464489
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We have defined NVMe controller initialization 'transport_retry_count' option, so
global 'spdk_nvme_retry_count' can be removed, we will remove the variable with
PCIe transport first, and make the retry count can be configured via RPC.
Change-Id: I4d54f78c8da2180d536635587e7291f44a57c4fb
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464472
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Skip token header length which varies for short,
medium and long atom.
Fix Issue #898
Change-Id: I2351193e5a43608495f3d816ff4e5932399a6312
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464502
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Purpose:
1 Do not caculated the psh_len every time.
2 Small fix, for ch_valid_bypes, and psh_valid_bytes,
we do not need to use uin32_t.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I9b643da4b0ebabdfe50f30e9e0a738fe95beb159
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464253
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The value of tcp_req->req was asserted after it was already
dereferenced. This patch fixes that.
Change-Id: I5eb01e88be09d41fb8e632c49d5a7ccf2315788f
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/462508
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Admin can enable user and add user to locking range.
Then the user can lock/unlock his range.
Change-Id: Ifc5a8cf5c6b5febeb59c86333981f0cf5b938500
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460891
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
There are many req leaks when a controller failure
occurs during submitting IO. It must free all of
the children before freeing the parent req.
If a part of the child req has been sent to the back end
and a part of the child req fails, removes the failed req
from the parent req and the parent req must be retained,
freeing the parent req after all of the submitted reqs return.
Change-Id: Ieb5423fd19c9bb0420f154b3cfc17918c2b80748
Signed-off-by: Huiming Xie <xiehuiming@huawei.com>
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/461734
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Existing children split functions defined in nvme_ns_cmd.c can
also be used in nvme_qpair.c to free children requests with error
paths.
Change-Id: I640b32884424709da67ee89ff780c2de45acc54c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/461372
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This change attempts to address the Trello request to decode I/O errors in
NVMe hello_world example.
See https://trello.com/c/MzJJw7hM/2-decode-io-errors-in-nvme-helloworld-example
As part of this change, spdk_nvme_cpl_get_status_string was declared
in nvme.h, and spdk_nvme_qpair_print_command and
spdk_nvme_qpair_print_completion were renamed and added to nvme.h,
allowing all three to used "externally."
To test the failing paths, two compile time defines were added to force a
write or read error (bad LBA) respectively.
As the example does a read after write, if the write fails, the example fails.
Signed-off-by: James Bergsten <jamesx.bergsten@intel.com>
Change-Id: Ib94b4a02495eb40966e3f49517a5bdf64485538a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457076
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The __sync builtin based implementation generates full memory
barriers on some non-x86 platforms. Replace it with C11 atomic
builtins can make:
·arm and ppc from full barrier to half barrier
·x86 code same as before
Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>
Change-Id: Ib6624ef8e45af497b9eced6ecfa7710bcc88a733
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/461590
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
nvme_transport_qpair_process_completions calls nvme_rdma_qpair_process_completions
There are some cases return -1 due to failure of "CQ errors".
Handle CQ polling failures by marking the controller as failed.
That a completion with an error will be treated as controller failed.
Requests will be aborted after retry counter exceeded. Otherwise, code will keep on
reporting errors without recovery.
This is to fix issue #850.
Change-Id: I0b324232310e107bf7fd5722aca54d402a19b14d
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460569
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
nvme_tcp_qpair_process_completions returns -1 on socket I/O
error. Unless the caller checks this return value (which
spdk_nvme_wait_for_completion_robust_lock currently doesn't),
on connection loss or any other fatal connection
error spdk_nvme_wait_for_completion will never exit the completion
check loop.
Change-Id: I92bb349beb071db312e6c31b84db2a7b51ec486c
Signed-off-by: Andrey Kuzmin <akuzmin@jetstreamsoft.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460657
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
SPDK NVMe-oF initiator driver could not transfer IO whose size is
more than 128KiB even if NVMe-oF target allows IO whose size is
more than 128KiB both for RDMA and TCP transport.
Some use cases need to transfer IO larger than 128KiB.
For RDMA transport, max_mr_size by ibv_query_device of RDMA devices
indicates the maximum size of a single memory region and is independent
from the actual I/O size, and is very likely to be larger than 2 MiB
which is the granularity we currently register memory regions.
Actually some RDMA NICs return UINT64_MAX for max_mr_size by ibv_query_device.
Hence use UINT32_MAX and let the generic layer use the controller data
to moderate this value.
On the other hand, for TCP transport, there is no limit for maximum IO
size and hence use UINT32_MAX.
Besides, for RDMA transport, max_sges should be the minimum of
max_sge got by querying RDMA devices and NVME_RDMA_MAX_SGL_DESCRIPTORS.
Hence do this change together in this patch.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Idc813afd3e525bf5f370c0fcd2623f9c146a5528
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459218
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Previously comparing the transport supported value and the target value
was done in RDMA transport layer. However this comparison should be
done in the generic layer like the maximum IO transfer size. Hence
change the comparison to do in the generic layer in this patch.
Besides, for MSDBD, the value 0 indicates no limit but we had handled
this as maximum number of SGS entries was 0 by mistake. This patch fixes
the bug together.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I54365cf114169b10180ec2c659f9c7302672674c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459574
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Using naming rules consistent with other related libraries is helpful
to ensure the quality as verified by this patch series.
This patch changes a few parts to use iov and iovcnt for SGL operations.
Besides, name of an array points to the head of the array and is
constant. So copying name of array to an another pointer is not
necessary and can be removed.
Change-Id: I2324f28126b3088098c1c767cf6c060f22c175c3
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455629
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
NVMe/TCP target had assumed the size of each iovec was io_unit_size.
Using nvme_tcp_pdu_set_data_buf() instead removes the assumption
and supports any alignment transparently.
Hence this patch moves nvme_tcp_pdu_set_data_buf() to
include/spdk_internal/nvme_tcp.h and replaces the current code to use it.
Besides, this patch simplifies spdk_nvmf_tcp_calc_c2h_data_pdu_num()
because sum of iov_len of iovecs is equal to the variable length now.
We cannot separate code movement (lib/nvme/nvme_tcp.c to include/
spdk_internal/nvme_tcp.h) and code replacement (lib/nvmf/tcp.c)
because moved functions are static and compiler give warning if
they are not referenced in lib/nvmf/tcp.c.
The next patch will add UT code.
Change-Id: Iaece5639c6d9a41bd35ee4eb2b75220682dcecd1
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455625
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Compilation Warning on fedora30.
In file included from nvme_ut.c:42:
/home/vagrant/spdk_repo/spdk/test/common/lib/test_env.c:517:17:
warning: The left operand of '>' is a garbage value
if (a1->domain > a2->domain) {
~~~~~~~~~~ ^
This is related to issue #822.
Change-Id: I2b61e821130b89af04db3c475e81d2e91a380a90
Signed-off-by: Hailiang Wang <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459923
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
nvme_tcp_pdu_set_data_buf() has been used to process C2H and H2C for
NVMe/TCP initiator.
In this case, NVMe/TCP cuts out the part of the input data buffer
and transfers the part, and repeats these cut and transfers until
the whole data buffer is transferred. NVMe/TCP uses two SGLs, and
use one to parse from the offset datao to datao + datal and another
to append from the offset 0 to datal.
However, the current nvme_tcp_pdu_set_data_buf() had used
data_length as not data length of this transfer but total length
of the whole transfers by mistake.
Recently DIF library updated to properly handle very similar
cases, and so this patch takes DIF library as a reference and
corrects the implementation.
The next patch will add UT code to verify the bug will be fixed.
The code size is pretty large and so UT code is separated.
Change-Id: Ibeed4de182b8b8740566e874e2757280dc21f9e8
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455623
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
This patch is the first patch of the patch series.
The purpose of this patch series is to correct the bug of
nvme_tcp_pdu_set_data_buf() when the multiple iovecs array is
passed, to share nvme_tcp_pdu_set_data_buf() between NVMe/TCP
initiator and target, and utilize nvme_tcp_pdu_set_data_buf()
not only for C2H and H2C but also in-capsule data in NVMe/TCP
target.
This patch is necessary to satisfy the second requirement, to
share nvme_tcp_pdu_set_data_buf() between NVMe/TCP initiator and target
because struct nvme_tcp_req and struct spdk_nvmf_tcp_req are different.
Four variables, iov, iovcnt, data_offset, and data_len are common,
and hence this patch changes the parameters of nvme_tcp_pdu_set_data_buf()
to accept them.
The bug is fixed in the next patch and tested in after the next patch.
Change-Id: Ifabd9a2227b25f4820738656e804d05dc3f874a5
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455622
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
spdk_dma_*malloc() is about to be deprecated.
Change-Id: I6c308ee546c28c479ceb903bc1749bf5209dc6fe
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448172
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: <uma.willpower@gmail.com>
The completion cb of outstanding_tr may submit new requeset to
the outstanding_tr list of the qpair, it's an endless loop.
We only abort the remaining outstanding trackers.
Fix#819
Change-Id: I342f52f4d1836f8ef620ef9e3add0b1986727282
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457755
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Adds fields to structure spdk_nvme_io_qpair_opts.
These fields allow specifying the locations of memory buffers used
for the submission and/or completion queues.
By default, vaddr is set to NULL meaning SPDK will allocate the memory to be used.
If vaddr is NULL then paddr must be set to 0.
If vaddr is non-NULL, and paddr is zero, SPDK derives the physical
address for the NVMe device, in this case the memory must be registered.
If a paddr value is non-zero, SPDK uses the vaddr and paddr as passed.
SPDK assumes that the memory passed is both virtually and physically
contiguous.
If these fields are used, SPDK will NOT impose any restriction
on the number of elements in the queues.
The buffer sizes are in number of bytes, and are used to confirm
that the buffers are large enough to contain the appropriate queue.
These fields are only used by PCIe attached NVMe devices. They
are presently ignored for other transports.
Signed-off-by: James Bergsten <jamesx.bergsten@intel.com>
Change-Id: Ibfab3939eefe48109335f43a1167082dd4865e7c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454074
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
According to the TP 8000 spec in Page 26:
Maximum Number of Outstanding R2T (MAXR2T): Specifies the maximum
number of outstanding R2T PDUs for a command at any point in time
on the connection.
This patch makes the current host driver implementation support one r2t.
We cleanup the code to do the right advertising to the target in the
icreq and avoid attempts to deal with multiple rt2s.
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: If06ad2e8bde31c2fd7e1c3739f651fb64040e3a9
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455750
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Or Gerlitz <gerlitz.or@gmail.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The current nvme_tcp_qpair_disconnect behaviour
is not exactly correct, we do not re-initialize
the state of some data structures of the tqpair.
And this caused the coredump.
Purpose: Fixes#808.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I4d2cad8fc0712dbebfc2f3e52373cbe3b9908bf7
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456755
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The length should be no larger than the remaining_size.
For example, The remaining_size(firstly, assigned by payload_size) is 128KB,
and user's sgl length is 1MB. Since we already split the I/O, so we should
not use the original length(1MB), but use the remaining_size.
Fix issue reported by: https://github.com/spdk/spdk/issues/808
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I0a7d0f2282c8ad0e253d8de7091b6c5b87018e9a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456760
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Previously, if the return value of nvme_tcp_qpair_process_send_queue
is not zero, we directly return but not continue receiving the pdu.
But this is wrong, we should only handle the case when the
return value is negative.
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I83453733f5a3e3350a0461b4cb0bc409fde32fea
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455899
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We will need to put the recently completed nvme_request
object on the qpair's STAILQ. We don't reference any
real data from the nvme_request in the completion path
since we've already stashed the cb_fn and cb_arg in
the nvme_tracker. But we will need to reference the
STAILQ_ENTRY to put it back in the qpair's STAILQ, so
prefetch that cacheline.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id76122afe4150c84a61fbe38bc874f10d606b3b3
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456673
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
There's no need to set this every time we allocate
a request.
While here, fix a typo near where we needed to modify
the unit test to remove the qpair assertion.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8af41a6c483415950f625d1ed2ef46088b75a622
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456270
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Request may be submitted several times via nvme_qpair_submit_request
function, such as request in queued_req queue being re-submitted.
With enabling timeout feature, nvme_qpair_submit_request compares
request->submit_tick to zero to check if this is the first submission
for this request. If true, record submit_tick for this reuqest.
So request->submit_tick needs to be set zero in allocation.
Change-Id: Ie3f420aa337802c5ad3962c3fdcd680dec1ccdcb
Signed-off-by: lorneli <lorneli@163.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456328
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is a small optimization.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib593908d3aeb17aac55be06b8e3be42e28a23061
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456268
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Using AVX512 or AVX2 ends up being a small pessimization.
I think AVX works better for copies when there are
multiple cachelines to copy. I see a 2-3% improvement
in high IOPs benchmarks when reverting to SSE.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3d70a1e359e98cec2a9da41ccf9af2de9baa5868
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456247
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Profiling showed these weren't getting inlined - so add
the inline keyword to make sure it happens. This helps
improve performance a bit.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia86edccc9163258efdcddcce6989a71fb180caf6
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456099
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
At 10M IO/s, we see a lot of CPU cycles wasted getting
the next tracker into cache. If we only get one
completion at a time, this is unavoidable, but when
there are multiple completions pending, we can prefetch
the second tracker while processing the completion for
the first.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9de702bee3719e4494eec6f05b09be3672f1e0ac
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456097
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Prior merge contained all of the code EXCEPT for the user-callable function.
Signed-off-by: James Bergsten <jamesx.bergsten@intel.com>
Change-Id: I1cb7105ab85ffae8ed4f600261fed86c9c778893
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456282
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Purpose: Make the variable definition consistent
with the same variable in the target side.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: Ibc4ff92b6346f0a1ad803dcb79d041289f5648b2
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455807
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie32a1bb144c239b923b5cbb9e608a7dfc9c05208
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456076
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
In Fedora release 28, plug in nvme device and run setup.sh,
the uevent is like this:
UDEV [1060.112118] add /devices/virtual/vfio/81 (vfio)
ACTION=add
DEVNAME=/dev/vfio/81
DEVPATH=/devices/virtual/vfio/81
MAJOR=509
MINOR=1
SEQNUM=8544
SUBSYSTEM=vfio
USEC_INITIALIZED=1060111894
UDEV [1060.122089] bind /devices/pci0000:d7/0000:d7:00.0/0000:d8:00.0 (pci)
ACTION=bind
DEVPATH=/devices/pci0000:d7/0000:d7:00.0/0000:d8:00.0
DRIVER=vfio-pci
ID_MODEL_FROM_DATABASE=PCIe Data Center SSD (DC P3700 SSD [2.5" SFF])
ID_PCI_CLASS_FROM_DATABASE=Mass storage controller
ID_PCI_INTERFACE_FROM_DATABASE=NVM Express
ID_PCI_SUBCLASS_FROM_DATABASE=Non-Volatile memory controller
ID_VENDOR_FROM_DATABASE=Intel Corporation
MODALIAS=pci:v00008086d00000953sv00008086sd00003703bc01sc08i02
PCI_CLASS=10802
PCI_ID=8086:0953
PCI_SLOT_NAME=0000:d8:00.0
PCI_SUBSYS_ID=8086:3703
SEQNUM=8545
SUBSYSTEM=pci
USEC_INITIALIZED=1060121805
Have tested several kernel versions such as v3.10, v4.10, v4.15, v4.19.
We didn't see an event which is like this:
ACTION=add
DRIVER=vfio-pci
Change-Id: I7299a2fb4d634edaa6bab3412ee8f363f66aae6f
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452053
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is the same intention as the patch for iSCSI in this series.
This change will be helpful to extract common part into a specific
helper library if necessary in future.
Change-Id: I1ce36b424ff2afb85f998149f4ef0d7153290802
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455621
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Recently DIF library refined SGL create operation by unifying
size and used count into unused count. This patch applies the
good practice in DIF library to create SGL in NVMe/TCP.
The next patch refines names of related function and variables
to be consistent in NVMe/TCP.
Change-Id: I1e73310c0e3650ede53672d76071a6c37dba82c1
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455473
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This avoids dereferencing the request to get the qpair
in cases where we already know the qpair. Adding a new
variant instead of just modifying nvme_free_request()
since there are 72 calls to this function and I don't
want to change all of them.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ifd6fd964e546bcd71ff180fd71d5bf5cbab79d4f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455287
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In some cases we have the qpair already when calling
this function. So pass the qpair to avoid having
to get it from the request. This shows about a 3%
performance improvement for high IOPs single core
tests.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I22fcca560492f4e7cf5ffedd252e41a027d0dd79
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455286
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Currently the nvme driver will always log any
request completed with error status. Some
applications may not want this behavior. So provide
an option to disable it at the controller level.
When this option is enabled, any failed requests
from queues associated with that controller
(including the admin queue) will not log the
failed request.
Of course the application will still receive
the failed status code and can decide to do its
own logging there.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia093fcd23cf321a820fd53183ee7e2dac4f9d378
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454081
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This will (finally) enable resets for fabrics
controllers.
Move some of the work previously done in enable_admin_queue
up to this new disconnect/connect logic.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6239f0c0f36192db921d33f2322b1874b9382a01
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453939
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This will be used on the adminq, and set while the
qpair is connecting. It allows the qpair_process_completions
routine to know that it should still try to process completions,
even if the controller is resetting.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I377b9c934295eb5f45f03efd90c2a268defb4bd4
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453938
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
For fabrics controllers, the fabrics cmds are what gets
the controller out of reset.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6804874e867466669a55dff11a0a865add8bbc99
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453937
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We cannot complete error reqs from spdk_nvme_ctrlr_reset -
this could result in completions on threads not expected
by the user for I/O queues.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2e266a2618f1791ef1a1b713d1940357f23f7bff
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453932
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We are going to use it earlier in this file in an
upcoming patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie388ca76370e53465edb73a99d191492580603c9
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453930
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This requires moving the function, but since it's so
small and the change is trivial, move and plumb it in
the same patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I69cf020736676d80c6426491865d2eb063e923df
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453739
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The RDMA transport was the only one implementing this
function, and it only does a connect - not a disconnect
followed by a connect.
A later patch will add a matching disconnect function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib68eb0ff2f8e59f437d6d8831bb37dfddf83e9a4
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453929
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Next patch will use this function earlier in the
file, so move the function now rather than in the
later patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I50de44f69d0aedffddd251d00491912fd4a0f503
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453780
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I12b4081d3cf57bda8b01911c25a9c13102a1115d
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453741
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Also fix call to this function that was treating the
print_on_error parameter as if it was dnr.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9f048e8873ae0fcf07c9c6d11329a3fb21d92bda
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453740
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This function returns a pointer to the PCIe I/O registers for a controller
or NULL if unsupported for this transport.
Used for PCIe only, other transports return NULL.
Use with caution.
Signed-off-by: James Bergsten <jamesx.bergsten@intel.com>
Change-Id: I849f9de9ad259a65b1eef9c1237345eb7195b9bf
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452927
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These were accidentally removed in a previous
patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idab274427c064ff8aff1cdca2dd80d7d24e8cce4
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453747
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This transport function is a complete nop now, so
remove it.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5cc6ac75795a3cf5311f24e2ac293fb53d4b9f8c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453487
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This will allow us to move more of the reset-related
functionality to the common layer, as part of enabling
resets for fabrics controllers.
The transport qpair_enable and qpair_fail functions
acted similarly - so those are both removed now and
replaced with this new qpair_abort_reqs function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9486630ad5b807239b0b5bcde50e8cfd313695d3
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453486
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Upcoming patches will move the actual is_enabled
logic to the common layer as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9fd4d8712280295f57134ad66f8ccbfe9736d30a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453484
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The nvme_qpair_disable functions will be going away in
an upcoming patch, so move this one bit of functionality
into a helper function in advance.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I61c2de535c2230b988d56dea13b00f39cb59dcfa
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453483
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We submit AERs to all controllers - both pcie and
fabrics. But currently we only manually abort the
aers when disabling the qpair for pcie. Make this
common instead by creating a new transport function
for aborting aers.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I1e926b61b8035488cdc6e8cb4336b373732f985e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453482
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This allows us to reconnect the qpair in the future,
avoiding reallocating memory every time.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0b86d89d8ebee27125b6081fd3845718e1118d5a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453481
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This better explains what the function is doing,
and makes the name more general so we can use it
for the adminq as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6b55761cb141a9a79cdef876be47995d8813b312
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453480
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This moves us towards not freeing and reallocating
this memory if and when we reconnect the qpair.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic20d3c221442f6206d161760a8bfa7f9b8989d4c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453479
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This will simplify some upcoming changes to reconnect
a qpair. In these cases we only need to re-register
the memory - we shouldn't have to allocate it again.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id8adff313f191fbf11d7502127a2b961f2ca2f6e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453478
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In order to truly support multi-sgl inline requests in the RDMA
transport, we would need to increase the size of the
spdk_nvme_rdma_req object dramatically. This is because we would need
enough ibv_sge objects in it to support up to the maximum number of SGEs
supported by the target (for SPDK that is up to 16). Instead of doing
that or creating a new pool of shared ibv_sge objects to support that
case, just send split multi-sgl requests through the regular sgl path.
Change-Id: I78313bd88f3ed1cea3b772d9476a00087f49a4dd
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452266
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Send the given NVM I/O command, I/O buffers, lists and all to
the NVMe controller.
This is a low level interface for submitting I/O commands directly.
This can only be used on PCIe controllers and qpairs.
This function allows a caller to submit an I/O request that is
COMPLETELY pre-defined, right down to the "physical" memory buffers.
It is intended for testing hardware, specifying exact buffer location,
alignment, and offset. It also allows for specific choice of PRP
and SGLs.
The driver sets the CID. EVERYTHING else is assumed set by the caller.
Needless to say, this is potentially extremely dangerous for both the host
(accidental/malicionus storage usage/corruption), and the device.
Thus its intent is for very specific hardware testing and environment
reproduction.
Signed-off-by: James Bergsten <jamesx.bergsten@intel.com>
Change-Id: I595fe02fe0dfa9c3ceba1ac116b6900357b02d2c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451994
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reset the device to its factory defaults.
Change-Id: I43f7dc8fb7bd5226283a4762beac0e2cf016f698
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/445253
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Also make some modification for invoking process in
nvme_manage tool.
Change-Id: Ib54db43d7336d3e839e7d9317c292b9a57b38f80
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/445059
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When a Namespace was removed all the field will be zeroed, which
may lead to being divied by zero error when IO is running, especially
with perf tool. The perf tool doesn't add hogplug support, so
we add the additional check here to avoid such issue.
Fix issues #728 and #629.
Change-Id: I0e387c8c1bd4f3d40130377e2e0f5143f43be6a3
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451762
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
logical_block_size and geometry_logical_block_size are uint32_t,
so correct conversion for endianness is used now.
Change-Id: Iaa5ff576da8bc4ca65fa21f28eb3afd30507b00a
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451867
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
The mode of dmb oshld can guarantees cpu sequential execution,
which has less impact on performance.
Change-Id: If30b6a682a2216eecd1da039267ed4f5471afc38
Signed-off-by: h00448672 <heyang18@huawei.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446827
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The upper levels of the stack allow for this, so we should follow that
pattern so I/O don't break here.
Change-Id: Ia862f14975a551b0675bafd7709fb7897d0d567e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450685
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The while loop in this function was structured such that if the final
child request did not contain a full number of SGE elements, it would be
truncated. We need to not only check whether we have a full number of
SGEs, but also if the current child has consumed the end of the parent
request.
Change-Id: I7df6c224e9ab66033c92d2cf1af10452f5cdfd9b
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450684
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is an attempt to fix device hotremove with VFIO.
A soft device hotremove request through sysfs [1] would
currently just block until the SPDK process manually
releases that device - e.g. upon an RPC request.
VFIO won't get unbound from the device untill userspace
releases all its resources. VFIO can signal a pending
hotremove request by kicking any file descriptor provided
by the userspace - and DPDK does provide such descriptor -
but SPDK does not listen on it.
DPDK does offer handy API to listen and in this patch
we make use of it inside our env/pci layer. Within
a DPDK callback we set an internal per-device hotremove
flag, which upper-layer SPDK drivers can poll with a new
env API - spdk_pci_device_is_removed().
The VFIO hotremove event will be sent to primary
processes only, so that's where we listen.
We make use of this new API in the NVMe hotplug poller,
which will process it just like any other supported
hotremove event.
Fixes#595Fixes#690
[1] # echo 1 > /sys/bus/pci/devices/<bdf>/remove
Change-Id: I03d88271c2089c740e232056d9340e5a640d442c
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448927
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This avoids a data dependent load to find which
callback to call in the completion path.
Change-Id: Ifa20790a7af3332a74bc45037e589668744af797
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450558
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The same information can be obtained by checking the req
field for NULL.
Change-Id: I9689ee0be33537fd6d3f35d8c3710fabab5e1928
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450557
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
When copying the NVMe command from the request to the actual
submission queue slot, use a non-temporal move instruction.
The submission queue slots are never read by software - only
written to. So don't pollute the CPU cache with their contents.
Change-Id: I112f721abfac03bd7b33ec9ddf783d4bf2952b42
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450193
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
There are several places where we have the tracker
pointer, yet we go find the tracker again by getting
the tr->cid and using that index to find the tracker
again in the qpair's array. That's really silly.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I54acd642a2c9821f2b95e17563904b859495081a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450308
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Current the idea is:
Convert the multiple SGL into the single SGL and send it
out.
Change-Id: I8e571704e9d7c7b583f889837eead7cac1982fcd
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448262
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
First submission. Implemented part of the Opal library
and "scan" function. Can be invoked by nvme_manage.
Change-Id: Iba86d86dd3af06a06b6805120ee5005af8183459
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/439335
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: yidong0635 <dongx.yi@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Purpose: To support the multiple SGL later.
Change-Id: I133a451100b736353cf98a6aaca879d290ff5b67
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448259
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This function will be exteneded later for multiple SGL
support.
Change-Id: I1f6962ec03c72e335efaa311a12d3891312fcc53
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449968
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Fixes#722. The state was set in nvme_ctrlr_identify_id_desc_async
Signed-off-by: cranechu <cranechu@gmail.com>
Change-Id: I232f0035e8c45d49eca2de7174c91860a299d804
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449527
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Logically we should complete any I/O in the completion queue and
abort the rest of the I/O after we delete the submission queue
and completion queue, so that we would not lost any complete I/O.
We alse should complete I/O and abort I/O before destroy I/O qpair
even though the ctrlr/device has been removed.
Change-Id: Ieb28ad7b4a3a7be553f70178b29ca870b5413191
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449316
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Similar to recently added nvme_pcie_qpair_ring_sq_doorbell.
Prepares for using this code for coalescing cq doorbell
writes in the future.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I565e103acf73b3d305e72a4440e1cc678c95faa0
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448871
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
120 seconds is too long for controllers which can't be
setup during initialization, because this value is only
used for Admin commands so also rename as it is.
Change-Id: I0a3d3192252c0f6fc0bef4d8b868eaef2ae40fe3
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448601
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The identify data is only valid if spdk_nvme_connect()
was used with Discovery Controller, so move this code
into the section where it belongs to.
Change-Id: I1897f38277eafc192552a09556a568e9152bb72d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448500
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The phys_addr param in spdk_*malloc() is about to be
deprecated, so use a separate spdk_vtophys() call to
retrieve physical addresses.
This patch also adds error checks against SPDK_VTOPHYS_ERROR.
The error handling paths are already there to account for
spdk_*malloc() failures themselves, so reuse them in case
of vtophys failures.
Change-Id: I377636e66b8c570d013c1bb2021f04bce4e6c0ce
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/416998
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Try to group data members that are used often into the
same cache lines. We still need to find more space in the second
cache line of spdk_nvme_pcie_qpair so that the important
parts of spdk_nvme_qpair fit.
Change-Id: Ib936cb2b1acc722de7ec313d6faa3812aacde394
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447968
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Don't touch the shadow doorbells if it isn't necessary.
The flag could be combined into a bit mask with other
flags in a future patch.
Change-Id: I9ffd16468d29f0f0868cf849f7fece327eb6a294
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447967
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Avoid ringing the submission queue doorbell until the
call to spdk_nvme_qpair_process_completions().
Change-Id: I7b3cd952e5ec79109eaa1c3a50f6537d7aaea51a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447239
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This is going to get called from two places shortly.
Change-Id: I2c67e719c91887987e6e65c5c0c384bed0431409
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448311
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Users may want to connect specified controller at running time,
so this API will connect to the controller and return probe context
to users, users must call spdk_nvme_probe_poll_async() to initialize
the controller to the READY state before using it.
Change-Id: I232886b000454ee826ea73c4e1043d0d18ee0ec6
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/445657
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The purpose is to use the single readv to read both
the payload the digest(if there is a possible one).
And this patch will be prepared to support the
multiple SGL in NVMe tcp transport later.
Change-Id: Ia30a5e0080b041a65461d2be13db4e0592a70305
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447670
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
For API spdk_nvme_connect(), users can only use NULL or
struct spdk_nvme_ctrlr_opts for their own driver options,
so parameter opts_size doesn't take effect here. Since
some applications outside SPDK's git repository are using
such API, so we only remove the internal data structure
for now.
Change-Id: Ia727d2950d56abba637214ad17ecfa3eeab71d38
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/445656
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Free the probe context after the controllers in the probe
context list become READY. Then users don't need to free
the context which is allocated by SPDK driver.
Change-Id: I2dcb76bacf26a401b5b559c4326764a4ddb97e83
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446820
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Borrow the ideas from iSCSI and optimize
the nvme_tcp_build_iovecs function.
Change-Id: I19b165b5f6dc34b4bf655157170dec5c2ce3e19a
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446836
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
New API added for upper level to get controllers'
supported flags.
Change-Id: I51e9d0e57c355fa37f092602a94f4c08deb8898c
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446091
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The nsdata assignment is strangely aligned with some
variable declarations - fix it to make it more clear.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I43b1a6d5a69ca035a21f3996e8f859a45bd10b9c
Reviewed-on: https://review.gerrithub.io/c/446447
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
VMWare Workstation NVMe emulation does not seem to write the
SHST_COMPLETE bit within 10 seconds, resulting in an ERRLOG
during detach/shutdown. So add a quirk to cover these VMWare
SSDs. But rather than squashing the ERRLOG completely for
these SSDs, just add a message instead indicating this is
somewhat expected on these VMWare emulated SSDs.
Fixes issue #676.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3dfcb631feda639926fd712f1f41abb66cbf2096
Reviewed-on: https://review.gerrithub.io/c/445942
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Users should not access the internal probe context fields when
using the asynchronous probe API, so change spdk_nvme_probe_async()
to let it can only return the probe context pointer.
Change-Id: I0413c2d8db6cbe4539ad80919ed34dd621a9df70
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/445870
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Move req->submit_tick assignments from specific transports to generic
qpair code.
Check whether submit_tick has been assigned before doing the actual
assignment, because a request may be submitted several times and the
original submit_tick shouldn't be covered.
Change-Id: I2de8018dc21763eb5a19bb9d48dfbdef764b036e
Signed-off-by: lorneli <lorneli@163.com>
Reviewed-on: https://review.gerrithub.io/c/444702
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Add a memory barrier for arm64 to prevent possible reordering
of tracker and cpl access,
because arm64 has less strict memory ordering behavior than x86.
Change-Id: I0a8716f7bfeffb0bbce27ee3174e214c8e4566b4
Signed-off-by: heyang <heyang18@huawei.com>
Reviewed-on: https://review.gerrithub.io/c/442964
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Purpose: to make the timeout work for NVMe TCP transport,
we miss this for TCP transport.
Change-Id: Iab4af988cc4796b4d6d98430453f3dbce1fcf313
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/c/445117
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch is used to implement this function.
Since we need to call nvme_tcp_req_complete in this
function, so we need to adjust the location of the
nvme_tcp_rep_complete funtion.
Change-Id: I5fc3693aec8dc166ac1eb03babcd2d73d7b00e63
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/c/444489
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
To enable the timeout function.
Change-Id: Id5c40848957743683b6a5c2d085e7f777f14497d
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/c/444803
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
nvme_ctrlr_submit_admin_request() will access admin queue, and we
should hold ctrl->ctrlr_lock when access it.
Change-Id: Iff576fe5e14e854eb38dbc64d6c6d9ec1ba17056
Signed-off-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-on: https://review.gerrithub.io/c/444793
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Also use the same style condition check for secondary process
with PCIE type.
Change-Id: I93c83126145255887914ef5efea1a493c8f7f767
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/444492
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
User can create a probe context to probe and attach controllers
asynchronously, the controllers will be added to the context list
for the first step, then users can poll the context until the list
becomes empty.
Change-Id: I3a96e2d8a9724332ff15542f78f9553fdab505e2
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/442664
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Existing NVMe driver uses a global list g_nvme_init_ctrlrs
to track the controllers during initialization, and internal
function will start each controller in the list one by one
until the list is empty. We introduce a probe context
and move the global list into the context, with the context
we can enable asynchronous probe API in the next patch, also
this can enable parallel probe feature.
Change-Id: I538537abe8c1a4a82fb168ca8055de42caa6e4f9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/426304
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Previously, function spdk_nvme_probe_internal() will probe
NVMe controllers and then bring up probed controllers
into the ready state after that. Broke up original two parts
with probe and start stage, this will help us to introduce
a probe context in the next patch.
Change-Id: Ie0c55a6a5463fb437f84349b0b2b33a217ba63e0
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/426303
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Althrough SPDK already provides a API to users which
can process runtime timeout NVMe commands, but it's
nice to have another API here, SPDK NVMe driver can
use it to break the endless wait. Also use the API
first in the initialization process, because we don't
want to add another initialization state with Intel
only supported log pages.
Change-Id: Ibe7cadbc59033a299a1fcf02a66e98fc4eca8100
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/444353
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This prevents us from overrunning the send queue.
Change-Id: I6afbd9e2ba0ff266eb8fee2ae0361ac89fad7f81
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/443476
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The next patch will use the string "prchk:reftag|apptag" as
per-controller prchk options for .INI config file.
Hence add helper functions for them beforehand.
Change-Id: I58c225cc36cc84bf594f108e611028996b5eedb9
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/443834
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
The patch is used to fix issue:
https://github.com/spdk/spdk/issues/638
Reason: For supporting sgl, the implementation of
function nvme_tcp_pdu_set_data_buf is not correct.
The translation is not correct for incapsule data
when using SGL. In order not to do the translation
via calling sgl function again, we use a variable
to store the buf.
Change-Id: I580d266d85a1a805b5f168271acac25e5fd60190
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/c/444066
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Currently infrequent cases in request completion path are marked as
unlikely. This patch applies that to submission path.
These cases are infrequent and marked using unlikely marco:
a. The sq tail reaches the end of queue.
b. The sq tail equals to sq head. (never happen if FW runs correctly)
c. The qpair is admin queue.
Change-Id: I8b873a18615788f2efbf7c683aad710c7007a082
Signed-off-by: lorneli <lorneli@163.com>
Reviewed-on: https://review.gerrithub.io/c/443451
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This type was actually two entirely different types for
the initiator and the target, so just make it void.
Change-Id: I15512d9d4efd790dce0fa4323b7230de66144bc6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/442438
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The purpose this patch is to fix the following issue:
https://github.com/spdk/spdk/issues/568.
The root cause of issue is in nvme_rdma_fail_qpair
since we want to recycle all outstanding rdma_reqs.
There is an aer req, the callback of which is:
nvme_ctrlr_async_event_cb. In this function, we
will call nvme_ctrlr_construct_and_submit_aer again,
however the nvme controller is already in shutdown state.
(The ctrlr->vcprop.cc.bits.en is set to 0).
Change-Id: I422f0fe5faf472e9a1cb6bbd174e806e6405b95c
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/440014
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
RDMA transport will report SPDK_NVME_SC_ABORTED_POWER_LOSS code
when fail the admin queue, however, SPDK_NVME_SC_ABORTED_SQ_DELETION
makes more sense here, because we know we are going to shutdown
the controller.
Fix issue #568.
Change-Id: I31da095ec92c06079511d89cc2743654ba2c001b
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/440132
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Necessary to avoid erroring out in the edge case where we have an SGL
request sent with two buffers that fit in the incapsule data size.
Change-Id: If51fb69c402482b564c737319584378cb03e7213
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/436062
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Makes the code slightly more readable.
Change-Id: Iebf8fb07bceacf433d4bdad0a30419a3faab7eee
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439370
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We use those values in various places in SPDK,
so let's define them in a single place now.
Change-Id: Iad9a5745d69166a6e6032370d4e5a0e604914e45
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439369
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This follows the same trend as the mem_map APIs.
Currently, most of the spdk_vtophys() callers manually
detect physically noncontiguous buffers to split them
into multiple physically contiguous chunks. This patch
is a first step towards encapsulating most of that logic
in a single place - in spdk_vtophys() itself.
This patch doesn't change any functionality on its own,
it only extends the API.
Change-Id: I16faa9dea270c370f2a814cd399f59055b5ccc3d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438449
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Currently for all the Intel drives nvme driver tries
to add Intel VS log pages support. When this log pages
are not supported whole init process fails.
This patch changes this behaviour by allowing to init
Intel drives which rejects VS log pages. This is valid
scenario for drives which are in states other than
healthy. Such a drives are still accesible via admin
queue, but does not expose some of the features, such
as this particular VS log pages.
Change-Id: I3764f2d67fd7153b6b1889273a9fedeb9c4213d3
Signed-off-by: Igor Konopko <igor.j.konopko@intel.com>
Reviewed-on: https://review.gerrithub.io/c/437162
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This helps us get rit of outstanding requests at the bdev layer.
Change-Id: I362c7c0c6641715fcd96e8eb465b308c368d34fc
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/431844
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
This allows us to specify the host-side configuration for each
controller to which we connect.
Change-Id: Iac2aed3934d4a326f45546f2f541e374308e2589
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/436219
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This code snippet will be generally useful for parsing information when
we add support for host address configuration.
Change-Id: Ic90f485de5a5db699901da029c9a29be4db477c7
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/437739
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
If the adminq is not created, we should call
nvme_transport_ctrlr_destruct (i.e., nvme_tcp_ctrlr_destruct
in tcp transport).
Change-Id: If2362c47ac89fd80289ddd8402cf9f74d7bb5801
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/436613
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The nvme/identify cmd issued some cmds to a ctrlr irrespective
of its type, and when the target was a Discovery ctrlr which only
accepts a very limited cmd set, that would result in errors observable
both on the initiator side (from nvme/identify) and in the output on
the target (nvmf_tgt). Introduce new API, spdk_nvme_ctrlr_is_discovery(),
and alter identify to make use of that in determining which commands
to send to the target.
Change-Id: I974a569843f1d2b9e1ece7bd3bf9ceee1bfae872
Signed-off-by: Lance Hartmann <lance.hartmann@oracle.com>
Reviewed-on: https://review.gerrithub.io/436225
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
This is a failsafe for finding and reporting data buffers that span
multiple Memory Regions. These errors should never be triggered, but
finding and reporting them will help any debugging.
Change-Id: I3c61e3cc510f5a36039fc1815ff0de45fce794d5
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/436054
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia65e235a85207c128ba274e1bab38d6c35344239
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/435563
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Fix the issue in both target and host sides.
Change-Id: I1bf31072b2164a3035b443fe6c5418a6a7829d81
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/436099
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
As a part of cleanup they're replaced by a device-agnostic
attach API, which is easier for us to manage.
Change-Id: I7558590e41e5c580a130a6aba7ae4f7dcff58da8
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/436478
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When the applications call spdk_nvme_ctrlr_alloc_io_qpair,
there will be cmd to the admin qpairs in nvme_ctrlr_get_cc,
so there is contention. We should use the lock to protect
nvme_ctrl_get_cc. Otherwise, the multiple threads will have
contention on the admin qpair, thus there will be coredump issue.
We get the bug when testing NVMe-oF TCP transport, and this
patch can address this issue.
Change-Id: I7247f98cdf890c2eafaf8fb94580ecd714010bd5
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/435577
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
According to the TP 8000 spec, the maximal in capsule
data size is defined as follows:
1 For the Fabrics command and admin, it should not exceed
8192 bytes.
2 For I/O command, it shoudld be defined according to ioccsz
in the Identify controller data.
Change-Id: Ic13eda33e1516858e1e8749ee89459e3148d9e37
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/435826
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
spdk_nvme_transport_id_parse() does not recognize the
namespace id, "ns", key as part of the transport id string
and thus logs an error message, but does not fail the call.
However, some SPDK applications, e.g. nvme/perf, in addition
to using spdk_nvme_transport_id_parse() also check for the
existence of a "ns" key in the transport id string to limit
the target to a specific namespace. This commit adds a
special case to spdk_nvme_transport_id_parse() to silently
ignore the presence of a "ns" key without logging it as an
error.
Change-Id: I49732b4d1b0227a38bb308eab1f6324dd241a2de
Signed-off-by: Lance Hartmann <lance.hartmann@oracle.com>
Reviewed-on: https://review.gerrithub.io/435192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This case isn't particularly supported, but still
caused a memory leak and rendered the pci device
inaccessible for the rest of the primary process
lifetime.
This happens when a controller is removed from the
primary process while a secondary process still
uses it. The controller will likely misbehave without
its primary process managing it, but at least there
won't be a leak.
Change-Id: I67581cffa33ce14ff516b5743d13c9ef7b351625
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434408
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
With various possibilities to leak the rte_pci_device in the
primary process, we could technically construct the controller
in secondary. The nvme stack is not prepared for this and
will fail to initialize the device, but will still leak the
device object memory.
This patch adds an extra check to prevent any controller from
being constructed in secondary process.
Change-Id: I772f42b541c5db53310362b6595cebf9a30e8491
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434407
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is necessary to confirm that a buffer that spans a 2_MB boundary is
still in a single MR.
Change-Id: If0d14e514ab2197a0d2e3af4f565f56d50591210
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/435179
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
This patch disables the header digest and data digest
by default, which will improve the performance.
In the another patch, we will make it configurable.
Change-Id: Icdf8cda28217ec35a6b87bb932cdb1e4f8492471
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/435209
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5e7d16e377c03165f338709a71d6e4f03beffc0a
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/434066
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Explicitly attaching a PCI device with spdk_pci_device_attach()
bypasses any kind of blacklists and should be only used
on a user request. Hotplug uevent is certainly not a user
request and should respect the blacklist, hence it's now
changed to call spdk_pci_enumerate() to probe new devices.
The enumeration callback will reject devices other that the
one we got hotplug request for, so no behavior is changed
in that matter.
This patch also fixes undefined behavior caused by reading
unitialized struct nvme_pcie_enum_cb;
Change-Id: I1399fbdd426152a13ed75c85a52bc7f0491ce287
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/433867
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
It is the first patch to follow the NVMe over fabrics
spec and implmenent the NVMe/TCP transport. It can be
divided into work in the host and target sides:
Host side: Add the TCP/IP transport in nvme lib (lib/nvme).
Target side: Add the TCP/IP transport in nvmf lib (lib/nvmf).
Change-Id: Idc4f93750df676354f6c2ea8ecdb234e3638fd44
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/425191
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Currently there are no timeout mechanism for Admin commands
when initialization, the NVMe driver may enter infinite loop.
While here, add a new parameter to the controller initialization
options, NVMe controller will report an error when timeout
happens during initialization.
Change-Id: Id0c6b6fa15abe5227b486bee95c8e02914b0d358
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424622
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
The max_send_sge and max_recv_sge values can be set to any value from
0...dev_attr->max_sge. WHen we actually set the attributes, we will
receive a qpair with values for max_sge greater than or equal to what we
initially set. We need to store the maximum number of SGEs for later use
when constructing work requests.
Previously we have not relied on these values since we assumed that we
would always be able to have more sges than we asked for initially. This
may change as we try to allocate more SGEs to handle splitting buffers
across memory regions.
Change-Id: Ibbeae1908b86baa3a96d9c6cd2051401aaa2197b
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/433307
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The RDMA memory map needs to be per-protection
domain, not per NVMe controller. Otherwise, when
an NVMe controller is removed, the memory map may
reference an invalid pointer to a detached
controller.
Change-Id: I0c5bd2172daee0c70efb40eab784839e0cde8bc4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/432590
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In some special cases, NVMe device with cdata.nn=0
may be used to do validation or other test work.
cdata.nn=0 means the device can't support NS at all.
Change-Id: I55f75a8cb21b8d1b99c5318e27c876a4371d6dd4
Signed-off-by: Liu Xiaodong <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/432191
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: joevannip <jparairo@nvxltech.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Add a mechanism to modify the RDMA transport's behavior
when creating protection domains and registering memory.
This is entirely optional.
Change-Id: I7cd850e76a673bf5521ca4815b779c53ab9567e8
Signed-off-by: zkhatami88 <z.khatami88@gmail.com>
Reviewed-on: https://review.gerrithub.io/421415
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In cases we probe without a specific trid, the underlying
rte_bus_probe() in spdk_pci_enumerate() might fail to
initialize some devices, but still return with code 0,
That's technically correct, as we asked just to probe
devices on the bus and that's what it did. Some devices might
have been initialized, others not. In secondary process we
blindly assumed all devices were probed successfully, which
might have eventually led to assert failures, as current
process was not on the ctrlr->active_procs list.
To fix it, just add an additional check before attaching
the controller in secondary process.
Change-Id: If015b1e562052a9189ed1a48091b209bd2dd5f2a
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/431727
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We only detached the PCI device on the controller destruction,
which happens just once - in the primary process, but secondary
process needs the PCI detach as well.
Requesting to hotremove the NVMe PCIe controller in secondary
process is broken, because DPDK will still keep the device
reference and won't allow SPDK to hotplug it again.
Fix this by detaching the local PCI device whenever removing
a secondary process from spdk_nvme_ctrlr. This does require
an additional transport check in the generic NVMe layer, but
I found it an overkill to create a multi-process transport
abstraction just for this case.
Change-Id: I812dc1c878ade5b149556806228a2afcb49f0b17
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/431487
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The underlying probe might fail, but we don't check its
return code. Right now we ignore the failure and in secondary
process we even continue referencing a locally-unitialized
controller struct. Then, a few calls later, we fail on assert
because current process is not on the ctrlr->active_procs list.
Change-Id: I65a59a9515a8e0196b60a181cee2af33434784dc
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/431486
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The time required to wait increases with the amount of submitted
FLR resets. Now that DPDK takes less and less time to initialize,
this starts to become an issue. We can even see on our CI within
regular tests where a single application is start-stopped in
a short period of time. This is also a problem if a device is
detached via RPC and immediately attached afterwards.
The time required to wait seems to cap at 2 seconds, so instruct
our driver to wait exactly that.
Change-Id: I18b6fbdea9b0dca5d7e1756e9ead7d97119f2fa2
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/429415
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This is an NVMe-specific issue and I/OA or VirtIO devices don't
need it. Additionally, the delay is now asynchronous, meaning
that potentially multiple NVMe controllers can wait all at once.
The drawback of this change is that we're needlessly waiting
even when using uio_pci_generic. However, since the delay does
not block anymore, its impact is significantly minimized.
Change-Id: I5d16a7fd7cb66c785acb687f14690e95f6188b9e
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/429414
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This indicates an out-of-spec device, so just print an error
message but don't bother retrying the AER.
While here, add status code type (sct) check for the other
status code check when an AER fails - it is not enough to
compare just the status code.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ibd26549aa08d3eb4814c239b6b2c6fe95e069a54
Reviewed-on: https://review.gerrithub.io/429533
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
With Identify Namespace Identification Descriptors can be
executed asynchronously, most of functions in the controller
initialization now can be executed asynchronously now, for
host with multiple controllers this can save some time during
initialization.
Change-Id: I70e3c6c2c691134d2ae4c5969288cced1538c6cc
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/428585
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I189ad8889c74937bf43bcf2c3029416ddb94976d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/425705
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
The initiator will now automatically retry sending
commands if the target is not ready.
Change-Id: I13354283f77d9ccba9645e83c77061cac1b07b0f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/428732
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This makes this particular function consistent
with all of the other functions in this file, and
I feel it is slightly more readable.
Change-Id: I99ace5b9eb45b0f706ca85a64b155444f45c9815
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/428730
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
While more verbose, this makes it much more obvious that
an array of SGL elements is being filled out.
Change-Id: I98b8e5d46af32c5d7dbb990e267fdfd594942081
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/428729
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Also add some comments.
Change-Id: I97c3a44f97aa3dadc114005c10bec83ae75994cf
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/428728
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
spdk_mem_map_translate() dereference a uint64_t * to get a
8-bytes long integer, but nvme_rdma_build_sgl_request() just passes
a 4-bytes long integer as last parameter, this causes a
stack-buffer-overflow error.
Reported in 3ba5ea9087.1539172863/fedora-05/build.log
Change-Id: Id1cda22114fef466dbb930b502e3a68310331f0e
Signed-off-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-on: https://review.gerrithub.io/428693
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This prevents the SPDK build from requirng unnecessary dependencies when
not compiling certain features. Also, fixes github issue #434
Change-Id: I7d0520474f3656ae32670313f2290e6b741c5ca8
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/426131
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Lance Hartmann <lance.hartmann@oracle.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Normally, there will be only one separator in transport id,
for example, either ':' or '='. But the users may input
this: trtype=PCIe traddr=0000:81:00.0.
Thus, there will be two diffrent separator '=' and ':',
and our function doest not handle this case correctly.
And this patch can fix this issue, and also update the
test case.
Change-Id: Ic3f10dc1e37c66647fede37c5cf9523fc2652677
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/428307
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Each file that need to check SPDK_CONFIG_* options need to include
spdk/config.h explicitly.
Change-Id: If9f2a91ac4c2b1a300dcf88ec3e2a12714ad344a
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-on: https://review.gerrithub.io/427221
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
When the NVMe SSD supports multiple namespaces, we
allocate the related data structures to hold the
namespace related information. Add a check of valid
namespace id before accessing the memory structure.
Change-Id: I3176099a80f718d9470ee172a040a2ccc353aae9
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/427058
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch adds struct for "Chunk Notification Log".
New log page is used to report the state of chunk, lblk or pu.
Implementation is consistent with Open-Channel
specification (rev. 2.0)
Change-Id: I8aaf01c14d2f0faf8b8f8a6a773b6cb9bf97b38d
Signed-off-by: Jakub Radtke <jakub.radtke@intel.com>
Reviewed-on: https://review.gerrithub.io/426233
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: I830aba773e1d247ec571ff31eaba970ced0fd7a0
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/425413
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This function will now check for whether or not a memory region is
contiguous accross 2MB map entries and return the total length of that
contiguous buffer up to the size specified by the user.
Also includes unittests
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: I2ce582427d451be5a317808d0825c770e12e9a69
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/425329
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: I90da6d4d31c669a3bf046f7721923dd743c5ef21
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/425328
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The function now takes a pointer as it's last argument, and copies the
size of the memory region for which the translation is validinto that
pointer.
For now, that will always be 2MB. However that behavior can change in
the future.
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: I8686c166ec956507f5ae55cf602341281482cb89
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/424888
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
use multiple sges, extended to support 2 sges.
add incapsule data support for contig and sgl type commands.
Change-Id: I9eaea0b5df4f4056e7a7f413542d557adad5899d
Signed-off-by: Potnuri Bharat Teja <bharatppotnuri@gmail.com>
Reviewed-on: https://review.gerrithub.io/423356
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Change-Id: I1e5be0e282b9e29f7bf7ca7d2720b9fd00539be0
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424776
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ide0c81b1cc29d67cec0c10ab877360db3699141e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424775
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Currently in the function nvme_ctrlr_start() the initialization
process is executed as a whole, in the case there are many controllers
in one system, which means other controllers must call the function
one by one. While here, we add several states here, which can
help refactoring the initialization process.
Change-Id: I209cf964bbf6e151823a7ecdc6a3f6e6e69df297
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424157
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I51b312a086f18a5b5f63de27dd69e43a8cc7225d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424914
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idd365df7fb61eafb502f415adf70638bb91ded0e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424773
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Identify Namespace Data Structure will be updated when format/
attach/detach/AER Namespaces, so we don't need to update it
in function spdk_nvme_ns_get_data().
Change-Id: Ia33804722bb09eb9a4e3fa1de251da47208a09b9
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424637
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Ib5e977b0bad15af7a2a71000c1fc4861b5b5b0af
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/424465
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
QEMU emulated NVMe SSDs report themselves with an Intel vendor ID,
but don't support the Intel vendor-specific log pages. So add
a quirk to avoid confusing error messages.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic41476801ede94d43acb9972217ea7420ca53679
Reviewed-on: https://review.gerrithub.io/423422
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
spdk_nvme_ns_get_sector_size returns the sector
size of the data only. This new function adds
in the metadata size, if any.
While here, modify the bdev/nvme driver to use this
function - this is needed for ongoing extended sector
size work through the bdev layer.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic8070a7f8d29b0b2ac2a2d65e0df5f4736488351
Reviewed-on: https://review.gerrithub.io/422445
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Release nvme rdma ctrlr before exit the nvme_rdma_ctrlr_construct function
when creating the admin qpair fails.
Fixes GitHub issue #363.
Change-Id: Ib988e0da2f627db06b68bd3fb72c117c52572cf8
Signed-off-by: Chen Wang <chenx.wang@intel.com>
Reviewed-on: https://review.gerrithub.io/420719
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: If25463ea76e8382c61f10636e1119376a00104dc
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/411594
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Change the type of nvme_timeout parameter in
spdk_nvme_ctrlr_register_timeout_callback from uin32_t to uint64_t.
Reason: This will make the timeout trigger test more flexible and
will not affect the original meanings.
Also for the configuration file, still maintain the compatability
support
Change-Id: I94c90f67b2e9c57220ab82ecea11a1590d62aed4
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/419326
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
If command is completed manually (e.g.: driver decide to fail this
command from some reason) not all NOTICELOG informations about this
command are valid. To not get confused who completed the command (driver
or a device) show addtional information when command is completed
manually (using NOTICELOG like nvme_qpair_print_command and
nvme_qpair_print_completion do).
Change-Id: I7211bd165aec142ad11e806fa3031371375edd2f
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-on: https://review.gerrithub.io/411293
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add some known Samsung controllers to those requiring the quirk,
NVME_QUIRK_DELAY_BEFORE_CHK_RDY. Addresses an issue for those
who may not have later firmware that corrects the problem.
Correspondingly, extend the delay from 2 secs to 2.5 secs.
Change-Id: Iee773905a2a49711775042c061f6c347e0da85e9
Signed-off-by: Lance Hartmann <lance.hartmann@oracle.com>
Reviewed-on: https://review.gerrithub.io/419273
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
The spdk_dma_zmalloc guarantee about physical memory contiguity
is about to be removed soon. For hardware rings that require
physical memory or IOVA contiguity we will now enforce hugepage
alignment and size restrictions to make sure they occupy only
a single hugepage.
Change-Id: Iebaf1e7b701d676be1f04a9189201c5d89dad395
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/418547
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
This is verified on QEMU NVMe which merged the related patch.
Change-Id: I7c62d7a91350e0a877d1c3651796c38b55122f98
Signed-off-by: Young Tack Jin <youngtack.jin@circuitblvd.com>
Reviewed-on: https://review.gerrithub.io/417077
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Matias Bjørling <m@bjorling.me>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
A few open-coded sequences equivalent to SPDK_CONTAINEROF() were
scattered around; replace them with the macro from spdk/util.h.
Change-Id: I95c6e6838902f411420573399ced7c58c2e4ef84
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/418126
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The spdk_dma_zmalloc guarantee about physical memory contiguity
is about to be removed soon. A single tracker is page size
aligned and is exactly one page big, so it is physically
contiguous, but we can't assume an array of those is physically
contiguous as well.
Change-Id: I3aa4d14dd677601c30aa2d8f15197886d6c46e58
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416840
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
The NVMe PCIe transport only requires physically contiguous allocations
for struct nvme_tracker and the I/O SQ and CQ entries, which are already
handled separately. Change the comments to indicate that struct
nvme_payload's contiguous type only requires the memory to be virtually
contiguous, since nvme_pcie_prp_list_append() already steps through the
buffer and translates each (4K) page independently.
Change-Id: I45ac8dfb2c033a0fcbf2effbe33af4efc1eb23cb
Reported-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/417045
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
There's no need to split a bufffer if it's physically
contiguous. We can now merge buffers that would be
previously split by the nvme_pcie driver and also
separate SGEs provided by the user that happen to be
physically contiguous.
Change-Id: I9c9de31d52a9dc9e384806555cb94609aff0ccf3
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/417061
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows NVMe PCIe devices to be used with
physically discontiguous I/O payload buffers.
So far this is just a dumb splitting which
doesn't check for physical contiguity. This is
improved in a subsequent patch.
Change-Id: I0ecc443149225eaa0e4156ddda78613bcf034406
Suggested-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/417060
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Future DPDK versions may drop physical memory contiguity
guarantee for common memzones. DPDK 18.05 introduces
an RTE_MEMZONE_IOVA_CONTIG (0x00100000) flag, which is
documented as follows:
> RTE_MEMZONE_IOVA_CONTIG - Ensure reserved memzone is IOVA-contiguous.
> This option should be used when allocating
> memory intended for hardware rings etc.
To preserve backward compatibility, SPDK introduces an opposite
flag, SPDK_MEMZONE_NO_IOVA_CONTIG.
Change-Id: I9ea79b096fdb094051f13c9a802740b0e4ccc98e
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416977
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This is a generic NVMe-oF command that may be used for other transports.
Change-Id: Id5fbf1f176ef5f75a221b40eff538e693817bcaf
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/416578
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
All controllers have a controller ID (cntlid), and this will be needed
in other NVMe-oF transports, so move it to the generic controller
structure.
Change-Id: Iaba5b93e1267e7bef3a6eb7c677c549a3d83985c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/416577
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This can be used for other NVMe-oF transports.
Change-Id: Ic8d2dc483220eb3690cb756bcd750d19c93d98e6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/416576
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
These functions are generic and may be used for other NVMe-oF
transports.
Change-Id: Idb3aa30d9b0b1be7b60b85ab4911b28db35977a4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/416575
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Fix issue #313.
For multi-process test scenarios, the secondary process may access
controller’s CSTS register in the shared ctrlr list. For this situation,
all the controllers are already in the primary shared ctrlr list, but
then each controller is added one by one in the secondary process, so the
secondary process may access CSTS before it is remapped for the BAR space.
In the rpc_config.sh test case, the spdk_nvme_ctrlr_get_regs_csts function
will be called in _nvme_pcie_hotplug_monitor function before calling
spdk_pci_nvme_device_attach. This step caused the secondary process iSCSI
Target access CSTS before it is remapped for the BAR space.
Change-Id: Ifd62c38adf8624f9877a9a2f965ca4db28839d99
Signed-off-by: Liang Yan <liang.z.yan@intel.com>
Reviewed-on: https://review.gerrithub.io/412594
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch adds support for following Open-Channel vector commands:
* Vector Chunk Reset
* Vector Chunk Write
* Vector Chunk Read
* Vector Chunk Copy
Implementation is consistent with Open-Channel specification (rev. 2.0)
This patch provides unit tests for new public NVMe
Open-Channel namespace commands.
Change-Id: Ic71be9357c61c5de82ca672e82a71aa933bd1875
Signed-off-by: Jakub Radtke <jakub.radtke@intel.com>
Reviewed-on: https://review.gerrithub.io/414969
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Young Tack Jin <youngtack.jin@circuitblvd.com>
Add FC definitions to nvme header and library functions.
Change-Id: I8980f55d834c1e1d4f415756cb7a46a3ff1c7db3
Signed-off-by: John Barnard <john.barnard@broadcom.com>
Reviewed-on: https://review.gerrithub.io/416434
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Users can set specified Admin commands or IO commands with
error status, when submitting new commands which are already
set with error status, the commands will return to the caller
with specified error code. So that users can emulate some error
status for their error condition code path.
Change-Id: I4b93c7e4f2b15a659da73b39e26bfa162eb5214e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/410870
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I7c3969c479c4fa1017423515b27d8db187142420
Fixes: 6b504fda5d ("nvme/rdma: add timeout support to RDMA transport")
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/416245
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: Id0408e571362527e7c2d4759223946a0b4d7c675
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/415896
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ifac14f74f6b7681fccbf463f744d145ae8299240
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/415892
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
There isn't a standardized way to identify Open-Channel
SSD, different verdors may have different conditions.
Here just take the Qemu simulated OCSSD device as a
start.
Change-Id: I1aceaac09db438c203875bcf37badd542618bdd7
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/411590
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jakub Radtke <jakub.radtke@intel.com>
Change-Id: I3e53773570e66c1f52a6c29c6709684fc393717e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/414883
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This will be used to check timeouts.
Change-Id: Ib3cabd3c65a87ffac451e21ac5c786403298c346
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/414882
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Future patches will need to remove elements from the middle of the list,
so a singly-linked list won't be efficient.
Change-Id: Idad5df3aea08d58fef2e19770509363441f757c7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/414881
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Some features are per namespace and require a namespace ID; the existing
get/set feature functions do not allow passing NSID, so add new API
functions that do:
* spdk_nvme_ctrlr_cmd_get_feature_ns
* spdk_nvme_ctrlr_cmd_set_feature_ns
Change-Id: Ia7dc96e57badf1a56489785fb288493592b58b80
Signed-off-by: Jakub Radtke <jakub.radtke@intel.com>
Reviewed-on: https://review.gerrithub.io/414706
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This patch adds to nvme driver new public API dedicated for
Open-Channel:
* spdk_nvme_ocssd_ctrlr_cmd_geometry
Change-Id: I69592618d9bcc280346258e771195683d5b33893
Signed-off-by: Jakub Radtke <jakub.radtke@intel.com>
Reviewed-on: https://review.gerrithub.io/412816
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
apptag and its mask are required to set proper PI
elements in nvme read/write cmds.
Change-Id: Ibabc4738f637d13ea16246d5e77e7d045f032af8
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/414786
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Young Tack Jin <youngtack.jin@circuitblvd.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Factor qpair destruction function so that we can put common
resource release together in future.
Change-Id: I44139947820c2a384b745ae2673799f1b736369c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/412604
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This will be used in other transports as well.
Change-Id: I05026b0dfea2647d61a173379aca368ca48a2f52
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413864
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If requests are in flight at the time a timeout callback is configured,
we can't retroactively get the submission time of those requests; treat
them as not having a timeout.
Change-Id: Ic589e874a0f2c8c7f4ce352afa7c3aea33b01fae
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413863
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This is the first step toward timeout handling for other transports.
Change-Id: I386dd990f667d449e94ba4bcedaa3435743755fd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413862
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This was partially fixed in commit ddeaeeec19 ("nvme: Only check
timeouts on requests from the same process"), but the function that
calls nvme_pcie_qpair_check_timeout() was also erroneously filtering out
the admin queue. Restore the original behavior of checking all queue
types.
Change-Id: I26a44ff5eb772735d314ce7b8322ba9222675911
Fixes: 31bf5d795e ("nvme: make timeout function per process")
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/411628
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
verified on QEMU NVMe for OCSSD 2.0 and RFC will be shared soon
Change-Id: I01c513e30768ac1961a5c5af20de3a2490da7729
Signed-off-by: Young Tack Jin <youngtack.jin@circuitblvd.com>
Reviewed-on: https://review.gerrithub.io/413854
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
For attach/detach command, the identify namespace data should be
updated during the namespace attribute notice event callback, while
here, in case of the driver may not set aer callback, so update the
namespace identify same with attach command.
Change-Id: Ie594b2ff646a67488d03af2771c00f9947395aba
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/412884
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When the shadow doorbell feature is enabled, no actual MMIO takes place
when a shadow update is sufficient; tighten the bounds of the updates to
g_thread_mmio_ctrlr in the two doorbell update locations so that we only
need to touch the thread-local variable when actual MMIO access is
required.
Change-Id: Ida974bec33f56cbb9f7d3611f483c6975ec773ab
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413856
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is consistent with the Open Channel naming we are using elsewhere.
Change-Id: Ib088359bed29a958f8b50e41cf34143a23429f54
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413840
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: I841d7b47bc85498abb608944587e7b7585138263
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/411588
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This reduces submission+completion time by 10-15
core clocks per IO on an Intel Xeon Platinum
processor. Similar improvements should be seen
on other processors as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3241ba53ef5f21a8eef930b523a951525922e6b8
Reviewed-on: https://review.gerrithub.io/413284
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
This variable will no longer be static in a future patch.
So make the name a bit more verbose since it will now
be visible outside of the module that defines it.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5e720ef44aa5c5f38e0fe91de091a89b9970fcb7
Reviewed-on: https://review.gerrithub.io/413283
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
The memset was zeroing a lot of bytes that get
initialized either later in this function or elsewhere
in the submission code path. Eliminating these
extra memsets saves a few nanoseconds of CPU overhead
in the NVMe submission path.
Note: one use of the cpl data member depended on
the nvme_allocate_request memset. Since this use
case is not in the primary I/O path, just memset it
in that specific location before using it.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ife483a4d9c24c033cc7d26d94ec1700905a936f4
Reviewed-on: https://review.gerrithub.io/413153
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Rather than storing nvme_payload::type explicitly, use the SGL reset
function pointer as an indicator: if reset_sgl_fn is non-NULL, then the
payload is an SGL type; otherwise it is a contiguous buffer type.
This eliminates the one-byte type member from struct nvme_payload,
making it an even 32 bytes instead of 33, allowing the removal of the
awkward packing inside struct nvme_request.
Change-Id: If2a32437a23fe14eb5287e096ac060067296f1dd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413175
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The definitions of these macros will change in an upcoming patch that
modifies the way nvme_payload is laid out.
Change-Id: Ic6edc18928542b07be7519a72bdbf6babbeb0131
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413174
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This will simplify upcoming patches that change the way nvme_payload
stores its type.
Change-Id: Idf0a5b8dfd7d66a10f89254d2c5c54fee2968a43
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/413173
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Number of Namespaces of controller identify structure
defines the maximum number of namespaces supported by
this controller, for physical NVMe controllers, the NN
is a fixed number, while here, we set the same rule for
NVMeoF controllers.
After NVMe driver got namespace notice event, it should
update the namespace identify data structure for NS
attach/detach commands.
Change-Id: Id72a2600a2ce9492fa2d6e09924667acbb77ae43
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/412883
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Fail when number of child requests is >= NVME_MAX_CHILD_REQUESTS.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I8c370053847c9f623b861137da8d2387a66fc030
Reviewed-on: https://review.gerrithub.io/408850
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
For some cases, especially for Admin commands, there maybe has
recursive commands, e.g.: in AER callback we may send a new AER
request, in such case, the current code can't process such
case. While here, move the completion queue head to next before
any response callback will fix such issue.
Change-Id: Ide56701d94615881790cf025ede2f07420b9b16e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/412766
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Factor out the common pattern of waiting for an internally-submitted
command to complete. This will give us a convenient central place to
add error checking.
Change-Id: I65334d654d294cfb208fc86d16fa387ac5432254
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/412545
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
As of NVMe 1.3b, there is only one command set. But pipe
this through the driver per-spec anyway.
Change-Id: I4faf8596f5ce638e5e2a500b424e00ceb6e89edc
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/412102
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
When IO is finished SPDK will trigger callback at controller layer,
while here, wrapper the completion callback into a function so
that we can add error injection at this function in following patch.
Change-Id: I7b7a6d278d87fd09a05f51f688398fdf2e9c4e05
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/411630
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
For the same reason as commit 31bf5d795e ("nvme: make timeout function
per process"), the AER callback also needs to be stored in the
per-process controller data structure.
Change-Id: I41425d81a2ab16c06ef9b900bef6a6128117fcb0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/410953
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The RDMA transport was not correctly registering new controllers on
creation. RDMA doesn't support multi-process, but it still needs a
single per-process structure for the owning process.
Change-Id: I337ab9d00b468671e7c7a21175682beed4cfdd2e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/410958
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The per-process controller data may only be touched while holding the
ctrlr_lock.
Change-Id: I18c8c4e43db4d58e6b86f0c0fd222f6d30830b85
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/410952
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I7598222db5d76c1a1578fbb5935d4348f7c62f54
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/410951
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Avoid sending CNS values other than 0 and 1 to controllers with the
Identify CNS quirk added in commit 6442451b10 ("nvme: add a quirk for
identify 0x2").
This probably doesn't affect either of the controllers with the quirk in
practice, since this command is already only sent to controllers that
report NVMe version 1.3 or newer, but let's add the check anyway for
consistency.
Change-Id: I858c28ab7d43714c501dd22d6797259054f0f0a8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/410941
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9cfc237a8514a1d323313851e14576ba2ba69077
Signed-off-by: Young Tack Jin <youngtack.jin@circuitblvd.com>
Reviewed-on: https://review.gerrithub.io/410529
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8573732b3049e2a5b471e5a0313e39019fdaad5f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/410518
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This will be used later instead of retrieving VS (potentially via a Get
property command for Fabrics) multiple times.
The Active NS List code was previously depending on the VER field of the
Identify Controller data, but this was only added with NVMe 1.2, so we
can't rely on it to detect NVMe 1.1 controllers; it is changed to use
the new cache VS value instead.
Change-Id: Iba9ed5ecbc82b4654973438d119daba0c4cf0724
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/408895
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
In addition to checking for the required version, also use the OAES bits
to determine which features to request.
Change-Id: Idb07e4175cca5609904876bd17d912b50bc6b62a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/409352
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Change-Id: Ia2fd4d31560c9117b167cbf12d2749c303e4cb8c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/408772
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
uint8_t is too small to handle huge payloads. 32M payload already
overflow this.
Change-Id: I083ba7d3ded25b99571d422b7a3a4e7653a8d231
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-on: https://review.gerrithub.io/408677
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I80521c4c01daf033319f88cf273255387a7b5248
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/408403
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This fixes the VFIO hot-remove path, which called remove_cb without
checking to see if it had been specified by the user. The normal uevent
removal path already checked for remove_cb.
Change-Id: I0ad8d2c90a77b16800a8b505cb69ea05b0706d70
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/408392
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is an internal NVMe driver function, so we don't need to allow for
the case where trid is NULL. All callers already passed an address of a
local variable except the unit tests, which can be trivially fixed.
Fixes a static analyzer warning about trid being dereferenced in
nvme_transport_ctrlr_construct() before being checked for NULL in the
caller.
Change-Id: I2bfeb5c92a302093b7c7f2949adcd18baa11855a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/408395
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This was broken by commit 31bf5d795
Change-Id: I8c81c7b76cd47db347ce9c3f8a0e8296b690cb49
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/408240
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9fee91ece79b204962a70fc49d9032abe2c55090
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/408218
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
As of glibc version 2.3.4 onwards getpid() is no longer cached. SPDK
makes calls to it in nvme_allocate_request() which is called for each
nvme request received. This results in a system calls up to millions of
times per second which slows down nvme submissions. Since the pid never
changes, it only needs to be called once in initialization per process.
This improves the performance of nvme_allocate_request() signficantly.
Change-Id: Idee0f06484d459906b9ce1d9b7360a33119c7e56
Signed-off-by: Jonathan Richardson <jonathan.richardson@broadcom.com>
Signed-off-by: Scott Branden <scott.branden@broadcom.com>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
Reviewed-on: https://review.gerrithub.io/407599
Reviewed-by: Scott Branden <sbranden@gmail.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The functions they were referring to have been renamed; rather than
fixing up the function names, use the spec-defined NVMe command names so
it's more understandable. (The second message was also incorrectly
referring to "set" instead of "get", which is fixed as well.)
Change-Id: Id140a91c837d8c913760d2f55318472689c00f45
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/407593
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Similar to the previous patch, some controllers may report 0 namespaces;
we should still be able to send admin commands in this situation.
Change-Id: Ia14e3ce773c63e645199e1c40ba5b37095da2473
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/407497
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
A controller that fails the Set Features/Get Features - Number of Queues
won't be able to create I/O queues, but it may still accept admin
commands.
Change-Id: Iec79d641f7d460448a8d8e1295764f1f03f98594
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/407378
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6e58baaeb09580b5f70e1acf5323376ca0b26bbf
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/407382
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
strncpy is going to be added to the list of banned functions because
it does not guarantee strings are null terminated.
Change-Id: Ic18623c281cca7c3d87732bc7677b284d57685c8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/407023
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I79dad84d1dc58e61eb36b461b41fbd7ee73631fc
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/406899
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Add the event status code to assist in debugging, and reorder the
message so it is easier to read.
Change-Id: I2a1fb900471a482707fbfb6db84bc81cb3b0cdad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/406636
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
- Add support for multi page CNS 0x2
- Use CNS value 0x02 (SPDK_NVME_IDENTIFY_ACTIVE_NS_LIST)
to query active namespaces
- Add an API to iterate the active list
Change-Id: Iea524881fa6e3610a7d85ab02a2005a92fd633df
Signed-off-by: John Meneghini <johnm@netapp.com>
Reviewed-on: https://review.gerrithub.io/401957
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We don't need to re-register memory with the same ibv protection domain
more than once; maintain a global list of already-registered PDs and
reuse the existing registration for other queues.
Change-Id: I7ec9c35ed93083faa00cca4cb5c61104200a4c66
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/405702
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This allows us to remove most uses of spdk_pci_get_device(), which looks
up a PCI device structure from an arbitrary PCI address. This function
is problematic, since it uses internal DPDK data structures that aren't
meant to be part of the public API. There is still one use in the
codebase, which will be cleaned up in another patch.
Change-Id: Ia1fe1f799c240195f6871c1d92821074f884c4e6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/405707
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This function retrieves the UUID from a namespace, if available.
Change-Id: I98c55375948b92eaf429b41fb36dfea4e2b780a2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/404734
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Convert nvme_ctrlr_cmd_identify_controller() and
nvme_ctrlr_cmd_identify_namespace() into a single function,
nvme_ctrlr_cmd_identify(), with generic parameters that should be
suitable for all current callers as well as future users.
These functions were internal-only, so there is no public API change.
Change-Id: I3dbb3e6b00308b67ba1f161f8a6b11b6333fca57
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/404733
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
A few places were accidentally storing namespace IDs in uint16_t.
Change-Id: Iae9d709fb20bc1ac0a584ccd9683b721ce5de961
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403886
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This lets us have a common place to put definitions like the length of
the UUID string, as well as abstract away some of the API warts in
libuuid (non-const values, no size checking for uuid_unparse, etc.).
Change-Id: I80607fcd21ce57fdbb8729442fbb721bc71ccb98
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/402176
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Automatically detect more whitespace errors.
All existing cases are fixed; only whitespace change (verify with
diff -w) except for one comment style fixup in include/spdk/nvme.h.
Change-Id: If750e54b9c8e3421ea6feda5f20184a31431631e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/402360
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
There was a bug reported by Cunyin Chang with regards to how the
cmb_current_offset was calculated when the CMB offset into the CMB BAR
is non-zero. This patch fixes this issue and also fixes the problem
that the last valid offset into the BAR *may* differ if registration
is utilized or not (due to the 2MiB alignment and length requirements
for registered memory).
Change-Id: Id08d6a5a40b828338f6a66599171cc8dd59768a3
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Reviewed-on: https://review.gerrithub.io/401832
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I02ca8aed1bba1922a69a9b22bca22bce06e9c55f
Signed-off-by: Pan Liu <liupan1111@gmail.com>
Reviewed-on: https://review.gerrithub.io/399536
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This can be enabled with -t nvme in debug builds.
Change-Id: I4392d0a7decd65cc567fdf6fd56cac6db0424cd8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/401052
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This function returns uint64_t, but the existing code was returning
the bool 'false' (which is also 0, so there is no functional change).
Change-Id: I546b15974c2d9e4107efe262950e3eb5613e0e9f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/401051
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Unify several similar functions checking for a buffer of all zero bytes
into the util library.
Change-Id: Idfbeffa22add34ac9ed1bd75ee27d6bd8b188940
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/400892
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
adding nvme_ctrlr_destruct_finish because nvme_transport_ctrlr_destruct may
use a destroyed mutex.
nvme_ctrlr_destruct() free "ctrlr_lock" and after that call
nvme_transport_ctrlr_destruct()->nvme_pcie_ctrlr_destruct()(with pci)->
nvme_ctrlr_proc_get_devhandle()->nvme_robust_mutex_lock(&ctrlr->ctrlr_lock);
Change-Id: I55714ea9097d2c9d844a00b5a88fa2d51a3f4469
Signed-off-by: Ehud Naim <ehudn@marvell.com>
Reviewed-on: https://review.gerrithub.io/399605
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2a3c7a272dc08be5a5ecb4339622816482c4cbb0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Reviewed-on: https://review.gerrithub.io/397036
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Enable address translation for I/O buffers within the controller memory
buffer region by registering the CMB using spdk_mem_register().
Change-Id: I44829757ad15fbc3ea96fa494b9fb32dd67a7138
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Reviewed-on: https://review.gerrithub.io/397035
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
For a given hardware queue size, only allow a quarter of the queue size
to be returned as completions in a single call to
spdk_nvme_qpair_process_completions(), and adjust num_trackers to match
so that num_trackers + max_completions_cap doesn't exceed the hardware
completion queue size. This ensures that there is room in the
completion queue if new I/O is issued in response to completions before
we ring the completion queue doorbell.
The choice of 1/4 queue size is arbitrary; this seems to be a good
compromise between completion batch size and number of trackers.
Change-Id: I2c5aad7b98bfc8b33e53242240b2c9254fa05b4e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/393529
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This patch remove need for additional buffer when translating error code
to string.
Change-Id: Iaa60088b5c450581d3cdddbb425119b17d55a44b
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/386114
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Only Makefiles for libraries that directly depend on DPDK (rather than
the SPDK env abstraction) should add $(ENV_CFLAGS).
Change-Id: Ifdf44d3ef8c42bbf7f20edd524b330d00658235b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/392818
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Commit ID "269910c0" removed the support of separate metadata,
for those controllers which can support this feature, SPDK driver
can't be used. SPDK provides APIs such as:
spdk_nvme_ctrlr_cmd_io_raw_with_md/spdk_nvme_ns_cmd_write_with_md/
spdk_nvme_ns_cmd_read_with_md, which can support separate metadata.
While here, re-enable this feature with this commit.
Change-Id: If77c21e9ac700c4b334548ebfa7e8e6286285a64
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/392440
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iedfa8d3de8520836e184f7ef0925822fb705fc67
Reviewed-on: https://review.gerrithub.io/391672
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8519a4b68db44cb8fe6dd251a52bf0f1dca73c32
Reviewed-on: https://review.gerrithub.io/391890
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch set the controller as removed in pcie level when the register
return specific value (0xffffffff), we also return the real value to the
upper level (nvme bdev), which will help the upper level do the work of
hotplug.
Change-Id: Ifad45c760cccbce522506ffbf86495318a6b393b
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/391327
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Require braces around all conditional statements, e.g.:
if (cond)
statement();
becomes:
if (cond) {
statement();
}
This is the style used through most of the SPDK code, but several
exceptions crept in over time. Add the astyle option to make sure we
are consistent.
Change-Id: I5a71980147fe8dfb471ff42e8bc06db2124a1a7f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/390914
Reviewed-by: <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
To make the error message more clear.
Change-Id: I95a9a3e06f4473da30d022134a8da764f165a070
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/388880
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Disambiguate the log components from the trace functionality
(include/spdk/trace.h).
The internal spdk_trace_flag structure and related functions will be
renamed in a later commit - this is just a find and replace on
SPDK_TRACE_* and SPDK_LOG_REGISTER_TRACE_FLAG().
Change-Id: I617bd5a9fbe35ffb44ae6020b292658c094a0ad6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/376421
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
NVMe specification 1.3 added a new Admin command: Doorbell buffer config,
which is used to enhance the performance of host software running in
Virtual Machine, and the Doorbell buffer config feature is only used
for emulated NVMe controllers. There are two buffers: "shadow doorbell"
and "eventidx", host software running in VM will update appropriate
entry in the Shadow doorbell buffer instead of controller's doorbell
registers.
Change-Id: I639ddb5b9a0ca0305bf84035ca2a5e215be06b46
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/383042
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This patch just make sure SPDK support the hotplug of vfio-attached devices,
but there still issues when we get hotplug events from kernel, this patch tested
based on my modified local vfio-pci driver which just for test purpose and not upstream.
Change-Id: I974177e70e4133d2a583d00e714a7607765d3548
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/390580
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
There are two ways to set stockets to nonblocking type:
- ioctl with FIONBIO
- fcntl with O_NONBLOCK
Those two should be equivalent for sockets used in SPDK.
During testing it was shown that VPP interprets only
the second type, so this patch changes all occurences of it.
When here, more descriptive error logs were set in case of
failure.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifa5b30e3a4fa04fe23f41fa2ae9dab4b01dd7d3c
Reviewed-on: https://review.gerrithub.io/388816
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Only multi-process shared controllers should be inserted into the shared
list in g_spdk_nvme_driver. To accomplish this, create a second
per-process global list of attached controllers (g_nvme_attached_ctrlrs)
and rename the driver struct field to shared_attached_ctrlrs to clarify
its purpose. Additionally, a new helper function, nvme_ctrlr_shared(),
returns whether a given controller should be on the shared or
per-process list.
Change-Id: I46d4e558ece8b7fc3d28868e32bb56d794f21aab
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/389190
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Simplify the PCIe transport by using an existing function to look up a
controller by transport ID.
Change-Id: I261865df1ba23069b052ca64944b7637d70c85ba
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/388701
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Each process should manage its own list of controllers that are
initializing; the list doesn't need to be shared between processes.
This is the first step toward preventing non-PCI controllers from being
added into the shared attached_ctrlrs list.
Change-Id: Ia6f85fe89e28a04f0950da5362bb2f49d1b76da9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/388695
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Use the env abstraction PCI functions to compare PCI addresses so that
details like whether or not domain is specified or whether 0-padding is
present don't affect the comparison.
For example, 0000:01:00.0 should compare equal to 01:00.0.
Change-Id: I9f3aaeb5f8fdbf3e246e31a41b4c09151288015e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/387202
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Because ppc64 has less strict memory ordering behaviour
than x86 we need to introduce a memory barrier when
polling for completions due to possible reordering
of tracker and cpl access.
Change-Id: Id17116c38b2ba69154c175c539fc97c60897deb0
Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
Reviewed-on: https://review.gerrithub.io/383728
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Change-Id: I482ad9e722f6fb775c1c91c8661212250a6914bc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/386722
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Some devices may report a RTD3E time that is shorter than their actual
shutdown time in practice; force the timeout to be at least 10 seconds
to allow for a reasonable amount of shutdown time.
This doesn't add any extra delay for devices that do complete the
shutdown process within their reported RTD3E time, since we will return
as soon as the device reports that it is finished shutting down.
Change-Id: I365e66ba6a938400be516df170bd3ff288810caf
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/386719
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
cmd and cpl rings must be aligned on page size boundaries.
Change-Id: I103eed32adfaa4388bc7d672ee166973f796b343
Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
Reviewed-on: https://review.gerrithub.io/383727
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When a ctrlr is failed to initialize and to respond other
requests, the nvme_ctrlr_fail() will be always called. Add
a log there to have the traddr information so that applications
can know which ctrlr has the problem.
Change-Id: I951062a51349af81a505472f79e3c00a1ead2fbf
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/386189
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I3db00323c20786713750d13a61b1531d8b1ce7f6
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/386087
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
NVMe 1.2 introduced a new Identify Controller field, RTD3E ("RTD3 Entry
Latency"), which allows the device to report the expected time for a
normal shutdown. Use this as the timeout for the shutdown process when
available instead of hard-coding 5 seconds.
Change-Id: I14e7223c81ba397771cf00b49f034f25d21b6e82
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/385301
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
spdk_pci_device_claim() can be used to ensure only one process
at a time uses any given PCI device. Previously this was only
used in the bdev_nvme driver - other apps like nvme/perf do
not use spdk_pci_device_claim() and could effectively rip out
the device from a running bdev-based app like the NVMe-oF target.
So instead of modifying all of the nvme apps, put this logic into
the core nvme driver instead so that all applications get the
benefit transparently. Save the fd when the controller is constructed
and then close it when the controller is destructed to handle the
detach (including hotplug) cases.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5dc48a2e41dc06707800f15a9e1f9141477628c6
Reviewed-on: https://review.gerrithub.io/385524
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
For using SoftRoce, there will be packets loss,
so we need to use this value for re-transmission.
retry_count means the maximum number of times that
a data transfer operation should be retried on the
connection when an error occurs.
Change-Id: Ibd81a33d3838bbdf0c054f01666e7ec59bb0a38f
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/382079
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
With this new API, callers can attach one specific ctrlr identified by
the transport ID directly along with optional ctrlr opts. If connecting
to multiple controllers, it is still suggested to use spdk_nvme_probe()
and filter the requested controllers with the probe callback.
Two primary use cases:
1) connecting to the NVMe-oF discovery controller
2) more straightforward way to connect a specific controller (avoiding
the probe callback)
A typical usage of this API with specific ctrlr_opts:
1. struct spdk_nvme_ctrlr_opts user_opts = {}
2. Call spdk_nvme_ctrlr_get_default_ctrlr_opts(&user_opts, sizeof(user_opts))
3. Modify the content of the initialized user_opts with user required value like
user_opts.num_io_queues = 8
4. Call spdk_nvme_connect(&trid, &user_opts, sizeof(user_opts))
Change-Id: Idf67ee5966f6753918c12604342c892d2f3bbe3a
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/370634
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This change is relating to add a new public API spdk_nvme_connect() under
include/spdk/nvme.h. This new spdk_nvme_connect() API will connect the user
specified trid and have a user optional ctlr opts. Rename this API and make
it as public.
A typical usage of this API as following:
1. struct spdk_nvme_ctrlr_opts user_opts = {}
2. Call spdk_nvme_ctrlr_get_default_ctrlr_opts(&user_opts, sizeof(user_opts))
3. Modify the content of the initialized user_opts with user required value like
user_opts.num_io_queues = 8
4. Call spdk_nvme_connect(&trid, &user_opts, sizeof(user_opts))
Change-Id: Ideec8247365ebf7dd15069e29821be8ea27b08be
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/380849
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
The newly added UT code will exercise the cases that opts_size is
smaller and equal to the current version of spdk_nvme_io_qpair_opts.
This is for the backward compatibilty when SPDK components are built
as the shared library later.
Change-Id: Ic906d765f5b638070e3d9c8e38827577c625b679
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/380893
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Add a new parameter for the default ctrlr opts initialization.
This is to make sure future compatibility when SPDK components
are built as a shared library. User's version and SPDK's version
may be in different size.
The change here is to make sure the backward compatibility when
new fields are added in the struct spdk_nvme_ctrlr_opts.
Change-Id: Icfc9640993cb06063b825d4df5835d920dd374e5
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/380846
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
A random host ID is generated per SPDK application startup if the user
doesn't specify a host ID during controller startup.
This also changes the default host NQN for NVMe-oF connections to a
random UUID NQN based on the host ID.
Change-Id: Ib0f70dd63e53087716842b412a1f134a9991d4da
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/380528
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This was removed in NVMe 1.2.1 ECN 002.
Change-Id: Icacd53c1f33043cf1c9f30bff1fb9dce1efa69b3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/378681
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
SPDK already uses DEFAULT_IO_QUEUE_SIZE and MQES to decide the correct
queue depth of NVMe queue pair, hardcoded it to NVME_IO_ENTRIES(512)
does not make sense if users want to set queue depth bigger than 512.
Change-Id: Iaa73fc79e055292ae9bd19af0c8c12f257ae5c46
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/379052
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Use SGL with contiguous physical buffer metadata as the default PSDT
if NVMe can support SGL.
Change-Id: I99192a8b5408f1b034d5112d77d13639b7558f01
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/378699
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
spdk_nvme_ctrlr_cmd_io_raw_with_md() will be verified
on Cosmos+ OpenSSD as soon as it will support meta data.
Change-Id: Ib5f3f1f1eba66d0147a566804395bfa5ec959c2f
Signed-off-by: Young Tack Jin <youngtack.jin@circuitblvd.com>
Reviewed-on: https://review.gerrithub.io/377428
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
If user specify a trid, we should not attach other ctrlrs
Change-Id: I73a4278c1d7551908feb56d01a1c41c0d049bb91
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/377653
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reason: In our default configuration, we use
nqn.2016-06.io.spdk:init
Change-Id: Ic840b41230f53d5d97166a38faf7c2109fa6b41a
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/377463
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Users don't need to provide nvme debug flag for the error cases.
Change-Id: I00c29e2b8ab470b0233d94acec52b4bec129728c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/376708
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Allow the user to set the source address when connecting to a NVMe-oF
controller.
Change-Id: Ice3add4b2cd3b64fdb8d0d7807d2235f90fd86b1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/375837
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Replace SPDK_TRACE_DEBUG with component-specific flags.
Change-Id: Iee7eafab5e6ac8713f247323a18552b5afb0e86a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/375834
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This matches the name to the behavior and prepares for addition of a new
log macro for "info" log level.
Change-Id: I94ccd49face4309d3368e399528776ab140748c4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/375833
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id2d38779b2178ce56391ac75781c1e41e4a83e1d
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/375130
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Certain vendors do not report correct number of queues allocated in "Set Features/Number of Queues" completion CDW0 per spec.
As a work around, issue "Get Features/Number of Queues" and rely on the value provided there.
Change-Id: Ib9cc4dcf1bdb732413becc751883a7311c6f672f
Signed-off-by: Konstantin Vyshetsky <kon.vyshetsky@stellus.com>
Reviewed-on: https://review.gerrithub.io/375234
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This can be used in callbacks when a user has a namespace but needs to
call controller specific functions such as alloc_io_qpair.
Change-Id: I00c931e2846e42f540c17f3254fe97a45e9bd079
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/375022
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
PAGE_SIZE is the host memory page size, which is irrelevant for the NVMe
driver; what we actually care about is the NVMe controller's memory page
size, CC.MPS.
This patch cleans up the uses of PAGE_SIZE in the NVMe driver; the
behavior is still the same in all cases today, since normal NVMe
controllers report a minimum page size of 4096.
Change-Id: I56fce2770862329a9ce25370722f44269234ed46
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/374371
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
An optional field was added in NVMe 1.3 to indicate the optimal I/O
boundary that should not be crossed for best performance. This is
equivalent to the existing Intel-specific stripe size quirk.
Add support for the new NOIOB field and move the current quirk-based
code so it is updated in nvme_ns_identify_update().
Change-Id: Ifc4974f51dcd59e7f24565d8d5159b036458c6e5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/373132
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
replaces all references to strerror in the spdk lib directory with
references to the thread safe strerror_r
Change-Id: I80d946cce3299007ee10500b93f7e1c8e503ee41
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/374012
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This also changes the SGL -> PRP case to translate each 4K page from
virtual to physical, in case the buffer is not physically contiguous.
Change-Id: If027f9d656c52c56504f0c64cd4464e16440df63
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/371616
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Unmapped blocks on some older nvme devices will read zero even if the
device does not explicitly define the unmap behavior.
Change-Id: Ia825a406cbd01f89192c300cfe35013fb4aed715
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/372006
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Change-Id: Ia45d356fbc2c4baea86a96eb28264f104f593a9c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/373156
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The value for lba_count is stored in a 0-based 16 bit register. here we
confirm that the value passed to that register is no larger than 2^16.
Change-Id: I234e55fc2b61338444dfe8f734e76f958d1f0443
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/372370
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
1) Rename _nvme_ns_cmd_split_sgl_request to
_nvme_ns_cmd_split_request_prp.
2) Add _nvme_ns_cmd_split_request_sgl. This
function will be used with controllers that
support SGL, to make sure the request does not
span more SGEs than the controller or supporting
software supports.
While here, add a useful error message for both
the prp and sgl cases, if the child_length is not
an even multiple of the lba size.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ifd04aee3c571359009260fe46400ce1844511aa3
Reviewed-on: https://review.gerrithub.io/372359
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
For pcie, this just equals the number of SGLs we can fit
into the per-tracker memory.
For rdma, this is just set to 1 for now since nvme_rdma.c
does not support multiple SGEs yet. Once that support is
added, this will change to use MSDBD (Maximum SGL Data Block
Descriptors) instead from the controller identify data.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I34a4c546b5ff46918a296a73ed8cbcc6c9879d5a
Reviewed-on: https://review.gerrithub.io/372358
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Requests may need to be split in which case a child
request may have a size smaller than the SGE that
contains the child request's payload.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6c7ab76104d56fa9dde168cfdddb6320c7157f98
Reviewed-on: https://review.gerrithub.io/372347
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
For ioq, target is supposed to respond with the same
cntlid as specified in the connect capsule, but the
Linux kernel target doesn't actually do that. It really
only needs to be set on the adminq connect, so just
do that.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4eec605f856bdad2c8614d505241566a9fe292ab
Reviewed-on: https://review.gerrithub.io/372345
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The Linux kernel NVMe-oF target can easily be misconfigured to report
newline-terminated strings from the discovery service, since its
parameters are set by echoing into sysfs files, and echo adds a newline
by default.
Newline characters are not allowed as part of the TRADDR and TRSVCID
fields, since they are defined as ASCII strings in the NVMe specification
sense, which limits the acceptable characters to the 0x20-0x7E range.
However, we can add a workaround for misconfigured targets with trailing
newlines that shouldn't impact any valid configurations.
Change-Id: I5db183d5637128fa8d50c245f4bfa965cc3ce8e2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/370593
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2b6a22591d0a5f7435d9fd9dd8d6a1c854da3d89
Reviewed-on: https://review.gerrithub.io/371978
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paweł Niedźwiecki <pawelx.niedzwiecki@intel.com>
Add a new struct spdk_nvme_io_qpair_opts to allow the user to override
controller options on a per-I/O qpair basis.
Existing callers with qprio == 0 can be updated to:
... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);
Callers that need to specify a non-default qprio should be updated to:
struct spdk_nvme_io_qpair_opts opts;
spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts));
opts.qprio = SPDK_NVME_QPRIO_...;
... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts));
Change-Id: I8ac3ea369535cfde759abbe75e1d974b6450a800
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/369676
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
gai_strerror() is documented as thread-safe, so this should be safe to
use from a library.
Change-Id: Ia0ec0b5c387db8993dd1ba3a3029562f25cc210d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/369642
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Support both SPDK_NVME_FW_COMMIT_REPLACE_IMG and SPDK_NVME_FW_COMMIT_REPLACE_AND_ENABLE_IMG.
Return code will specify if conventional reset is required.
For now, return error if subsystem reset is required.
Change-Id: I41a05675a210dd0bbf170517b32ee9e05da9a797
Signed-off-by: Isaac Otsiabah <iotsiabah@us.fujitsu.com>
Reviewed-on: https://review.gerrithub.io/367287
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5b77212335e2f10ce263dc59baad87236b98048a
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/367305
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Also add adrfam to the NVMe bdev JSON config output.
Change-Id: I9472bda04947cffc0df9b02eba0035bac01b7d7b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/367292
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Nothing in the spec indicates that NQNs should be case insensitive, and
we have fixed this elsewhere (e.g. commit df70bc1559: "nvmf: use
case-sensitive comparison for NQNs").
Change-Id: I4a48d1c7f25ec5af9ce4d73f1bf2fa543236503a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/367106
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I6010465d573decd93ddf9881392b3d807cd52918
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/366663
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
PCIe transport IDs are a non-standard extension to the NVMe-oF transport
address, and they only use the transport type and address fields of the
structure. Add a special case so that the rest of the fields are
ignored for PCIe addresses. All other transport types are NVMe-oF
addresses and should compare all fields.
Change-Id: I45ed143ea1712d17c6de8082677deeefd395c8a2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/365916
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Sync the spec header with the major changes for NVMe 1.3.
Some of the added fields may have been added in previous
versions of the spec.
Change-Id: Ia50a52f5192cf450bb5cc2d18fcc1f92ebce7f77
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/362046
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I941a4cb5563cebb2e68b48d3a74b4b73af0e9657
Reviewed-on: https://review.gerrithub.io/365662
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Change-Id: I1e272c63a5d863c92f1aa8299a9d98dcb72d0b13
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/365082
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
These commands should be treated as aborted by spec,
so correctly deliver abort notifications when a
qpair is deleted.
Change-Id: I8af47a3f42f5695ef8e1a70813662e69102720b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/364681
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Preivously, we can only get part of the log page
entries due to the limitation of buffer. With this
patch, we can get all.
Change-Id: I81b8afc73c108dac65d1ff5fe8d03bda41fa29a1
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/363323
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Also provide an option in perf tool let users to
disable it.
Change-Id: If4952513d77cecaa4f9403fbea811d86916ee87c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/363311
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
- rename spdk_malloc_socket to spdk_dma_malloc_socket
- rename spdk_malloc to spdk_dma_malloc
- rename spdk_zmalloc to spdk_dma_zmalloc
- rename spdk_realloc to spdk_dma_realloc
- rename spdk_free to spdk_dma_free
Change-Id: I52a11b7a4243281f9c56f503e826fd7c4a1fd883
Signed-off-by: John Meneghini <johnm@netapp.com>
Reviewed-on: https://review.gerrithub.io/362604
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
An ECN to the NVMe 1.3 specification has clarified that the NQN may
contain 223 bytes before the null terminator. Make all of our NQN
length checks consistently enforce this behavior.
Change-Id: Iebfd57d11abea64964c7a6ad9d886e40efa243c3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Unit tests will be added as part of a separate patch updating all UT for
nvme.c. Global used for timeout value so it can be easily overwritten
by the upcoming unit tests for this function.
Change-Id: I7fc15aab91601ac57c94cae266b212c0998d2495
Signed-off-by: paul luse <paul.e.luse@intel.com>
The VirtualBox emulated NVMe device will intermittently
hang on the first read/write command after an I/O
qpair has been allocated. The frequency of the hang
diminishes if a delay is added after allocating the I/O
qpair - until it disappears completely with a 100us delay.
So add a quirk to insert this delay.
Note - the 100us delay was tested by running
the hello_world example app 50000 times.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I237e31b1b8a1a1e28262851ae0a21cd7345f0f1a
Fixes a scan-build warning about using qpairs after they have been
freed.
Change-Id: I263eabd6b784acf540c66136965f7705ef110a78
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fix up the existing comment blocks misaligned in the first column.
Also add line numbers to the comment checks.
Change-Id: I9d28c365271df36e7013d74cbb02d0023ab4f581
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
A 100us is so small that applying the quirk to the specific
SSDs that require the delay is more trouble than it is worth.
So remove the quirk and always wait 100us before re-enabling
the NVMe SSD during initialization.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id6a8cc6e35d103fffdf135580301fc3e5b27e722
Also avoid an spdk_get_ticks() call in the default
case where a timeout_cb_fn is not defined.
On my Intel(R) Xeon(R) E5-2699 v3 system with an
Intel(R) P3700 SSD, these modifications reduce software
overhead per I/O by 3-5% (as measured by the SPDK
overhead tool).
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5bb5a87747b15d9e27655fabcd2bc1a40b0b990e
Queue aborts that would exceed the abort command limit
in software as a convenience for the user.
Change-Id: I8c1f0380984cc6c0cdb453db961939a7f571b336
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Instead, pass NULL when an ADMIN command times out.
We don't expose the admin queue to the user.
Change-Id: If0768d329a689f6f7c3734c9d419e680d7378ed1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
For each command that times out, call the timeout
callback one time if the user registered one.
Change-Id: Iaad39a886468e89bef63fe292c5cad1dce97a57c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
It has been discovered that some devices require
a very small delay before writing CC.EN to 1 after
CSTS.RDY goes to 0.
Change-Id: I73d31726d17ebf5bbec7ee528e2f98fcd05234dd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This isn't the indentation pattern I would have chosen, but
it's a complicated negotiation between what I want and what
astyle will let me get away with.
Change-Id: I4909587823931842ac3f227134e1d05e7d80da74
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Any Intel device reporting device ID 0x0953 needs this quirk.
Change-Id: I690b01ecf05105df00ec8cf6f2da7f7c0a601aa8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This prevents a deadlock if the user immediately
calls spdk_nvme_detach.
Change-Id: I79f28abe163cbbf184bea907692c44aa4e1c8893
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The message about the uevent socket is not a fatal error; it just means
that hotplug monitoring will not work.
Change-Id: I29f6a253e96a86420c0fde9e19135f9f1d229bb9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
remove the unnecessary rte_eal_pci_probe_one() in function
spdk_pci_device_detach(), this could cause error message when we
terminate the application, it will also not make sense try to probe one
device after we detach it, we could call spdk_pci_nvme_device_attach()
instead of spdk_pci_nvme_enumerate() when we have one given device address,
dpdk will try to scan the device and add it back to pci device list then.
Change-Id: I35f5bb412249bb20da57394f0531c10a49691906
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Register all spdk_malloc() memory regions as ibv_mr in a spdk_mem_map
so we can look up the RDMA key for the user's buffer and pass it in the SGL
directly, rather than copying through a pre-registered bounce buffer.
Change-Id: I7340bc2020b5256750c95dbd24ba67961404e5e7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The extended LBA format flag should be initialized after namespace
capability flag.
Change-Id: Iad479b454bb4e31120c17d40ae23937a099c6f8f
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
The send completions must be processed prior to the
recv completions. However, if the completion queues
are separate this leaves a small window where
a send+recv completion arrive between polling
the send_cq and the recv_cq, resulting in the code
seeing the recv completion prior to the send
completion.
By combining the completion queues, this eliminates
any potential gap. The send completion will always
be processed before the recv completion.
Change-Id: I06bfef6af48559d0b9e00524ebc10f1a102e7387
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The actual uses of intrinsics are already guarded by feature-specific
ifdefs in nvme_pcie_copy_command(), but the header itself should also
only be included when it will actually be needed.
Change-Id: Ife65d6432b8dfd9d9db80fe4e385ab76491874c0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These allocations need to be from memory registered with the SPDK env
library to allow future work on automatic ibverbs memory registration.
Change-Id: I6ec6999ecd6d6bf6ba4ab159630f7d01f3d46154
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This can be used for issuing an abort for the timed-out command.
Change-Id: I3c5727fdddc156cd7c8f99afbc3e6da8e73bba56
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Move the ibv_recv_wr initialization in
nvme_rdma_alloc_rsps. Thus we can save some
CPU times
Change-Id: Id449b2684290431f8b3ba97ec4058171d34038bf
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
We do not need to set it for submission since the contents
are same
Change-Id: I345094e2e8a858b318be73d28f09393566587d95
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Move cb_arg to the first argument to match the other NVMe callback
function signatures.
Change-Id: I4e699c8071dcb7ba4ce3cdb82ee985600208204c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This has been reported for a number of different device
types. We suspect these devices are technically out of
spec, but they work with most other available NVMe
drivers on accident.
Change-Id: I529cfc03fc314cbab2a1cd40620bf1dd5b54182d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
For namespaces with end-to-end protection information, metadata size
of exactly 8 bytes, and extended LBA configured, the NVMe driver would
calculate the size of the data block incorrectly. The NVMe spec has a
special provision for this specific case (8-byte metadata only) and
PRACT = 1 that requires that the host does not send the metadata as part
of the host memory buffer.
To fix this, clean up the calculation of the per-block data transfer
size by adding a new extended_lba_size field in the namespace, which
represents the total size of data to be transferred per block based on
the namespace's configured metadata size and whether it transfers
metadata as part of the data buffer. Then add the special case for
PRACT = 1 and PI configured and extended LBA in the R/W helper
functions.
Change-Id: I0b383a58c773cac06e6c018858b57129064c6059
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These were repeated a few different places, so pull them into a common
header file.
Change-Id: Id807fa2cfec0de2e0363aeb081510fb801781985
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This removes one addition from the submission path (negligible, but a
nice side effect), but also opens up the possibility of reporting the
total time an I/O took - since we are always tracking the submission
time anyway, there is no extra cost to report it in the completion
callback.
Change-Id: I7129e7c09d20da8082042a7622d045846461dd9c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For infinite timeout states, instead of printing UINT64_MAX as a
decimal number, interpret it as "no timeout" instead.
Change-Id: I579f5857f96286734940ab5f493261e60354c4fe
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The PCIe transport initializes the quirks directly, so the generic hook
to get PCI ID is no longer necessary. This path was dead code.
Change-Id: I25bdaa598db53e4312a264d9d8356d1b416696e5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The logic to fail queue pairs when the controller is failed should be
handled in the generic code, not in the individual transports.
This also allows nvme_qpair_fail() to be private to nvme_qpair.c.
Change-Id: I6194576dceb35073b9af8847e59314900028637c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is more CPU efficient than only grabbing one
completion per call to ibv_poll_cq.
Change-Id: I0c70d33639f0f345482d9e7c810f9c6723937058
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
In our previous code, we did not ack the event in
exceptional cases when we get a event via rdma_get_cm_event.
Thus, the code may block with in this statement:
rdma_destroy_id(rqpair->cm_id);
in some exceptiaonal cases. And this patch will solve this
issue.
Change-Id: Iddb6fb5356a5ee0ed04e261a040ba53042fca302
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
This make sure the qpair failure could be started from upper level application.
Change-Id: I7e04fe36929cc634ddf0078db96fbc40afb38f8c
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
The g_thread_mmio_ctrlr should be not NULL pointer when it enter the
handler function.
Change-Id: I45dba601c672b16e2c6feafd9059bafde0d8f1b4
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
If namespace is formatted with per lba metadata feature and also disable end-to-end protection
feature, host couldn't use per extended-lba metadata area.
Signed-off-by: Zhihao Zhang <thomas.zzh@alibaba-inc.com>
If the user asked for a specific PCI address in spdk_nvme_probe(), we
need to return 1, not 0, for the other PCI addresses that don't match
when enumerating. 0 means to attach the PCI driver, whereas 1 means to
continue enumerating.
With the previous behavior of returning 0, all NVMe devices would be
attached to the DPDK PCI driver, even if the user did not request for
them to be probed, and further calls to spdk_nvme_probe() would not find
any devices.
Change-Id: Ifbbcd7d1abe8ab535b6957855172e66a3e69fbe4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Currently we use the pci functions provided by DPDK,
it identifies the device by class id related
info but not by pci bdf info, so we can add the filering
by pci_addr in pcie_nvme_enum_cb function.
Change-Id: I5942e98853f00fc10fa6aae5c113517653d1b357
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Since nvme_ns_cmd.c now walks the SGL, some of the test code
needs to also be updated to initialize and return correct values
such as ctrlr->flags and sge_length.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I521213695def35d0897aabf57a0638a6c347632e
Preivously, we only supports probe the NVMf target
via discovery info, now we can support to directly
to connect it.
Change-Id: I08ce1d95de6744286357e68b48c97b773b902ac8
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
(1) Add nvme_rdma_build_sgl_request function
(2) Merge nvme_rdma_pre/post_copy_mem to nvme_rdma_copy_mem
Change-Id: I86abab821b32b4da0aa9489a6b9f7dc430333159
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Now that the hotplug code is isolated in nvme_pcie.c, it can call the
PCIe transport attach function directly.
Change-Id: I2df3b9168473b537cc9b13367e06d3d3b6fa22be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The spdk_nvme_qpair::num_entries value is never used in the common code,
so move it to the individual transport qpairs to make it clear that it
is a transport-specific implementation detail.
Change-Id: I5c8f0de4fcd808912ba6d248cf5cee816079fd32
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The definition of SPDK_UNREACHABLE uses the build-time DEBUG definition,
which is not available in the public API.
Change-Id: I1862c99fa5c85ccd3483f94e9c35de531da57f3c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Instead of passing the work completion, just pass the
response index. This keeps the work completions localized
to the polling function.
Change-Id: I0e6a1d8564200b5ac3aa43dfd58ae152d439bbd8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This eliminates an if statement, since the two callers
of this function know the desired queue size.
Change-Id: I28fabac8613f7b8fc7d96cf95b085b6e4dcf985f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Just call the regular qpair create function instead.
Change-Id: Ic35b1eb6fcdf0d82733ea573a493f583dd63d5bd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Use the num_entries value in the generic qpair instead. These
values had to match anyway.
Change-Id: Ia6400fbaba97df3ef6db4dc07a2ab95af1e5143f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Avoid an extra level of pointer chasing when we are filling out the NVMe
SGL.
Change-Id: I1a40af16fda80f7480c419524876bfb1a1902eb8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow it to be better be reused for some future patches
enabling splitting of non-PRP-compliant SGL-based requests.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ica38fd6cf191f72baa524bcc4896b3c9939ab762
This intermediate function is no longer needed.
Change-Id: I3523cc6d8f3b290165a953d42cca8b76eda762c5
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Sending the fabric connect command is part of establishing
a connection, so move it into the main connection-establishing
function.
Change-Id: I55e7ffdd16b576c81b51d7d3910203f9afc1f4c2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This function initializes the members of an existing
qpair struct. It doesn't construct one from scratch.
Change-Id: I0b9afac1ad25cfb217efd146702f693c74f5f697
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
No need to allocate all of the requests and responses until
we know a connection can be established.
Change-Id: I072a10aadfd7ced773634448f7d7e788622d0a4c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The code is clearer if this function is incorporated
into its only caller.
Change-Id: I33901cddf80ae27896b2acfd1b9e7d212f21f5f3
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is resolving the address and route to the target, not
binding a socket to an address.
Change-Id: I80055481ed2e020410a1e186a4e7371b60faaee9
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
They were very close to the same already, so finish the job.
Change-Id: Ifba9e3b2d11a3e70cbfbe46f57a67552db2757ed
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
We should be sending the bounce buffer's remote key to the target so it
can put it into an RDMA SGE on the remote side.
Change-Id: Icded155ad2292c67baa722f001c9c07178bc2754
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
There is no particular reason for this to be 127; make it 128 to at
least be consistent with the PCIe transport.
Change-Id: I60500e0044d3549ba6350e1f35f09d624848bd21
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This function was only called from one place and saved no
lines of code.
Change-Id: If5e653732df57c1f2c93e20cf4f286eac31df91c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This matches the behavior of nvme_pcie, which queues a request if no
tracker is available.
Change-Id: Idbf6c951c89451cfea22ec6bc553ff46f988f818
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the caller pre-allocate an rdma_req and change req_init() so it
only does initialization, not allocation.
This is necessary to distinguish between rdma_req allocation failure and
other types of failures, which will become important in future patches
when requests will be queued if rdma_req allocation fails.
Change-Id: Ie6edebc1b5f05001b42fc959a29ce0ea6875e41e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Simplify the control flow and match the name of the function to its
purpose.
Change-Id: I65bad7e3b2ef710ca29eff9799b8dcaae3999315
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the qpair construct functions private to the transports - it
doesn't need to be called from generic code.
Change-Id: I5f730a4bcf60ce231fe27bc8f4c3c39cb647dd2d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a transport callback to return the maximum queue size, and enforce
it in the generic nvme_ctrlr layer.
This allows the user to tell what io_queue_size was actually selected by
the transport via the ctrlr_opts returned during attach_cb.
Change-Id: I8a51332cc01c6655e2a3a171bb92877fe48ea267
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Equivalent to commit 6ab28a201b except now
for commands instead of responses.
Change-Id: Ibe4382dc0f65c1b90c2cee2ad285bbdd21b96a89
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The only field of bb_sgl that was actually used is lkey, and that is
already stored in bb_mr.
Change-Id: I790369a06ce223f88e356df20a9d9a74a93ff225
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Refactor the discover log page processing into a loop that calls a
function for each log page entry. This sets us up to add support for
multiple Get Log Page calls to handle larger discovery service lists.
Change-Id: I85676ada375d0dadda2a3f4ab6331123ac7aaf60
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This isn't used yet in the NVMe library, but it will be necessary later
for supporting non-IPv4 addresses.
Change-Id: I167ce63ad25b0e0c9aa192b12d764c8d078e67f9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This better describes what the field controls (it does not affect the
admin queue size).
Change-Id: I851ae46fb4ed0fce819af07ae235824e0fc817e6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For requests split in _nvme_ns_cmd_split_request() the payload offset is
set after children are created using recurrent call _nvme_ns_cmd_rw().
This makes impossible to reset SGL to proper offset in
incomming patches that split non-PRP complaint SGL requests.
To change this the payload offset is set after each request is allocated
in _nvme_ns_cmd_rw() not in _nvme_ns_cmd_split_request().
Change-Id: I9d3b2e3bbd9d93a4c8a37e1db8c4e01276e2cacb
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
This is preparation for handling non-PRP compliant SGL.
Change-Id: I445790f9802292971256cf821d9730814c95a073
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
This is preparation for handling non-PRP compliant SGL.
Change-Id: I49c3745498411c5ff9e17cd08f181d4d434c2d08
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Don't leave garbage from previous discover entries in the trid we are
returning to the user.
Change-Id: I60ae5932db4a95cedb8df1ff98a2479220b55ce4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The other simplifications to probe_info and trid made the
trtype argument redundant.
Change-Id: Ie7bea4e2204e690dc4909eeacd065e0722b53272
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
