Commit Graph

246 Commits

Author SHA1 Message Date
Ben Walker
958c68f175 nvmf: Add an rdma_device struct
There is one of these per ibv_context.

Change-Id: I660fed954c5a77fed9a6bda2321583ef05b30439
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/373837
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-08-14 15:45:43 -04:00
Ben Walker
ecc436fcab nvmf: Remove global g_rdma
The transport API now allows for multiple transport
objects, so allocate them on demand instead of using
a single global.

Change-Id: I5dd35f287fe7312e6185c75ae75e2488ec8cc78e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/371990
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-08-11 19:44:35 -04:00
Ben Walker
baa936a173 nvmf: Replace transport controller with poll_group
For now, this is a name change of controller to poll_group
in the transport layer. Over time, the poll_group will
become a more general concept than a controller, allowing
for qpairs to be spread across cores.

Change-Id: Ia92a2934541ad336f462f73175d53aaaf021f67b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/371775
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-08-11 19:44:35 -04:00
Ben Walker
6fb907323f nvmf: Replace pointer math with SPDK_CONTAINEROF
Change-Id: I10857de922d5a17131910aca92c73995ea6ab8f6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/373828
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-08-11 19:44:35 -04:00
Ben Walker
7e3b9f25ba nvmf: Clarify transport API for listen and accept
There are now three simple functions on the transport:

listen(transport, trid)
stop_listen(transport, trid)
accept(transport)

This makes the code quite a bit simpler.

Change-Id: I550343a084b5c095240703952c8c07ae535b5c16
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/371774
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
2017-08-08 15:46:28 -04:00
Ben Walker
31d033f944 nvmf: Transports are no longer global
Create one transport per nvmf target. Today, there is just
one global nvmf target, but this paves the way for multiple.

Change-Id: Iaa1f8c5e7b3c1e87621ef2a636c68c2dd8fd929e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/371748
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
2017-08-08 14:09:45 -04:00
Ben Walker
1d304bc5d8 nvmf: Rename spdk_nvmf_conn to spdk_nvmf_qpair
Match the terminology used in the NVMe-oF specification,
which is queue pair. For the RDMA transport, this maps to
an RDMA queue pair, but may map to other things for other
transports. It still is logically a "connection" from
a networking sense.

Change-Id: Ic43a5398e63ac85c93a8e0417e4b0d2905bf2dfc
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/371747
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-08-08 13:46:48 -04:00
Ben Walker
03788f93df nvmf: Rename spdk_nvmf_session to spdk_nvmf_ctrlr
This is just a rename - the functionality hasn't changed.
Use the same terminology as the specification (which is controller)
so those familiar with the specification can more easily
approach the code base.

This is still conceptually equivalent to a "session" in the
networking sense.

Change-Id: I388b56df62d19560224c4adc2a03c71eae6fed0d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/371746
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-08-08 13:46:48 -04:00
Ben Walker
1c83074299 nvmf: Use spdk_nvme_transport_id
The NVMe-oF target was written before we defined
spdk_nvme_transport_id. Now that we have it, go back
and replace all of the locations where we individually
tracked traddr, trsvcid, trtype, etc. and use a trid.

Change-Id: I84334a12c7581f414c1e84680f122fe885a3b9dd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/370744
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
2017-08-07 17:07:58 -04:00
Ben Walker
29f6172a56 nvmf: Use trtype enum in transport instead of strings
Change-Id: Ie05f58e677107072fea6cc7702bab47a077cb595
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/370743
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
2017-08-07 17:07:58 -04:00
Daniel Verkamp
b0e128db3d nvmf/rpc: add adrfam to listen address
This allows the user to optionally specify the address family for
construct_nvmf_subsystem (default is IPv4).

Note that the RDMA transport still only supports IPv4 because of the way
it binds to the listen address; this will be fixed in a separate patch.

Change-Id: I534ed75f6f81e53559d1bebcd2f34f1a2b210a97
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/367429
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2017-07-05 13:01:20 -04:00
John Meneghini
8a44220b1a env: Rename spdk_malloc/zmalloc/realloc/free to spdk_dma_(func)
- rename spdk_malloc_socket to spdk_dma_malloc_socket
  - rename spdk_malloc to spdk_dma_malloc
  - rename spdk_zmalloc to spdk_dma_zmalloc
  - rename spdk_realloc to spdk_dma_realloc
  - rename spdk_free to spdk_dma_free

Change-Id: I52a11b7a4243281f9c56f503e826fd7c4a1fd883
Signed-off-by: John Meneghini <johnm@netapp.com>
Reviewed-on: https://review.gerrithub.io/362604
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2017-05-31 15:30:27 -04:00
Ben Walker
b961d9cc12 include: Move the remainder of the code base to stdinc.h
Change-Id: I6a142feeaad3117bd3c75e7c5cb7231a1cfa78ae
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-05-08 13:20:36 -07:00
Ziye Yang
e48e569d83 nvmf/rdma: Replace the queue empty check
Replace with it with check the returned req
via spdk_unlikely macro

Change-Id: I1202b3955af9a68496d8ced7cf66c20cf26f7fff
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-05-08 09:40:15 -07:00
Ben Walker
1f38243914 nvmf/rdma: Combine RDMA WRITE and SEND into single post
When data needs to be transferred from the controller
to the host, do a single ibv_post_send containing
both the data and the completion.

Change-Id: I072c545b31593e0e324c97ed700b42c6a4c358e1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-17 14:42:06 -07:00
Ben Walker
6696fce7ae nvmf/rdma: Manually inline nvmf_post_rdma_send
This call had been reduced to a simple wrapper
around the ibv call. Delete it.

Change-Id: I42926d123db262617119a9cff77bc0d0eb1e8f31
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-17 14:42:06 -07:00
Ben Walker
7488116a78 nvmf/rdma: Manually inline nvmf_rdma_post_[read|write]
These functions were only called from one place and
their functionality has been reduced to a wrapper
around the underlying ibv call. Remove them.

Change-Id: I65182012dbe6393b9d57f4191fd327bcd025a6c8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-17 14:42:06 -07:00
Ben Walker
a4a3b5e7fd nvmf/rdma: Move all SGL setup to the prep_data step
This keeps all SGL handling in the prep_data function.

Change-Id: I9bfeed3748c1b329288350b85aa87bd604cfce4e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-17 14:42:06 -07:00
Ben Walker
e0bdfb14ee nvmf/rdma: Manually inline nvmf_rdma_post_recv
Now that all of the SGL mappings are static,
this function just called ibv_post_recv. Delete
the function and call ibv_post_recv directly.

Change-Id: I45216170a157709249b08c4cb0ebdb1adb906049
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-17 14:42:06 -07:00
Ben Walker
cc294653ca nvmf/rdma: Send data and completion at the same time
For an NVMe read, send the completion immediately
following the RDMA WRITE, without waiting for
the acknowledgement. RDMA is strictly ordered,
so the WRITE will arrive before the completion.

Change-Id: I7e4e01d7a02c2130b655ef90f5fdaec992d9361a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-16 16:44:45 -07:00
Ben Walker
d0fb728ad2 nvmf/rdma: Don't use in-capsule buffers for READ/WRITE
Except for a CONNECT capsule, always use the central data
pool for RDMA READ/WRITE operations. The in-capsule
data buffer is associated with the receive operation
while the pool data buffers are associated with the
completion, and using the in-capsule data buffer
causes a lifetime mismatch.

Change-Id: Ieb45e521d78daa7c706078a3dd5c5a146f8dc1d6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-16 16:44:45 -07:00
Ben Walker
23f273c0b1 nvmf/rdma: Revert 8fefa7e
After commit b654e9b, this is no longer required.

Change-Id: I0cf1a7059d7fba0303aca5ad5a15afe3890b4172
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-16 16:44:45 -07:00
Ben Walker
1db3a03712 nvmf/rdma: Do not assume acks are ordered.
The RDMA protocol this module uses is strictly ordered,
which means messages are delivered in exactly the order
they are sent. However, we have detected a number of
cases where the acknowledgements for those messages
arrive out of order. This patch attempts to handle
that case.

Separate the data required to post a recv from the
data required to send a response. If a recv arrives
when no response object is available, queue the
recv.

Change-Id: I2d6f2f8636b820d0c746505e5a5e3d3442ce5ba4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-16 16:44:45 -07:00
GangCao
89d10dd80e nvmf: add support of default subsystem identifier
Change-Id: I607e800fea00c21dfd28c8fee0a981ae59e46a5c
Signed-off-by: GangCao <gang.cao@intel.com>
2017-03-14 15:33:25 -07:00
Cunyin Chang
39b386a5c5 nvmf: Add interface for transport layer to check if one connection is idle.
Change-Id: I094d5adb85af8279c9823528f40190fa28801b7b
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2017-03-14 13:29:32 -07:00
Daniel Verkamp
50947d5557 nvmf/rdma: use SPDK_COUNTOF to count SGL elements
This clarifies the relation between the values assigned to sg_list and
num_sge (no functional change).

Change-Id: I8e81d47dd97a033b17cd3b813b06e4887127146c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-03-14 09:53:53 -07:00
Ben Walker
8fefa7e9ee nvmf/rdma: Match queue depth math to Linux kernel initiator
Change-Id: Iffe8c35ae76f5541aa95480f1aaaf654fcdfeffd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-13 15:50:05 -07:00
Ben Walker
75990199ae nvmf: Remove sge log messages. The mappings are static now.
The mappings are all static, so it isn't interesting
to print them out on each I/O.

Change-Id: I85301b4518d4523a7c031f6ca9ff678d91428504
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-13 15:11:13 -07:00
Ziye Yang
5c36cd7481 nvmf/rdma: Use separate ibv_send_wr for RDMA READ/WRITE
This allows pipelining of READ/WRITE with completion.

Change-Id: Ib3ab5bffb8e3e5de8cbae7a3b2fff7d9f6646d2d
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-03-13 15:11:13 -07:00
Ziye Yang
0239003a72 nvmf/rdma: Initialize SGL for recv work request just once
Change-Id: I22b76cafe1a496f40cf7661daee3361ba1dee324
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-03-13 15:11:13 -07:00
Ziye Yang
4e74233835 nvmf/rdma: Separate cmd wr from rsp wr
This allows static initialization of the scatter
gather list as well as future optimizations
around pipelining commands with data.

Change-Id: I8af8f3e3425610bc720677c9bc84f163cfb6278a
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-03-13 15:11:13 -07:00
Ben Walker
b4ed77ef0d nvmf: Remove workaround for kernel initiator queue depth
The first version of the Linux kernel NVMe-oF initiator had
a bug when reporting queue size where it was off by 1. We
had a workaround to deal with this. Now that the kernel
has been fixed, remove the workaround.

Change-Id: I0ad4a5c6db68cfa9683ab93e6f5210772c713b55
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-13 14:33:42 -07:00
Ziye Yang
d706fa417f nvmf,rdma: Remove spdk_nvmf_rdma_request_ack_completion
The sq_head handling is already done in
spdk_nvmf_rdma_request_send_completion, so do not need to
do again.

Change-Id: I527ff8adfcbdf43ac79794cb5c7777c0e8ef6973
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-03-09 15:04:14 -07:00
Daniel Verkamp
7f448daaa3 nvmf: remove SPDK_NVMF_REQUEST_EXEC_STATUS_RELEASE
It is no longer used now that AER handling holds the request until it is
triggerred.

Change-Id: I71a75e86f82bc06f677cf26defa701e60b9aa1bd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-03-03 10:22:11 -07:00
Daniel Verkamp
effea17dad nvmf: make invalid rdma_req case into an assert
The wr_id should never be NULL - it will always correspond to a request
we previously posted.  Convert the check to an assert() so we notice if
this ever happens (which would indicate a programming error somewhere
else).

While we're here, add a more robust check to make sure the request is
actually in the correct array of requests for the connection being
polled (also in an assert, since this should never fail in normal
execution).

Change-Id: I855763d7d827fb8cf00a775c7bc2ccb579db8d0f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-02-28 10:52:08 -07:00
Ziye Yang
97e15fedb8 nvmf,rdma: Solve nvmf host reconnect issue.
Kernel nvmf host always tries to connect nvmf target
when we does not issue nvme disconnect command. Thus,
we face rdma_create_qp issue, the reason is that we call
rdma_listen too early, and the event retrieved from
rdma_cm_get_event is too late.

And this patch solves this issue.

Change-Id: I153a8aea7420a86a236301dad9bd54af97f60865
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-02-28 00:16:17 -07:00
liu-wenhua
0ccb72c6da nvmf: call spdk_nvmf_request_complete on error (#123)
Calling spdk_nvmf_request_complete to complete spdk_nvmf_request
causes some fields in completion queue entry not set correctly.
Calling spdk_nvmf_request_complete fixes the problem.
2017-02-24 16:12:40 -07:00
liu-wenhua
84501695fd nvmf: Data transfer cannot be simply determined by NVMe opcode (#120)
Whether a nvme command having data transfer cannot be completely
determined by command opcode. For set features command, some features
don't require data transfer.
Change spdk_nvmf_request_prep_data to fix this issue.
2017-02-22 14:54:30 -07:00
Ziye Yang
849d140b4d nvmf,rdma: remove unnecessary memset
Reason: the 4 fields of struct ibv_recv_wr is already
set in the following 4 lines.

Change-Id: I97437ee2e4c6e944154813bb48b1740b182220df
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-02-21 11:17:01 -07:00
Daniel Verkamp
84d904841f util: move common helper functions to util.h
These were repeated a few different places, so pull them into a common
header file.

Change-Id: Id807fa2cfec0de2e0363aeb081510fb801781985
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-02-10 09:38:52 -07:00
GangCao
fcdb601ea5 nvmf: continue rdma_poll even meets error
Change-Id: I82fc3e939cc82a696b4c0b186bbc1d6cd82b8b54
Signed-off-by: GangCao <gang.cao@intel.com>
2017-02-08 10:09:02 -07:00
Ziye Yang
38980dedfa nvmf,tgt: fix issue while shutting down nvmf tgt with CTRL + C
The phenoemon is that we can not shutdown the nvmf tgt.
The solution is that we need to adjust the shutting down orders of
nvmf tgt subsystem and rdma trasport layer.

Change-Id: Ie39657370b1574960e0ee7cf604cc5872db0bed3
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-02-07 17:43:45 -07:00
Ziye Yang
6f8a505d23 nvmf: cleanup spdk_nvmf_rdma_close_conn
No need to return

Change-Id: I902162dfa8bc21cc160f690bb20085d56bc29a13
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-02-07 15:10:03 -07:00
Ziye Yang
4440cd8d28 nvmf: Solve subsystem add/delete issue
When we do frequent same subsystem add/delete,
we will face the adding issue. For example,
1  Add subsystem A
2  Delete subsystem A
3  Add subsystem A  (Fail in this step).
The reason is that we did not correctly free
the listener resources of subsystems, and this patch
can solve this issue.

Change-Id: I6765a306a3f10c9a0f38c95dbba12e2a4073e705
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-01-30 11:14:38 -07:00
Ben Walker
765173a7ca nvmf: Make RDMA private data required.
This is not actually optional - it contains required
information for setting up the connection.

Change-Id: I21136de12794a0f4f5c14c5d3e2e3f2306c5c102
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-01-20 10:02:32 -07:00
Ben Walker
b2a86421f4 nvme/rdma: Fix specification of R/W queue depth
The meaning of the parameters was swapped.

Change-Id: I3f4938c446fd06afd1852bcf08e779b7b988855d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-15 16:35:00 -07:00
Daniel Verkamp
d27b24c94b log: split internal TRACELOG macro into new header
The SPDK_TRACELOG macro depends on a CONFIG setting (DEBUG), so it
should not be part of the public API.

Create a new include/spdk_internal directory for headers that should
only be used within SPDK, not exported for public use.

Change-Id: I39b90ce57da3270e735ba32210c4b3a3468c460b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-11-16 13:33:51 -07:00
Changpeng Liu
f903336eed nvmf: fix the wrong cleanup logic in the RDMA initialization process
Print the error information when the kernel RNIC driver did not load
properly, and fix the cleanup logic for the exceptional exit.

Change-Id: I97a45e73d830280b994818f3defc491bc2b6b020
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-11-02 10:30:39 -07:00
Daniel Verkamp
1290f02f78 nvmf/rdma: add event_channel NULL check in fini
If the RDMA transport failed to initialize, g_rdma.event_channel may be
NULL.

Change-Id: I4510ee5893389f244f0fbaa1cd4a182868939b25
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-10-31 10:15:36 -07:00
Ben Walker
20dc7f7d19 nvmf: For iWARP, register buffers with IBV_ACCESS_REMOTE_WRITE
For iWARP devices, buffers that are intended to be the
target of an RDMA read initiated by the target must additionally
have IBV_ACCESS_REMOTE_WRITE permission. This is because iWARP's
RDMA read path essentially requests the remote side to do
an RDMA write.

This is unfortunate because there is no way to differentiate between
memory that the remote side can do an RDMA write to and memory
that will only be the target of RDMA reads initiated by the
target. There is nothing we can do about this serious deficiency in
the specification, however, so we have to live with it.

Change-Id: I3d2f2814ce0cb1df4e5347296ef371db4d16be21
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-31 08:58:40 -07:00
Ben Walker
f64690d49a nvmf: Handle NICs that don't report RDMA initiator_depth
I believe this is required for NICs to report, but handle
the case where it isn't reported.

Change-Id: I38d10c3590d1df8bb902ab312af0f9e01b9e5032
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-21 09:09:06 -07:00
Ben Walker
a1a085cf6b nvmf: Transports now allocate sessions
This makes it consistent with the way connections and
requests work.

Change-Id: Ifb97499ba72f7dfd02ac54ba1b622726d266262c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-21 09:09:05 -07:00
Ben Walker
8b487155fa nvmf: Fail when sessions span RNICs
The shared memory pool for a session is associated with
a particular RNIC via the protection domain. New connections
attempting to join a session that came in on a different RNIC
can't use that memory, so must be rejected.

Change-Id: Ibd79fe90566a231f76b7472e5e9b484c3e528454
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-21 09:09:05 -07:00
Ben Walker
349295caf2 nvmf: Rearrange rdma file to match callback order
Rearrange the functions in rdma.c to match the order
of the function pointers in the transport. No other
code changes.

Change-Id: I9dbc68912ecd5dfdf53f20b4807d4116933a3c3a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-21 09:09:05 -07:00
Ben Walker
b0c1149953 nvmf: Replace rdma_reg_* with ibv_reg_mr
Use the lower level registration functions. The RDMA-CM
examples use the ibv_* versions, so who knows if the
rdma_reg_* wrappers are even well tested.

Change-Id: I8e8250ab09a1401e636aebe2fc04a60806f7a827
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-21 09:09:05 -07:00
Ziye Yang
379ebca018 nvmf,rdma: only call spdk_nvmf_rdma_conn_destroy to free rdma_conn
Previously, we mixed use free and spdk_nvmf_rdma_conn_destroy to
free allocated spdk_nvmf_rdma_conn structure, which sounds not
exactly free all the resources.

Change-Id: I2917b442c34d63ba5c014add58f429ae4b831595
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2016-10-20 16:24:38 -07:00
Ben Walker
769468fc52 nvmf: Allocate RDMA wr on heap instead of stack.
The RDMA API doesn't say whether the wr is copied, so be
safe and allocate it on the heap.

Change-Id: I091af50aa031e1861333f19d864eb52335d6b756
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-20 16:22:58 -07:00
Ben Walker
cfafcc3edd nvmf: Init transport before adding listeners
Change-Id: I1fbaca75c6b95127ff2b6ac7c0ea0f0a918b9405
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-20 16:22:58 -07:00
Ben Walker
1e6ffa0394 nvmf: Add a public API header
This changes as little code as possible while still creating
a single public API header. This enables future clean up
of the public API and clarification of the exposed
concepts.

Change-Id: I780e7a5a9afd27acf0276516bd71b896ad301c50
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-14 15:26:31 -07:00
Ben Walker
18a2cc11c1 env: Remove unused DPDK headers.
Remove #includes for all DPDK headers that weren't
necessary.

Change-Id: Ib02522e0f04e64a1c98afceb7508cc0e8d931a9d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-12 09:53:32 -07:00
Ben Walker
0aa2986475 Replace rte_get_tsc calls with spdk_get_ticks
Change-Id: I809b900321433693ff9f2498183ad0dcdbb15030
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-11 13:34:09 -07:00
Ben Walker
2224ff2162 env: Replace rte_malloc with spdk_zmalloc
Use the env library to perform all memory allocations
that previously called DPDK directly.

Change-Id: I6d33e85bde99796e0c85277d6d4880521c34f10d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-11 13:34:09 -07:00
Ben Walker
d4652fd37f nvmf: Share completion queue channels
All completion queues for the same listen address
now share a common completion queue channel.

Change-Id: I42c149fe7e221951e8a3826b1713482c37a265b8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-07 09:15:11 -07:00
Ben Walker
a0a92ff4c2 nvmf: Combine acceptor_init/fini with transport init/fini
These 4 callbacks can be condensed into two callbacks, which
simplifies the API.

Change-Id: I069da00de34b252753cdc8961439e13a75d1cc68
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-10-07 09:14:53 -07:00
Ben Walker
a17ad921e2 Replace RTE_VERIFY with assert
We already require the assert header from the C standard library,
so use that instead of RTE_VERIFY to further isolate DPDK
dependencies.

Change-Id: I4a718af858c88aff6080e33e6c3dd533c077b8f4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-09-28 10:15:56 -07:00
Ziye Yang
b1e3279ae0 nvmf: change nvmf_session to spdk_nvmf_session
Make the structure definition consistent

Change-Id: If147066cee041a2a357bc7efcb1f882f914e5b42
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2016-09-27 09:57:17 -07:00
Ben Walker
f3d90c9165 nvmf: Reap entire batches of completions for RDMA at once
Instead of polling for only 1 completion at at time,
poll for batches of 32.

Change-Id: I5ef99a270489e7b3d2a58cb765915f187775a93e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-09-23 10:08:10 -07:00
Ben Walker
4bf7878521 nvmf: Combine nvmf recv and send cqs
ibv_poll_cq is actually an expensive call to make, so take
steps to begin to minimize the number of times it is called.

Change-Id: I6fc64979604220eb8cacd612b46e3a3b1bca0924
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-09-22 09:11:38 -07:00
Cunyin Chang
b3ca958328 nvmf: Make the function spdk_nvmf_rdma_acceptor_init() reentrant.
Change-Id: Ie715affe7e451fc517c5f57f1597d0bc70f69947
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2016-09-13 09:49:25 -07:00
Roland Dreier
4c0aab3f2a nvmf: Correct log messages when polling recv CQ
Fix copy-and-paste errors - when polling the recv CQ, we should print
"Recv" instead of "Send" in log messages.

Signed-off-by: Roland Dreier <roland@purestorage.com>
2016-09-13 08:26:31 -07:00
Ben Walker
af8b5732d3 nvmf: Optimize nvmf_ibv_send_wr_init
This function always shows up as one of the hottest functions when
profiling. I believe it is the memset that is expensive, so instead
use default initialization when the wr is declared on the stack
and just set the members that need to be updated in the function.
Also make the function inline for good measure.

Change-Id: I29e24cdd375311fa033b5a6df772ff4f73e35302
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-09-12 15:33:43 -07:00
Daniel Verkamp
bc0867dcaf nvmf: move subsytem poller to nvmf_tgt app
The application is now entirely responsible for scheduling subsystem
pollers and sending events between threads.

Change-Id: I88da1f53b5e8852c7c4acd6f0a7a1e2219fbed41
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-09-12 10:10:27 -07:00
Ziye Yang
1c5093358e nvmf,rdma: update spdk_nvmf_rdma_conn_destroy
Merge two if judge statements into one.

Change-Id: I72f7e4a936bc0b628c06a9ebb7c816150805f495
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2016-09-06 13:19:02 -07:00
Ziye Yang
1675e153fb nvmf: Update get_rdma_conn and get_rdma_req
The "+" is not correct, should be "-". Currently,
the issue doest not happen since the offset is 0,
then both + and - is OK. But if we adjust the location
of spdk_nvmf_conn or spdk_nvmf_request, we can find
this bug.

Change-Id: Ib358dc729da901a69442d0402a6089989f49b05c
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2016-09-06 13:18:26 -07:00
Daniel Verkamp
39cda18718 nvmf: rename trsvc -> trsvcid for spec consistency
Change-Id: I5f73ef20ba231a7b1721562964dfd50de9b2c735
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-23 09:03:28 -07:00
Daniel Verkamp
37402f490e nvmf: pad discovery TRSVCID and TRADDR with spaces
These are specified as "ASCII string", which means they should be
left-aligned and padded with spaces, according to the NVMe base
specification.

Change-Id: I25babe0ca417c2e16137b0bfc41fc7834277114e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-22 12:58:59 -07:00
Changpeng Liu
2641c31af2 nvmf: Listen for incoming connections only on addresses specified
Currently the NVMf target listens for new connections on any address.
Instead, listen only on the addresses specified by the user.

Change-Id: Idb6d37c422e442fc70a8673bd3fcfb9c27b57828
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-08-19 15:34:01 -07:00
Daniel Verkamp
4c6e4d4963 nvmf: move acceptor poller into nvmf_tgt app
The NVMe over Fabrics target library now exposes a simple function call
that polls the acceptor once, and the application handles registration
of the poller.

Also rename the transport function pointers related to the acceptor so
they better reflect their purpose.

Change-Id: I5fa0d516586bf17e73afeb88ff3c2d5b0d46794d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-17 10:01:37 -07:00
Daniel Verkamp
47dde07521 nvmf: allow target to start without RDMA devices
This will become more important when other transports are added.

For now, it is also useful to be able to start nvmf_tgt on systems
without RDMA hardware.

Change-Id: I6b9002cc7711f928c4e6b73adcd9b677349ebdd6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-17 10:01:37 -07:00
Changpeng Liu
8a23223e1b nvmf: Allow users to configure which lcore each subsystem runs on
Users can specify the core for each subsystem and the acceptor listen routine
to run on different cores for performance consideration.

Change-Id: I4bd1a96f39194c870863b4b778e6ea7cf8fc1a2d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
2016-08-16 09:20:42 -07:00
Daniel Verkamp
5d8c94536a event: hide struct spdk_poller internals
This leaves more flexibility for future changes to the poller
representation without requiring API changes (after this one).

It also prevents the user from accidentally using poller fields in a
non-thread-safe way, since they can't be accessed directly anymore.

Change-Id: I7677d5b93668665d29ae39c5e0ba74333ad3f878
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-12 10:42:12 -07:00
Daniel Verkamp
184984603f nvmf: use Connect command SQSIZE to manage SQHD
The NVMe submission queue head wraparound point can be determined in the
generic NVMe over Fabrics layer; it should not be using the RDMA
connection queue depth.

Change-Id: I9da8f09e4f057f8fdc1ff4c6cc5f48cea7123e11
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-08 08:45:38 -07:00
Daniel Verkamp
be0dae64be nvmf: simplify spdk_nvmf_rdma_conn_create()
Use the max_queue_depth parameter rather than rdma_conn->max_queue_depth
so that we can start to eliminate rdma_conn->max_queue_depth.

Change-Id: I1670c634e6d12aa004fb5a10338b7624850fbc4a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-08 08:45:38 -07:00
Daniel Verkamp
60a91f33cc nvmf: add messages for session_init failure cases
Change-Id: I079df9a15ac023d160b842676aab162393f6da80
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-05 08:51:02 -07:00
Daniel Verkamp
5f6f676cbf nvmf: fix rdma_reg_msgs size argument
The large buffer pool allocation was using the per-connection queue
depth, whereas the RDMA memory region registration was using the global
RDMA max queue depth.  These sizes need to match, so use the global RDMA
max queue depth for both calls.

Change-Id: Iae161b719e09e19ca3e81df6593b68a4a2e86614
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-05 08:51:02 -07:00
Daniel Verkamp
e7a6744a88 nvmf: switch RDMA acceptor to spdk_poller
Use the new timer-based poller functionality to replace rte_timer.

Change-Id: Ic40653306cc73b40139fe18e06bab29b35721a43
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-08-03 09:00:54 -07:00
Ben Walker
f1a584a9f7 nvmf: Use a shared memory pool for large data buffers.
Change-Id: Iab66335cee2a1e6c1774edd34978735be6763ce1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-29 15:36:10 -07:00
Ben Walker
caf8860900 nvmf: Allow higher queue depths
Allow higher queue depths by allowing many more send/recv
operations than read/write.

Change-Id: I66c424a6463e5e09be6d5463667241ce9271404b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-28 13:55:43 -07:00
Ben Walker
eee64c69f7 nvmf: Re-post the capsule immediately upon sending a completion
The target can only provide updates to sq_head inside
of completions. Therefore, we must update sq_head prior
to sending the completion or we'll incorrectly get into
queue full scenarios.

Change-Id: If2925d39570bbc247801219f352e690d33132a2d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-28 10:27:31 -07:00
Ben Walker
7e23841d28 nvmf: Separate the send and recv completion queues.
This allows the target to poll for internal completions
at higher priority.

Change-Id: I895c33a594a7d7c0545aa3a8405a296be3c106fb
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-28 09:00:48 -07:00
Ben Walker
04beb5661e nvmf: Send completions after RDMA write has completed.
This ensures that the data buffers are not in use
when we go to send the completion.

Change-Id: I30467b3e3964001150f81b21e5b695dcd0974b0c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-28 09:00:48 -07:00
Ben Walker
d7b8da3b81 nvmf: Add a transport specific session
This is useful for holding session-wide buffer pools.

Change-Id: I7024da24b210a2205bf1e159d5935e0093b81120
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-28 09:00:47 -07:00
Ben Walker
52a4a388fb nvmf: Make RDMA WRITE operations signalled
Change-Id: Iad9e216144d88c899b52220ae9b32c24e3cbb252
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-28 09:00:47 -07:00
Ben Walker
8a701c3f8d nvmf: Use the inline SGL for keyed SGLs if the size is small enough
For small SGLs, even if they are keyed and not inline, use the
buffer we allocated for inline data.

Change-Id: I5051c43aabacb20a4247b2feaf2af801dba5f5a9
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-26 15:24:01 -07:00
Ben Walker
ca0c13387a nvmf: Calculate queue_depth separately from rw_depth
Read/Write depth is much lower than Send/Recv depth.
Calculate them separately to prepare for supporting
a larger number of receives than read/writes.

Currently, the target still only exposes a queue depth
equal to the read/write depth.

Change-Id: I08a7434d4ace8d696ae7e1eee241047004de7cc5
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-26 14:24:41 -07:00
Ben Walker
756df04485 nvmf: Remove g_nvmf_tgt global usage from transport layer
Change-Id: Id788312f597abf6ea937beb7d1d1bd5a168ae0f0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-26 14:24:39 -07:00
Ben Walker
296add8bb1 nvmf: Add config options for inline and max I/O size
These don't actually work quite yet, but pipe the
configuration file data through to where it will
be needed.

Change-Id: I95512d718d45b936fa85c03c0b80689ce3c866bc
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-26 14:23:25 -07:00
Ben Walker
3d52e57cd0 nvmf: Allocate rdma reqs as a single contiguous buffer
For each connection, allocate a single buffer each
of requests, inline data buffers, commands, and
completions.

Change-Id: Ie235a3c0c37a3242831311fa595c8135813ae49e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-26 09:40:25 -07:00
Ben Walker
b43945830f nvmf: Simplify error handling in rdma conn create
Change-Id: I5380c7785a066f4414aaa1a27a467089d7b50031
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-26 09:40:25 -07:00
Ben Walker
5ade1c40f4 nvmf: Standardize names of rdma conn create/destroy
Change-Id: Id1b3328deceeeaa7da8ee2bda992a006286886b0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-07-26 09:40:25 -07:00