Commit Graph

27 Commits

Author SHA1 Message Date
Gregory Shapiro
14032a984c NVMF: Add model number as parameter to construct_nvmf_subsystem (-d option).
Change-Id: Ia1a458a0ac1c5a17d2955a3f31c6dfe77538eb17
Signed-off-by: Gregory Shapiro <gregory.shapiro@kaminario.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/438562
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-04-23 16:51:16 +00:00
Darek Stojaczyk
728af01741 event/nvmf: allow running without [Transport] in the legacy cfg file
While useful in NVMf target, this is really annoying
in the consolidated spdk target, where NVMe-oF doesn't
even have to be used. OCF tests currently use iscsi_tgt
just because spdk_tgt requires the additional [Transport]
section in the cfg file. Let's remove that requirement.

Change-Id: I418b47d62dcc06b9513f9f0496dc1e39b9d5a554
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450056
Reviewed-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-04-04 03:58:22 +00:00
Evgeniy Kochetov
ed0b611fc5 nvmf/rdma: Add shared receive queue support
This is a new feature for NVMEoF RDMA target, that is intended to save
resource allocation (by sharing them) and utilize the
locality (completions and memory) to get the best performance with
Shared Receive Queues (SRQs). We'll create a SRQ per core (poll
group), per device and associate each created QP/CQ with an
appropriate SRQ.

Our testing environment has 2 hosts.
Host 1:
  CPU: Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz dual socket (8 cores total)
  Network: ConnectX-5, ConnectX-5 VPI , 100GbE, single-port QSFP28, PCIe3.0 x16
  Disk: Intel Optane SSD 900P Series
  OS: Fedora 27 x86_64
Host 2:
  CPU: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz dual-socket (24 cores total)
  Network: ConnectX-4 VPI , 100GbE, dual-port QSFP28
  Disk: Intel Optane SSD 900P Series
  OS : CentOS 7.5.1804 x86_64
Hosts are connected via Spectrum switch.
Host 1 is running SPDK NVMeoF target.
Host 2 is used as initiator running fio with SPDK plugin.

Configuration:
- SPDK NVMeoF target: cpu mask 0x0F (4 cores), max queue depth 128,
  max SRQ depth 1024, max QPs per controller 1024
- Single NVMf subsystem with single namespace backed by physical SSD disk
- fio with SPDK plugin: randread pattern, 1-256 jobs, block size 4k,
  IO depth 16, cpu_mask 0xFFF0, IO rate 10k, rate process “poisson”

Here is a full fio command line:
fio  --name=Job --stats=1 --group_reporting=1 --idle-prof=percpu \
--loops=1 --numjobs=1 --thread=1 --time_based=1 --runtime=30s \
--ramp_time=5s --bs=4k --size=4G --iodepth=16 --readwrite=randread \
--rwmixread=75 --randrepeat=1 --ioengine=spdk --direct=1 \
--gtod_reduce=0 --cpumask=0xFFF0 --rate_iops=10k \
--rate_process=poisson \
--filename='trtype=RDMA adrfam=IPv4 traddr=1.1.79.1 trsvcid=4420 ns=1'

SPDK allocates the following entities for every work request in
receive queue (shared or not): reqs (1024 bytes), recvs (96 bytes),
cmds (64 bytes), cpls (16 bytes), in_capsule_buffer. All except the
last one are fixed size. In capsule data size is configured to 4096.
Memory consumption calculation (target):
- Multiple SRQ: core_num * ib_devs_num * SRQ_depth * (1200 +
  in_capsule_data_size)
- Multiple RQ: queue_num * RQ_depth * (1200 + in_capsule_data_size)
We ignore admin queues in calculations for simplicity.

Cases:
1. Multiple SRQ with 1024 entries:
   - Mem = 4 * 1 * 1024 * (1200 + 4096) = 20.7 MiB
     (Constant number – does not depend on initiators number)
2. RQ with 128 entries for 64 initiators:
   - Mem = 64 * 128 * (1200 + 4096) = 41.4 MiB

Results:
FIO_JOBS   kIOPS     Bandwidth,MiB/s  AvgLatency,us  MaxResidentSize,kiB
       RQ       SRQ     RQ      SRQ    RQ       SRQ      RQ       SRQ
1      8.623    8.623   33.7    33.7   13.89    14.03    144376   155624
2      17.3     17.3    67.4    67.4   14.03    14.1     145776   155700
4      34.5     34.5    135     135    14.15    14.23    146540   156184
8      69.1     69.1    270     270    14.64    14.49    148116   156960
16     138      138     540     540    14.84    15.38    151216   158668
32     276      276     1079    1079   16.5     16.61    157560   161936
64     513      502     2005    1960   1673     1612     170408   168440
128    535      526     2092    2054   3329     3344     195796   181524
256    571      571     2232    2233   6854     6873     246484   207856

We can see the benefit in memory consumption.

Change-Id: I40c70f6ccbad7754918bcc6cb397e955b09d1033
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/428458
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-03-15 19:19:17 +00:00
Seth Howell
3709db7383 nvmf: add buf_cache_size to conf and rpc functions
Change-Id: I11514dc5981e9750d693202ff86af22b2064ae72
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439434
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-01-18 16:57:37 +00:00
Ziye Yang
58f1624497 nvmf: add the transport shared buffer num configuration option.
Previously, we allocate the buffer size according
to the MaxQueueDepth info, however this is not exactly
a good way for customers to configure, we should provided
a shared buffer number configuration for the transport.

Change-Id: Ic6ff83076a65e77ec7376688ffb3737fd899057c
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/437450
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-12-20 19:55:57 +00:00
Ziye Yang
6134d778d4 conf: Transport should be explictly configured in conf file
According to the currrent logic, it should be explictly defined
in the configuration file if the NVMe-oF target is started by
configuration file without using rpc calls. If no transport is
defined, we do not need to parse the following subsystems
but should terminate the NVMe-oF target .

Change-Id: I2e2db8406a30a9bf7e54d38f8d08a8d92ef158c9
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/432376
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2018-11-15 04:49:08 +00:00
Seth Howell
962ba4e89a nvmf: remove tgt_opts from nvmf_tgt
This option is deprecated. Also, rename the rpc and configuration
options for setting the opts to reflect that they now only set the max
number of subsystems

Change-Id: Iaabcbf33dd0a0dc489d81233fda74e9e7f3e0d2e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/430161
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-11-08 23:08:26 +00:00
John Barnard
183d81d0c6 nvmf: Move target opts to transport opts (part 2)
- Add independent functions to create transport with specific opts
  and add to target while maintaining backward compatibility with
  current apps and rpc configuration that still use the add listener
  method to create a transport.
- Add new rpc function to create transport and add to target.
  + Update json reporting to include new rpc function.
  + Update python scripts to support new rpc function.
  + New nvmf test script (cr_trprt.sh) to test new rpc function.

Change-Id: I12d0a42e34c9edff757755f18a78b722d5e1523e
Signed-off-by: John Barnard <john.barnard@broadcom.com>
Reviewed-on: https://review.gerrithub.io/423590
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-09-17 20:42:16 +00:00
Avinash M N
1f3f160581 nvmf: Add IP based load balancing
This patch adds support for IP based load balancing. All the rdma connections
coming from a particular IP will be assigned the same core on target. The load
balancing policy - roundrobin/hostip can be configured through rpc during startup only.
Roundrobin is the default policy.

Change-Id: I7bb216cb4cb2639fa401e23090143c4cf1d7bb5a
Signed-off-by: Avinash M N <avinash.m.n@wdc.com>
Reviewed-on: https://review.gerrithub.io/422190
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
2018-09-11 23:55:25 +00:00
Daniel Verkamp
194d6af65d nvmf: allow setting namespace UUID in config file
Change-Id: I768129eec3b08c43ef27c3ba8c237d77ebd49178
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/414681
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2018-06-12 20:24:00 +00:00
Shuhei Matsumoto
46a3d50baa subsystem/nvmf: Add set_nvmf_target_options/config RPCs
Two new RPCs allocate and initialize options and conf dynamically.
Initialization of NVMf subsystem skips initialization of them
if they are already allocated.

To use and test them, add python script too.

Change-Id: I3be83c33edf9960cb4340d52abb47b7331a6fb92
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/406932
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-06-12 17:41:29 +00:00
Srikanth kaligotla
8580daa1ac nvmf: SGL support for NVMF RDMA Driver.
Change-Id: I447754c69de432b5a65dc8c1d9ae690926e88c51
Signed-off-by: John Meneghini <johnm@netapp.com>
Signed-off-by: Srikanth kaligotla <kalis@netapp.com>
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/410302
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
2018-06-04 17:15:49 +00:00
Ben Walker
1d94536417 nvmf: Bound the maximum number of namespaces that can be configured
Make the bound enormous, but bound it so that we don't run out
of memory.

Change-Id: I014b3df3be92ae8bd85fea95f514225adc3eefe2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/408247
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
2018-04-19 15:48:36 -04:00
Ben Walker
4d2e54a635 nvmf: Handle addresses that don't contain a port
Change-Id: I0c0c455ccf6f00ff95e7f1d2f0391fc44c6b4b2a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/408219
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-04-18 15:35:12 -04:00
Daniel Verkamp
be0eef0a0d nvmf: check for missing NQN in conf parsing
Avoid dereferencing a NULL pointer if NQN wasn't specified in the
configuration file.

Change-Id: Ie43111fce2c5fcf06bad00af10413a665dfbc45f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/407484
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2018-04-12 17:56:21 -04:00
Ben Walker
2b79861deb nvmf: Make spdk_nvmf_tgt_listen asynchronous
This was internally asynchronous already, but make it
explicitly asynchronous so other code can properly
wait on the operation to complete.

This fixes an intermittent CI crash.

Change-Id: I81c9b19673566047dcffa94796236ca9fd7fa7d0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/406226
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-04-05 12:54:31 -04:00
Daniel Verkamp
9770ee7817 nvmf: reduce log level of informational messages
Change-Id: I595f0ec871174557a35811fe3b102c33a5a60ab0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/405931
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-04-05 12:06:45 -04:00
suman chakraborty
5e98dfd126 nvmf: add capability to add namespaces dynamically during active connection
1) The user should provide MaxNamespaces during the construction of the subsystem
2) The namespace which is added should have nsid less than or equal to MaxNamespaces
3) If the user does not provides MaxNamespaces then the exsisting behaviour continues where the nisd can grow dynamically when it is not connected.

Change-Id: I54769d9669575a5f6bf56fe5a262191ac51c474d
Signed-off-by: suman chakraborty <suman.chakraborty@wdc.com>
Reviewed-on: https://review.gerrithub.io/405375
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2018-03-30 19:24:29 -04:00
Daniel Verkamp
bf131d8443 nvmf: remove unused struct spdk_nvmf_probe_ctx
This was left over from direct mode.

Change-Id: I967e59f95cc097da09bb35ea77eb5f7423a4ac8a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403385
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2018-03-14 12:54:15 -04:00
Daniel Verkamp
270df2bfd7 nvmf: move g_tgt.tgt to a single global variable
Eliminate struct nvmf_tgt and the g_tgt global variable.

Change-Id: I6b737c4b85dc7d063ead1df5417d344c0f35a30d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403384
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2018-03-14 12:54:15 -04:00
Daniel Verkamp
0048f79819 nvmf: remove unused #defines in conf.c
Change-Id: I2ef159d41d375c907fa529ff24cb0bd9d7654ea4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403382
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2018-03-14 12:54:15 -04:00
Daniel Verkamp
bbbd0d643a nvmf: remove nvmf_tgt_create_subsystem()
After the previous cleanup patch, this was just calling
spdk_nvmf_subsystem_create() without any other functionality, so call
spdk_nvmf_subsystem_create() directly from its call sites and remove it.

Change-Id: I5dc1a0a00025a120d71270c5a5c94e8d74c2a3ab
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403381
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2018-03-14 12:54:15 -04:00
Daniel Verkamp
3f987aa84d nvmf: remove spdk_nvmf_construct_subsystem()
After the previous cleanup patches, this function was just calling
nvmf_tgt_create_subsystem() with no other functionality.

Change-Id: I1b20887cade7496093e0b69778df30f818eeb764
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403379
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2018-03-14 12:54:15 -04:00
Daniel Verkamp
1d48012929 nvmf: set serial number after creating subsystem
Remove the serial number from the parameters of
spdk_nvmf_construct_subsystem().

Change-Id: Ib6d394cc98fddf85fc878fe0a8c8aa0fdcddf7f4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403378
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2018-03-14 12:54:15 -04:00
Daniel Verkamp
72d72f503c nvmf: add hosts after creating subsystem
Remove the array of allowed hosts from the parameters of
spdk_nvmf_construct_subsystem() and add the hosts individually now that
we have the capability to add them after subsystem creation.

Change-Id: I8b05e44d6a254f8d5848975e4bcfb7847e2cf4f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403377
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2018-03-13 14:11:40 -04:00
Daniel Verkamp
333e9299b1 nvmf: add listen addresses after creating subsystem
Remove the array of listen addresses from the parameters of
spdk_nvmf_construct_subsystem() and add the listen addresses
individually now that we have the capability to add them after subsystem
creation.

Change-Id: If9b9e5f5ce2e6af2bddbda7c3926170f2296abdd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403376
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2018-03-13 14:11:40 -04:00
Daniel Verkamp
2491021f64 nvmf: convert nvmf_tgt into an event subsystem
This is required as part of our effort to build a single target app that
can run e.g. NVMe-oF and iSCSI targets in the same process.

Change-Id: Ic8426df08899070d709e3675a57f47f64c23c5eb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/403218
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2018-03-09 09:56:44 -05:00