Simplify and remove a direct call to a DPDK function.
Change-Id: I08eaf86a48df67e3248eeaa764ae924b784d9277
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Store each reactor's per-socket event mempool in the spdk_reactor
structure to avoid calling rte_lcore_to_socket_id() on every iteration,
and make the function definition an internal, inlineable version
that takes the reactor pointer directly.
Change-Id: I841f7d7594308d7c572f5b7f609913c428bd13d7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Minimize the number of times spdk_get_ticks() is called
because it is expensive.
Change-Id: I2f34ca724ec28f42866b76d224dacbe1f31e7a41
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
64k sessions over the lifetime of a single target is something
that really could happen, so handle this case.
Change-Id: Iaed92b9ff6cd078fcd7c1efe88cf0c860c77c4ac
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
For iscsi read/write, expected_data_xfer_len
is 0, dxfer_dir is set to SPDK_SCSI_DIR_NONE.
But we can still have read/write op in SCSI layer.
This patch solves this issue.
Change-Id: I950e163fffb06fefaf8a913d1f6de29c96a52264
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The g_thread_mmio_ctrlr should be not NULL pointer when it enter the
handler function.
Change-Id: I45dba601c672b16e2c6feafd9059bafde0d8f1b4
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
If namespace is formatted with per lba metadata feature and also disable end-to-end protection
feature, host couldn't use per extended-lba metadata area.
Signed-off-by: Zhihao Zhang <thomas.zzh@alibaba-inc.com>
If the user asked for a specific PCI address in spdk_nvme_probe(), we
need to return 1, not 0, for the other PCI addresses that don't match
when enumerating. 0 means to attach the PCI driver, whereas 1 means to
continue enumerating.
With the previous behavior of returning 0, all NVMe devices would be
attached to the DPDK PCI driver, even if the user did not request for
them to be probed, and further calls to spdk_nvme_probe() would not find
any devices.
Change-Id: Ifbbcd7d1abe8ab535b6957855172e66a3e69fbe4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is not actually optional - it contains required
information for setting up the connection.
Change-Id: I21136de12794a0f4f5c14c5d3e2e3f2306c5c102
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This isn't used anywhere yet, but it will be for
NVMe-oF 1.1.
Change-Id: Ieae0688e6ad5b7a44568e5760382b5716b02e6f0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The code doesn't actually use this property of cntlid
for anything yet, but we will need it later.
Change-Id: I5fd514d75b903cc8769e7b9f196a4624e9cf876c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is necessary to process asynchronous events, as well as keep-alive
support for NVMe over Fabrics connections.
Based on a patch by Edward Yang <eyang@us.fujitsu.com>
Change-Id: I3e81f3d5061f75b12b625fa1a06629c6dc3dc61b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
There is only a single device ID for all channels on the SKX
implementation of I/OAT.
Change-Id: I90ee79b1b673a199754f1ca4c9e38e934294e261
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This prevents the need for bdev users and modules to manipulate the
internal bdev_io error.nvme fields.
For now, all non-NVMe error types are treated as a generic device error,
but translation from SCSI to NVMe could be added in the future.
Change-Id: I4e831b26a2f41bf2f405c7576d5019bb898d4d1b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Currently we use the pci functions provided by DPDK,
it identifies the device by class id related
info but not by pci bdf info, so we can add the filering
by pci_addr in pcie_nvme_enum_cb function.
Change-Id: I5942e98853f00fc10fa6aae5c113517653d1b357
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Since nvme_ns_cmd.c now walks the SGL, some of the test code
needs to also be updated to initialize and return correct values
such as ctrlr->flags and sge_length.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I521213695def35d0897aabf57a0638a6c347632e
Convert the number parsing function into a linear sequence with a goto
label for each state, rather than a single loop with a state variable.
This makes the code easier to read and also improves speed (better
branch prediction and smaller inner loops for the common case).
On my test system, jsoncat citylots.json > /dev/null improves from
~1.7s to ~1.2s.
This changes behavior of some number parsing test cases: inputs matching
the number grammar as defined by JSON will be returned even if there is
trailing garbage, consistent with the rest of the parser. For example,
the input 01 will be parsed as a valid number 0 followed by trailing 1.
This only makes any difference when the full input is a single
number value, since if the value was nested in an object or array, the
trailing garbage will not match the expected syntax and the whole parse
will fail with SPDK_JSON_PARSE_INVALID (e.g. [00 will parse the first 0
as a number and then fail on the second 0, since only a comma or right
square bracket would be accepted).
Change-Id: Ifabfaed611219b3e0a06c8677190a28b87e8a13b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This improves output speed significantly, especially if the write
callback is expensive (e.g. issues a syscall or takes a lock).
On my test system, jsoncat citylots.json > /dev/null improves from
~2.8s to ~1.7s.
citylots.json: https://github.com/zemirco/sf-city-lots-json (~181 MiB)
Change-Id: I7d411ce92366712ed87ad5fc6e9b64828541db4d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If a blockdev module calls spdk_bdev_io_complete() within its
submit_request function, and the user's completion callback issues a new
I/O, it is possible to cause infinite recursion, consuming all available
stack space.
To avoid this, track whether a bdev_io is being processed by
submit_request, and if io_complete() is called in this case, defer the
completion via an event.
Change-Id: I6ccdb8ed4ee0d5738e6c9840d35431de52bd5fa2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Preivously, we only supports probe the NVMf target
via discovery info, now we can support to directly
to connect it.
Change-Id: I08ce1d95de6744286357e68b48c97b773b902ac8
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
I do not see any reason to ignore using this channel. If that,
we should give comments in the file, otherwise we need to add it.
Change-Id: I56ad491c67a23831befc8c761ad0a02e721a15a4
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Because of the addition of io_channel support to the bdev layer, there
is no longer a need to re-run a completed I/O through the submission
event pipeline; it can be freed directly.
Change-Id: I2b9163c87293345acf0e85f6d0c1032f30209659
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Include timer-based pollers in the active/idle check that uses
last_action to determine when a reactor last executed an action.
Change-Id: Ib8f1253675b57aeb59206d099c6257f6d07f5acf
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
One microsecond is not really long enough to detect an idle condition
where calling the OS usleep() makes sense. Increase the minimum time
spent spin-waiting on events and pollers from one microsecond to one
millisecond.
Change-Id: I678118e357330f133251f4cfada8ff27e10158a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
When a connection enters full-feature phase and is assigned to an lcore,
we need to increment the counter for the new lcore, not the connection's
existing lcore.
Change-Id: Idced4090b6e8ac35a767fd223fbd81ba824615d3
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Claim the block devices used by iSCSI LUNs and NVMe-oF subsystems so
they can't accidentally be reused.
This will also be used by virtual block devices to allow layering of
bdevs.
Change-Id: I5384923fbf24f13f4ce720a797c5a628053d49f4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
(1) Add nvme_rdma_build_sgl_request function
(2) Merge nvme_rdma_pre/post_copy_mem to nvme_rdma_copy_mem
Change-Id: I86abab821b32b4da0aa9489a6b9f7dc430333159
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Use a plain function pointer + callback context for the bdev I/O
completion callback. This is possible now because each I/O channel will
be polled on the core that submitted the I/O.
Change-Id: I29ee8e4a3430df11c74845adab840395b9bc5010
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
An old prototype SPDK AHCI driver would return
TASK_SET_FULL if all NCQ slots were full on a given
disk. This would kick the SCSI task back to the LUN
to be retried later. Since then, we have pushed
responsibility onto the bdev modules themselves
to handle this kind of queueing/retry logic.
Removing this logic allows us to make some additional
changes that enable tasks to get completed inline without
an extra event callback to handle completion. We also
no longer need to worry about checking if pending tasks
need to be executed in the complete_task() routine, since
the execute() routine will now always exhaust the pending_task
list.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If2dc3ab017e0dbc225c8f627e1f87c5a8e9b1e3e
Now that the hotplug code is isolated in nvme_pcie.c, it can call the
PCIe transport attach function directly.
Change-Id: I2df3b9168473b537cc9b13367e06d3d3b6fa22be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The reactor structures are allocated in a contiguous array, and each
reactor is accessed from a different core, so align the reactor
structure to avoid false sharing.
Change-Id: I95162620ccb58fae060b2d95e47a38621dfbd140
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It is private to lib/event/reactor.c and does not need to be exposed in
the global namespace.
Change-Id: Idfff0365a0afdd90a0567825d520adf61d99fd2b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Previously, we did not calculate the ref for the LUN.
Change-Id: If2b7bc7d129e7efd994a7987ae2c421048969acb
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
An SGE could be for a payload that is greater than the NVMe
devices MDTS (i.e. 128KB), but that SGE may not be aligned
on a sector-size boundary. We can safely assume that each
iov is individually physically contiguous - the DPDK
mempools for example guarantee this.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8143ed01814c3154d0a06b8bbc548484437c1e88
The spdk_nvme_qpair::num_entries value is never used in the common code,
so move it to the individual transport qpairs to make it clear that it
is a transport-specific implementation detail.
Change-Id: I5c8f0de4fcd808912ba6d248cf5cee816079fd32
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The 'next' event pointer was never used in the entire code base (always
NULL).
Change-Id: I75f999d3a2e10512d86edec1a5a46ef263e2635b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use 'struct spdk_event *' directly for consistency with the rest of the
API.
Change-Id: Ib41a9bf47f5b18f4aebf5f4dee055455cb12ef7d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows the elimination of the spdk_event_get_arg1() and
spdk_event_get_arg2() macros, which accessed the event structure
directly; this was preventing the event structure definition from being
moved out of the public API header.
Change-Id: I74eced799ad7df61ff0b1390c63fb533e3fae8eb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The public API user is supposed to retrieve the defaults via the
spdk_app_opts_init() function.
Change-Id: Ie2bd6e809b2d47dbd5d62d396e8715f89f4052d9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The spdk_poller_register() function provides a way to pass an event to
call once the poller is registered, but it is always NULL in the current
code base.
Change-Id: I459bf40ae4d050589577d113b7984f1563aaa9cc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The event->next field can be accessed directly from within the event
library implementation, and public API users should not be using it.
Change-Id: I98a1f0017e03e951d0c4eee3c7989b04324e57d1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It is only used within bdev.c and can be static.
Change-Id: Id6e2cd9e5dd61a3ef1e1a27993d7a5ea7728bff2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is consistent with the other internal-only API headers.
Change-Id: I2c4748977d38a6c173311d26197d6273c168da7d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The definition of SPDK_UNREACHABLE uses the build-time DEBUG definition,
which is not available in the public API.
Change-Id: I1862c99fa5c85ccd3483f94e9c35de531da57f3c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Instead of passing the work completion, just pass the
response index. This keeps the work completions localized
to the polling function.
Change-Id: I0e6a1d8564200b5ac3aa43dfd58ae152d439bbd8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This eliminates an if statement, since the two callers
of this function know the desired queue size.
Change-Id: I28fabac8613f7b8fc7d96cf95b085b6e4dcf985f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Just call the regular qpair create function instead.
Change-Id: Ic35b1eb6fcdf0d82733ea573a493f583dd63d5bd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Use the num_entries value in the generic qpair instead. These
values had to match anyway.
Change-Id: Ia6400fbaba97df3ef6db4dc07a2ab95af1e5143f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change the reactor name, replacing space with underline.
Since Linux system didn't recommend file name with space.
And when reactor crashed, the core dump file name has space in it.
Change-Id: Iba36ba7903c95db09a9decbc023a01e5e6ab18b4
Signed-off-by: Liang Yan <liang.z.yan@intel.com>
Avoid an extra level of pointer chasing when we are filling out the NVMe
SGL.
Change-Id: I1a40af16fda80f7480c419524876bfb1a1902eb8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow it to be better be reused for some future patches
enabling splitting of non-PRP-compliant SGL-based requests.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ica38fd6cf191f72baa524bcc4896b3c9939ab762
This intermediate function is no longer needed.
Change-Id: I3523cc6d8f3b290165a953d42cca8b76eda762c5
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Sending the fabric connect command is part of establishing
a connection, so move it into the main connection-establishing
function.
Change-Id: I55e7ffdd16b576c81b51d7d3910203f9afc1f4c2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This function initializes the members of an existing
qpair struct. It doesn't construct one from scratch.
Change-Id: I0b9afac1ad25cfb217efd146702f693c74f5f697
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
No need to allocate all of the requests and responses until
we know a connection can be established.
Change-Id: I072a10aadfd7ced773634448f7d7e788622d0a4c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The code is clearer if this function is incorporated
into its only caller.
Change-Id: I33901cddf80ae27896b2acfd1b9e7d212f21f5f3
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is resolving the address and route to the target, not
binding a socket to an address.
Change-Id: I80055481ed2e020410a1e186a4e7371b60faaee9
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
They were very close to the same already, so finish the job.
Change-Id: Ifba9e3b2d11a3e70cbfbe46f57a67552db2757ed
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
We should be sending the bounce buffer's remote key to the target so it
can put it into an RDMA SGE on the remote side.
Change-Id: Icded155ad2292c67baa722f001c9c07178bc2754
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
There is no particular reason for this to be 127; make it 128 to at
least be consistent with the PCIe transport.
Change-Id: I60500e0044d3549ba6350e1f35f09d624848bd21
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This function was only called from one place and saved no
lines of code.
Change-Id: If5e653732df57c1f2c93e20cf4f286eac31df91c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This matches the behavior of nvme_pcie, which queues a request if no
tracker is available.
Change-Id: Idbf6c951c89451cfea22ec6bc553ff46f988f818
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the caller pre-allocate an rdma_req and change req_init() so it
only does initialization, not allocation.
This is necessary to distinguish between rdma_req allocation failure and
other types of failures, which will become important in future patches
when requests will be queued if rdma_req allocation fails.
Change-Id: Ie6edebc1b5f05001b42fc959a29ce0ea6875e41e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Simplify the control flow and match the name of the function to its
purpose.
Change-Id: I65bad7e3b2ef710ca29eff9799b8dcaae3999315
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the qpair construct functions private to the transports - it
doesn't need to be called from generic code.
Change-Id: I5f730a4bcf60ce231fe27bc8f4c3c39cb647dd2d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a transport callback to return the maximum queue size, and enforce
it in the generic nvme_ctrlr layer.
This allows the user to tell what io_queue_size was actually selected by
the transport via the ctrlr_opts returned during attach_cb.
Change-Id: I8a51332cc01c6655e2a3a171bb92877fe48ea267
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Equivalent to commit 6ab28a201b except now
for commands instead of responses.
Change-Id: Ibe4382dc0f65c1b90c2cee2ad285bbdd21b96a89
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The only field of bb_sgl that was actually used is lkey, and that is
already stored in bb_mr.
Change-Id: I790369a06ce223f88e356df20a9d9a74a93ff225
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This also changes the default listen address from 0.0.0.0 (accept any
connection) to 127.0.0.1 (accept only connections from the local host).
Change-Id: I3de09c582c95126d240795550a56be7aedea639c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Generate the full discovery log page in a memory buffer, then copy just
the requested part of it for each Get Log Page call.
Change-Id: I12730c59c0395cdac57aaab96337e938952e3011
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Refactor the discover log page processing into a loop that calls a
function for each log page entry. This sets us up to add support for
multiple Get Log Page calls to handle larger discovery service lists.
Change-Id: I85676ada375d0dadda2a3f4ab6331123ac7aaf60
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Note that the offset is not actually used yet, just sanity checked.
Change-Id: I9464dc934e94e3d38ac0d474fce876552650f92b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows hosts to determine when the discovery log page has changed
when reading it across multiple Get Log Page calls.
Change-Id: I3c3459959c6246a88938e4f82e3e0046419e7d00
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This keeps the existing subsystem list (and therefore the discovery
service log page) in order when new subsystems are added dynamically.
Change-Id: I071639be0fef4139f8f017b433185c786ae55378
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This isn't used yet in the NVMe library, but it will be necessary later
for supporting non-IPv4 addresses.
Change-Id: I167ce63ad25b0e0c9aa192b12d764c8d078e67f9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This better describes what the field controls (it does not affect the
admin queue size).
Change-Id: I851ae46fb4ed0fce819af07ae235824e0fc817e6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For requests split in _nvme_ns_cmd_split_request() the payload offset is
set after children are created using recurrent call _nvme_ns_cmd_rw().
This makes impossible to reset SGL to proper offset in
incomming patches that split non-PRP complaint SGL requests.
To change this the payload offset is set after each request is allocated
in _nvme_ns_cmd_rw() not in _nvme_ns_cmd_split_request().
Change-Id: I9d3b2e3bbd9d93a4c8a37e1db8c4e01276e2cacb
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
This is preparation for handling non-PRP compliant SGL.
Change-Id: I445790f9802292971256cf821d9730814c95a073
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
This is preparation for handling non-PRP compliant SGL.
Change-Id: I49c3745498411c5ff9e17cd08f181d4d434c2d08
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Comply with the definition format used by other bdev
modules
Change-Id: Iac108bac540687b32fea4bb70374c22534c60aa0
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The vendor ID for Intel shoudl be "INTEL", according
to the following page:
http://www.t10.org/lists/vid-alph.htm#VID_V
Change-Id: Ib9611e5604c8b5e3eaec8101548aaf4a3c45597a
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Don't leave garbage from previous discover entries in the trid we are
returning to the user.
Change-Id: I60ae5932db4a95cedb8df1ff98a2479220b55ce4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The other simplifications to probe_info and trid made the
trtype argument redundant.
Change-Id: Ie7bea4e2204e690dc4909eeacd065e0722b53272
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The probe_info was reduced to just containing a
transport_id, so remove probe_info entirely.
Change-Id: Ica9a22d126cd14e282decd3eea1a0afe0460f099
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This can be obtained by parsing traddr into a pci_addr,
then getting a handle to the pci_dev and asking for all
of the pci information.
Change-Id: I1948cbd3ec65611293192ef5558ace19dd444d4c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This function will return a device handle from a pci
address.
Change-Id: I323d92c71014ef571f3df9f19c2ec887844707e8
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This bug was preventing multiple calls to spdk_nvme_probe() from
working, since the first call would return 0 from all of the DPDK driver
init callbacks and prevent other devices from ever being enumerated in
subsequent calls.
Reported-by: Tsuyoshi Uchida <tuchida@us.fujitsu.com>
Change-Id: I871aa170bbd03be111604eeabe3a7a7a4f40ce89
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the standard quirk mechanism to specify which devices
need software assisted striping.
Change-Id: Id8156876a90b4caf9d687637e14c7ad4a66ceda6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The emulated NVMe controller exposed by virtual subsystems does not
provide the Intel vendor-specific commands and behaviors, so it should
not use the 0x8086 vendor ID.
Change-Id: Iab4f0513d30f610feb62b1899da1b6316f11691c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This way, all new controllers discovered will be initialized
in parallel.
Change-Id: Iebedb3905eb2787a3708f74425afae40ca31253d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
If the first call to spdk_nvme_probe probes a device and
the driver elects not to take it, still call the probe
callback for that device on subsequence calls to
spdk_nvme_probe.
Change-Id: If06467cf6796c827a0bbfba6e36d5b91534526fc
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Move this down a level so it happens on all paths.
Change-Id: Iea9913f0e102353882466c8dea4ee39abb857520
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Scanning the transport may result in both new
devices and removed devices, so pass the callback
for both operations.
Change-Id: I6f73dbe6fd7cf61575c354b43f8ae3e2a01e2965
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Simplify the arguments to nvme_transport_ctrlr_scan to take
a transport id that identifies the discovery service (or
NULL to scan PCIe).
Further, separate scan into two functions - scan and attach.
Scan is for scanning an entire bus, attach is for a specific
device.
Change-Id: I464f351a02a04bc5a45096dcf5dc8fc5ac489041
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Instead of repeating the fields, just embed a transport_id.
Change-Id: I282704c9d59784abd5f7c93be4e47c673fcf6dde
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is a small step toward making discovery more like
scanning a local PCI bus.
Change-Id: Ie7149ad060f2eeb56939b1241187bdf09681f2aa
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Before adding readv/writev support in nvme_rdma,
using this patch.
Change-Id: I25ff0df61d0346f22560d011158d7f80e72007ea
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
We cannot quit the process when user did not Logout from the session,
because the active connections always bigger than zero. User cannot
use Ctrl+C to quit SPDK iSCSI target. Add a new state to connection
to avoid destruct connection more than once.
Change-Id: I8efa79aa47534bd6ead965713769f751d9802e47
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Drop the complicated buffer size/strlen math and just split the version
string formatting into two cases depending on whether the tertiary
version is set.
Change-Id: I4b4983cb8805e8734c408f473dd8c592ec8e8138
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The printf # specifier adds 0x for %x values, but the field width then
includes the 0x part, so for example printf("%#04x", 0x1) prints "0x01"
rather than the intended "0x0001".
Rather than increasing the field width, just manually insert the 0x in
the format string and drop # for less confusion.
Change-Id: Ie6044619a22b51b39562bfa5c0c0239933bf38c8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
NUMDU was added with NVMe 1.2.1 and allows a larger log page size to be
described.
Change-Id: I1a4ac42393c1a21175b3564980d56b6e7a6ae80d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The NVMe over Fabrics transports should already be setting this in the
initial admin queue Connect command, so setting it again is not useful.
The kernel NVMe over Fabrics target additionally has a bug in the Set
Features - Keep Alive Timeout handler (it is extracting the KATO value
from the wrong offset in the command), so this works around the kernel
bug by not sending the Set Features command at all.
Change-Id: I0d7f09b71fcea116acf8810c5880157bb9315a04
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The reason is that kernel nvmf target will check the
value. If not set, it will fail the other commands later.
Even for discovery ctrlr, kernel nvmf target will
check the cc value.
Change-Id: I998327f91ba96281d261952878eb84d648a823da
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
It's not the whole transport - it's just an enum for the
type of transport.
Change-Id: Ia435a21792f221ddf50ddf4f0923c6152622eccb
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
When we enabled the ERL1 configuation, for the DATAIN task release
process, we will queue the task to the SNACK list firstly, and then
remove the list when got ACK from initiator, but for this part of
logic, the reference count of primary task was not released correctly.
Change-Id: Ic5959cf644c74f676be0b84c5650292dc426b2d8
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change it according to the spec thus we can test
kernel nvmf target
Change-Id: Ica98dd40503a40c0f0de8efaefb1f6f67a89cde8
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change the PCI enumeration API to individual functions per device type
so that only the drivers that are actually in use get linked into the
final executable. All of the common code is still shared internally in
the env_dpdk library.
Change-Id: I2ba83afe59202a510f999a0674e23e60b6581221
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It is not necessary, and it prevents the linker from removing unused
object files.
Fix the iscsi_tgt Makefile's library order so that env is added at the
end after the libraries that use it.
Change-Id: I241eb46703c12691444037a350be65143259e82e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add the following infromation.
- PCI Address
- Vendor ID
- Model Number
- Serial Number
- Firmware Revision
- NVMe spec version
- Namespace sector size
- Namespace total size
The user's remove_cb should detach the NVMe controller when it can
ensure that it is no longer in use. In the interim (between remove_cb
and spdk_nvme_detach()), the controller will remain in a failed state,
so any new I/O submissions will return an error code but not crash.
examples/nvme/hotplug is not yet updated for this change, but that will
be done in a separate patch.
Change-Id: I8827ba36f9688ccb734e7871f20f11ec11e88f96
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
While we're here, fix up typos and add error logs for all error exits
in nvme_rdma_qpair_connect().
Change-Id: I236fe6571c2012ca047aa8a447638d9227454c2f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This version of multi-process support needs to have DPDK 16.11 builtin.
Change-Id: I3352944516f327800b4bd640347afc6127d82ed4
Signed-off-by: GangCao <gang.cao@intel.com>
The discover and probe 'nqn' fields are subsystem NQNs, so name them
subnqn to be consistent with the spec and the rest of the code and to
distinguish them from host NQNs.
Change-Id: I4a80fbc1f4b037c8a4f91c8f28d2a96e47c66c47
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Allow the host NQN to be overriden when connecting to NVMe over Fabrics
controllers.
Change-Id: I8fcf2e89ae7d9722677e834f76a8fe805c52f91b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This makes the function and file/line info actually useful (instead of
pointing to the helper function itself).
Change-Id: I22bac68827115880a49d456706a7eaecdc12e9b5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Each transport should handle its own qpair cleanup internally.
Change-Id: I7dd737be820ea6bad686f4aad7d74044fad58a47
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Let the transport access the controller options during
ctrlr_construct().
Change-Id: I83590c111e75c843685dd9315f0f08416168356d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_rdma_req_get() is an internal function, and its only caller already
checks for a valid rqpair, so the NULL check is unnecessary.
Also clean up the redundant STAILQ_EMPTY/STAILQ_FIRST logic and use
STAILQ_REMOVE_HEAD.
Change-Id: Ic3828e8b5e881879173cb59350e39c5fac90e6ef
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_rdma_pre_copy_mem() does not have any failure cases, so remove its
return value and remove the never-taken branch in its only caller,
nvme_rdma_qpair_submit_request().
Change-Id: I91011734ed0c20f8db691d62172fe1a3021dd3a1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_rdma_req_put() is an internal nvme_rdma.c function, and all of the
callers already have the rqpair, so pass it directly. We also already
verify that all of the callers have a valid rqpair and req before
calling nvme_rdma_req_put(), so it doesn't need to check for NULL
pointers.
This also means that spdk_nvme_rdma_req doesn't need to hold a pointer
to its rqpair anymore.
Change-Id: I893a46a9074f0a843e379d10c123f9292eb3b1a4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The only place where outstanding_reqs was checked was in
nvme_rdma_req_put(), but the error case there could only happen if some
kind of internal programming error occurred (e.g. calling
nvme_rdma_req_put() on an invalid request).
Change-Id: I71e40ce562a8720dfaf70437ffd4c6493327c091
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_rdma_ibv_send_wr_init() was only called in one place, so just move
its contents into nvme_rdma_qpair_submit_request() since it allows
simplification of the code:
- req was always NULL, so remove the code that used req entirely.
- wr and sg_list are never NULL, so remove the checks for those.
Change-Id: I12a4f3502219d3681607686945e343f6808c0d2f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
We currently don't handle discovery service referrals, so skip those, as
well as any other unknown subsystem type.
Change-Id: I64f889e9272fb57b5cf9bb5467b3abca3955baf5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
QEMU's virtual NVMe controller device does not support the AER Set
Feature, so ignore its failure and continue.
Change-Id: I8b5c217a3112edabb6f76ec3e5f4ef774981a1d7
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Catch SIGBUS and handle it by remapping new memory into the
location where the BAR previously was.
Change-Id: Ie8d00a60a0bbe7f7ec57a5c39c0a63c5d9443206
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
These functions will attach or detach from a PCI device. Attaching
typically means mapping the BAR.
Change-Id: Iaaf59010b8a0366d32ec80bb90c1c277ada7cfe7
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
This patch make sure the connection in normal state before any further
operation on this connection.
Change-Id: I776740b5b33b1de6707990c09d9131c385adf556
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
spdk_nvme_probe frees ctrlr when nvme_ctrlr_process_init is failed. But
ctrlr has already been freed while calling nvme_ctrlr_destruct. So
spdk_nvme_probe doen't need to free ctrlr.
The generic NVMe library controller initialization process already
handles enabling the controller; the RDMA transport should not need to
set EN itself.
For now, the discovery controller is cheating and not using the normal
initialization process, so move the EN = 1 hack to the discovery
controller bringup until it is overhauled to use the full
nvme_ctrlr_process_init() path.
The previous code where CC.EN was set to 1 before going through the
controller init process would cause an EN = 1 to EN = 0 transition,
which triggers a controller level reset.
This change stops us from causing a reset during the controller
startup sequence, which is defined by the NVMe over Fabrics spec as
terminating the host/controller association (breaking the connection).
Our NVMe over Fabrics target does not yet implement this correctly, but
we should still do the right thing in preparation for a full reset
implementation.
This patch also reverts the NVMe over Fabrics target reset
handling hack that was added as part of the NVMe over Fabrics host
commit to its previous state of just printing an error message.
Change-Id: I0aedd73dfd2dd1168e7b13b79575cc387737d4f0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Most of the NOTICE level messages should have been TRACE.
Change-Id: Icbc4d398ab2580cf3a2349be11441b7a09603020
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Verify that qpair is not NULL before doing pointer math on it.
The NULL check after calling nvme_rdma_qpair(qpair) would not
trigger if qpair was NULL.
Fixes a crash if the Connect command failed, causing
nvme_rdma_ctrlr_create_qpair() to return NULL.
Change-Id: I158a5b1752892a7d5a72a9ac20c0c5b2cd781a81
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows us to print better error messages when connecting to a
subsystem that exists but does not allow a specific host.
Additionally, we can now return the correct error code for a host that
is not allowed.
Change-Id: I16cd4ac2745cf50bb54601b464b0d23954f86fda
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Official installs of DPDK place headers in a 'dpdk'
subdirectory under include, so detect that.
Change-Id: If64421c84c91cae31688994484c22fce398dc622
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The status.done flag polled by nvme_ctrlr_set_keep_alive_timeout()
was never initialized.
Change-Id: I323fae5f4ce12209a9699965ce07894bc3c6205a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The code in virtual.c and direct.c was identical - move it to session.c
to share it.
Change-Id: Ic6e4e9238e8ffacb212e76293c440109aa839f8c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Move the current Virtual mode implementation to session.c and use it for
Direct as well.
Change-Id: I3f0ac93b4247b93d158b0dcb77e257b4b91be129
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Store the host identifier from the Connect command and report it via Get
Features.
Change-Id: I79bc27e05c5944549e7986aadb919c19748e7474
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Also return Invalid Field rather than Invalid Opcode to be more
accurate. The spec doesn't seem to define any more specific error code
for this case.
Change-Id: I992c6cca3020ff80b8495c71170222bc75316800
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
None of the log pages are actually implemented yet, but at the very
least, we don't want to leak random bits of uninitialized data.
Change-Id: Ic889260eb18d49122f2f250b645bdc5be3561dc5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the NVMe over Fabrics spec definitions for TRTYPE rather than the
internal library transport type.
Change-Id: Idead559a8f8d95274fc580d10e82033822e6eda8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These need to be available for the lifetime of the probe_info structure,
so they can't be pointing at e.g. temporary buffers on the stack.
Change-Id: I5aaa898acf9314aab51600dd756f966965d37fd0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
-Wformat-nonliteral needs to be disabled since clang triggers it on the
call to vsnprintf() now that it is nested two calls deep.
Change-Id: I228b9d099cfc2b65181941cbb4798b7f8eae3baa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is a counterpart to spdk_strcpy_pad() which determines the length
of a string in a fixed-size buffer that may be right-padded with a
specific character.
Change-Id: I2dab8d218ee9d55f7c264daa3956c2752d9fc7f7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It always points to the same internal RDMA request complete function, so
just call that function directly.
Change-Id: Ic1fb6236bf43eaad62413df77d43be9ab855e5c7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
We can't transfer more than the bounce buffer in a single command, so
report that rather than some bogus value.
Change-Id: I39b147916dcc2ee478470917298763a239a6a35a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Record the user-provided asynchronous event configuration set via Set
Features, and return it in Get Features.
This value is not actually used, since AER is not implemented yet in the
virtual controller model, but it at least implements the mandatory
Set/Get Features.
This allows the hack in the NVMe host code that ignored the Set Features
failure to be reverted.
Change-Id: I2ac639eb8b069ef8e87230a21fa77225f32aedde
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fill in the cached copy of CAP in the generic NVMe controller to match
the PCIe transport.
This is not really early enough, since CAP is used during the reset
process to determine the reset timeout, but that will have to be fixed
separately by rearranging some of the transport callbacks.
Change-Id: Ia8e20dbb8f21c2871afb9e00db56d0730e597331
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make sure the entire NQN field is zero-padded, rather than using
strlen() on the input.
Change-Id: Icee68bd033feed057813beeb30cec102ed90840e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This fixes a compiler warning about unhandled enum cases in a switch.
Change-Id: Icecb56b47a05c13f390f03b877f8eae243b481a6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
- add SPDK_NVME_OPC_KEEP_ALIVE to admin_opcode
- add SPDK_NVME_SC_INVALID_SGL_OFFSET, SPDK_NVME_SC_INVALID_SGL_OFFSET,
SPDK_NVME_SC_HOSTID_INCONSISTENT_FORMAT, SPDK_NVME_SC_KEEP_ALIVE_EXPIRED
and SPDK_NVME_SC_KEEP_ALIVE_INVALID to generic_status
Make it easier to use SPDK libraries by putting them all in a single
directory that can be added with -L rather than scattered around the
source tree.
Change-Id: I5c0f5dd6e7058b5f92fa9bc41548190ffc064761
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Track the maximum copy task size as modules are registered rather than
recalculating it every time spdk_copy_task_size() is called.
Change-Id: I141aca61e7075402dac41915080d1b43faee32ce
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the public API clearer - if the user wants to allocate a
spdk_copy_task directly, they need to allocate spdk_copy_task_size()
bytes.
Also change the return type to size_t for consistency.
Change-Id: I0f3757056757c510421d680c5b4532edd9bc2561
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The SPDK_TRACELOG macro depends on a CONFIG setting (DEBUG), so it
should not be part of the public API.
Create a new include/spdk_internal directory for headers that should
only be used within SPDK, not exported for public use.
Change-Id: I39b90ce57da3270e735ba32210c4b3a3468c460b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Remove usage of the conf structs so they can be moved out of the public
API header.
Change-Id: I1c7375ec7708b323f50af09aeb7b2b2c9c770df4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Considering the process can be terminated in the cases like ctrl+c,
kill command or memory fault, the ref is tracked in the per process
structure spdk_nvme_controller_process and whenever there is other
process attaches or detaches the controller, a scan will be issued
to cleanup those unexpectedly exited processes.
Change-Id: Ib4f974f567a865748d42da4ead49edd383dfc752
Signed-off-by: GangCao <gang.cao@intel.com>
These APIs can be used to register/unregister regions
of pinned, huge page memory that are separate from
huge page memory allocated by the default DPDK
allocations. These APIs will be used by an upcoming
SPDK vhost-scsi target to enable SPDK to target
NVMe DMA operations directly to VM memory that has
been allocated by QEMU using pinned huge pages.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I649a4adeeb758b29bd29cd42c8872eed3d5d6ce9
Now that the env PCI framework already requires enumerating devices
based on an enum of specific device types, it is not useful to query the
class code of a PCI device handle.
It is currently unused and does not work in its current form on FreeBSD
(it reads a file from /sys). This lets us drop a big chunk of file
reading and parsing code.
Change-Id: I1d720398416ba3d6f91e077b807ec11a6de562cf
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The details of the structure were removed earlier, but
now remove all references even to a pointer to the
structure. The user can refer to transports by their
string name.
Change-Id: I273356f46329ea5372dcd951eda6f14767477d69
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is a step toward abstracting away the definition
of the subsystem.
Change-Id: I88b2aa107b27152620f51a1ca2a153792b4c85e9
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Remove 4k allocation size in spdk_scsi_task_alloc_data(). From now on
all commands must obay allocation length.
Change-Id: Ica9384c62d431483ae1d0bd2e6fdee18b570861f
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
This refactor MODE SENSE 6 and 10 related functions to respect buffer
size parameter.
Change-Id: I03bad456bac0554a8bf7b56f69d1f9cf5b1991f6
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
This patch is preparation for fixing alloc_len overrun in SENSE 6/10 and
READCAP 6/10. To simplify code forbid usage of iov outside of
scsi/task.c.
This also drop SPDK_SCSI_TASK_ALLOC_BUFFER flag that obfuscate code. As
a replacement assume that if field alloc_len is non zero it mean that
iov.buffer is internally allocated. Functions
spdk_scsi_task_free_data(), spdk_scsi_task_set_data() and
spdk_scsi_task_alloc_data() manage this field.
Change-Id: Ife357a5bc36121f93a4c5d259b9a5a01559e7708
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Use the len field from the generic spdk_bdev_io instead of duplicating
it in blockdev_rbd_io.
Change-Id: I3ebfab8dd1303add83bc2206fc87319ba7d605b3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This function needs to check for SGEs that straddle a
2MB page boundary, and ensure it does not return
a length that will cross that boundary.
This cannot happen in practice currently with SPDK
since all buffers are allocated using rte_malloc(),
but an upcoming vhost-scsi target may produce
SGEs from a guest VM's physical memory that span
a 2MB boundary.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8b83c7c39c4cf33815abb22ff2ebc90941b21e28
No functional change, but removes a few assumptions
that will be invalid in a future patch that fixes a
bug in this function. Primarily we no longer assume
that this function will always increment the
iovpos and reset iov_offset to 0.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I770f2f24c37626063e113af850a2af792aed332a
The bdev function table should not be part of the public API.
Change-Id: I5d6f40d1b37c4471041c1c9d6253a3f92e9e9701
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It was written but never read (and the I/O channel is already stored in
the generic spdk_bdev_io).
Change-Id: Id33392e9d3940b2c1439e9fed2553aa091ecedf8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
No need to duplicate the bdev-defined I/O type.
Change-Id: I15cb68c3c68b3f25b286b04500b53081ed5e7881
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The status field in blockdev_rbd_io was only used within
blockdev_rbd_io_poll(), so replace it with a local variable.
Change-Id: I3629225f28b752a3acc7521699c33bc98f1e4b7b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Instead of the next_sge callback returning the physical address
directly, make it return the virtual address and convert to physical
address inside the NVMe library.
This is necessary for NVMe over Fabrics host support, since the RDMA
userspace API requires virtual addresses rather than physical addresses.
It is also more consistent with the normal non-SGL NVMe functions that
already take virtual addresses.
Change-Id: I79a7af64ead987535f6bf3057b2b22aef3171c5b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Remove the complex list management for pool_name and just strdup() it
directly. It is not worth the trouble to save a few bytes.
Change-Id: I8a4f7eeea619bd824ea593854423e317041c540e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Remove a DPDK dependency from generic code.
Change-Id: I8e3e2c0a36d980b426a1967ed1f88fb8b855c382
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Custom bdev modules can return any SCSI status and SCSI sense
information to a host by this patch. This is usefull when a custome bdev
module detect an error in the module and need to return meaningful
information to a host.
Function pointers will not work for the DPDK multi-process model (they
can have different addresses in different processes), so define a
transport enum and dispatch functions that switch on the transport type
instead.
Change-Id: Ic16866786eba5e523ce533e56e7a5c92672eb2a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make a wrapper that spdk can call a function without thread affinity, and
call this wrapper to open rbd image.
Change-Id: Iadc87a948f43632abf497f88165483a0e269ba54
This enables using SPDK within a larger process that
is SPDK-centric. In this case the process may start
SPDK and then wish to stop it explicitly (without a
signal).
While here, remove an incorrect comment - DPDK mempools
can be used from non-DPDK threads. Also set the
g_shutdown_event to NULL after it is called. After the
event executes, the event is freed and is no longer valid.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie4f07bee7d05fae683c72f6680cb3bcce2d4a119
The initialization of dev_addr was replaced with probe_info.pci_addr,
but its use in spdk_pci_addr_compare() wasn't replaced to match.
Fixes commit fcb00f3780 (nvme: expand
probe information to a struct).
Change-Id: Ic4c273d2aa0bf1f9e3e1527f3ab09d3c019158cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Since we are usually going to be removing multiple events from the queue
at once, use the DPDK burst dequeue interface to improve efficiency.
Also rework the event queue runner to always process a fixed maximum
number of events per timeslice for simplicity. This removes the
rte_ring_count() call from the hot path and improves fairness between
events and pollers.
Now that events are dequeued in bulk, we can also put the event objects
back into the mempool in bulk. Add an env wrapper around
rte_mempool_put_bulk() and use it to free all of the events at once.
Basic performance benchmark using test/lib/event/event/event -t 10
is improved: previously ~40 million events per second, now ~46 million
events per second.
Change-Id: I432e8a48774a087eec2be3a64c38c339608af42a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It always returns NULL in the current DPDK env implementation and was
not used outside of a few ioat examples where it is not particularly
informational.
Change-Id: I14b237c33bc25ddebc6b36bfbd6a4edf6762e3ca
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This removes the 2 bytes of SenseLength from the beginning of the SCSI
sense_data buffer, so now the offsets within sense.data match up to the
expected values from the SCSI spec.
Change-Id: I9188560096a9ec5a8fcf83bec95201521b127494
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
spdk_nvme_probe() will now provide a struct spdk_nvme_probe_info to the
probe and attach callbacks in place of the PCI device pointer.
This struct contains the useful information that could be retrieved from
the PCI device during probe.
The goal of this change is to allow expansion of the probe information
in the future when other transports (specifically, NVMe over Fabrics)
are added that do not necessarily use PCI addressing or device IDs.
Change-Id: I59a2a9e874e248ce5fa1d7f4b57c8056962ff3cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a helper function that converts a PCI address from a string into a
struct spdk_pci_addr and use it in place of the various sscanf()
invocations throughout SPDK.
Change-Id: Id2749723f76db741567e01b4bcb0fffb0e425fcd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Print the error information when the kernel RNIC driver did not load
properly, and fix the cleanup logic for the exceptional exit.
Change-Id: I97a45e73d830280b994818f3defc491bc2b6b020
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
As we can support multiple sessions now for each Subsystem, the Host
will use cntlid field to create IO queues, if 2 different Hosts
connected to the same Subsystem, for IO queues' creation process, it
will use cntlid field with 0 for current code logic.
Change-Id: I6fd437892e8eb3146f62f4b211c0baadd70b505e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Add an RPC interface to list all blockdevs and their properties.
Change-Id: I50db730d5eff8cffcbe8fe5df6b3461457e8581e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The PCI device claim function does not need the whole spdk_pci_device
structure, just the address.
Change-Id: If59df512043ee062cf9f759bdc104fc522625ba8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The NVMe over Fabrics target was storing the PCI device pointer for each
direct-mode controller, but it only really needs the PCI address, which
is exposed via the get_nvmf_subsystems RPC.
Also update the same code path to use the new spdk_pci_device_get_addr()
function for brevity.
Change-Id: I0708b3331b7c279c1a86f0d7459b5deb40dd7c89
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the new public PCI ID structure in the NVMe library to replace the
previously private struct pci_id.
Change-Id: I267d343917f60bdae949a824bc0fe67457cbbc0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
- Split the part that gets a PCI device's address into its own function,
spdk_pci_device_get_addr(). This is useful outside of the comparison
function and is orthogonal to comparing addresses.
- Make the comparison function take two addresses instead of a device
and an address. The more general form will be useful with addresses
that are not directly associated with a device. Because of this, also
rename the function from spdk_pci_device_compare_addr() to
spdk_pci_addr_compare().
- Return a signed value similar to strcmp() so that addresses can be
ordered, not just compared for equality.
Change-Id: Idf304454af09ea57f1e1d5dc3a39b077378cecad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename the construct_rbd_bdev "size" parameter to block_size so that it
is consistent with other bdev construct RPCs.
Change-Id: I88f8ed35444495ffce9550dc224fbcbd58231787
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
When creating a bdev via the RPC interface, there was no way to know
what name it was assigned (other than predicting it based on the
numbering scheme). Change all of the relevant RPC interfaces to return
an array of bdev names so they can be used to construct LUNs/subsystems
dynamically in scripts.
Change-Id: I8e03349bdc81afd3d69247396a20df5fcf050f40
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a field to struct spdk_nvme_ctrlr_opts that allows the user to
specify a keep alive timeout, and add automatic submission of Keep Alive
commands to spdk_nvme_ctrlr_process_admin_completions().
Change-Id: Ib282299a571d8edc59c7933418751bc3a6c98b40
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Specify SPDK_JSON_WRITE_FLAG_FORMATTED when creating a write context to
output more human-readable JSON.
Change-Id: Ie1f0451496aae7e36e4cdb1f05edb4bc4963be17
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If the RDMA transport failed to initialize, g_rdma.event_channel may be
NULL.
Change-Id: I4510ee5893389f244f0fbaa1cd4a182868939b25
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For iWARP devices, buffers that are intended to be the
target of an RDMA read initiated by the target must additionally
have IBV_ACCESS_REMOTE_WRITE permission. This is because iWARP's
RDMA read path essentially requests the remote side to do
an RDMA write.
This is unfortunate because there is no way to differentiate between
memory that the remote side can do an RDMA write to and memory
that will only be the target of RDMA reads initiated by the
target. There is nothing we can do about this serious deficiency in
the specification, however, so we have to live with it.
Change-Id: I3d2f2814ce0cb1df4e5347296ef371db4d16be21
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This patch adds support for spdk_bdev_readv in scsi layer.
It also fixes write so that it uses multiple iov's instead of one.
Currently we should use only task->iov (for single vector operation)
or task->iovs (for multiple vector operations).
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Ia3b2f6d18fd212b11d7b63b11dc46ec5bbc74788
Make the quirks mechanism generic in preparation for quirks for devices
from other vendors.
Change-Id: Ic003b020a38f1b966021db30e3f2bce9cf6a1a0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch removes reduntant field in spdk_scsi_task and
fixes all logic to use iov.iov_base
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Ie2fa1e2357b6383c118d05aec9206d1c60537d40
Previously, if spdk_rpc_setup() returned early due to the RPC service
being disabled in the configuration file, it would leave itself
registered as a poller and continue to run for the life of the app.
Change-Id: I0532fe23a732b87d68f83847b2db7627f87e9a1c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
I believe this is required for NICs to report, but handle
the case where it isn't reported.
Change-Id: I38d10c3590d1df8bb902ab312af0f9e01b9e5032
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This makes it consistent with the way connections and
requests work.
Change-Id: Ifb97499ba72f7dfd02ac54ba1b622726d266262c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The shared memory pool for a session is associated with
a particular RNIC via the protection domain. New connections
attempting to join a session that came in on a different RNIC
can't use that memory, so must be rejected.
Change-Id: Ibd79fe90566a231f76b7472e5e9b484c3e528454
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Rearrange the functions in rdma.c to match the order
of the function pointers in the transport. No other
code changes.
Change-Id: I9dbc68912ecd5dfdf53f20b4807d4116933a3c3a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Use the lower level registration functions. The RDMA-CM
examples use the ibv_* versions, so who knows if the
rdma_reg_* wrappers are even well tested.
Change-Id: I8e8250ab09a1401e636aebe2fc04a60806f7a827
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Add a transport function to get the max data transfer size to break the
dependency on NVME_MAX_XFER_SIZE.
Change-Id: I846d12878bdd8b80903ca1b1b49b3bb8e2be98bb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Move the PCIe-specific admin queue setup to nvme_pcie_ctrlr_enable.
Change-Id: Ic3f5625fa804f719040ba86b7fc3bf82fcc057c0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Previously, we mixed use free and spdk_nvmf_rdma_conn_destroy to
free allocated spdk_nvmf_rdma_conn structure, which sounds not
exactly free all the resources.
Change-Id: I2917b442c34d63ba5c014add58f429ae4b831595
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The RDMA API doesn't say whether the wr is copied, so be
safe and allocate it on the heap.
Change-Id: I091af50aa031e1861333f19d864eb52335d6b756
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
A status member of spdk_bdev_io structure is set after the if block.
Therefore a status parameter should be checked instead of a status
member.
Change-Id: I4030a7fcdb36d9c589802ec5b4e424591dc2a3b6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The value of CAP should not change during the lifetime of a controller,
so read it once during ctrlr_construct and store it in the ctrlr.
Change-Id: I089d4141b4e0c9aae6c53abf9bb0ef6577dabe0b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than embedding adminq directly in the spdk_nvme_ctrlr structure,
change it to a pointer to a spdk_nvme_qpair. This is necessary to allow
the transport to extend the qpair structure.
Change-Id: I041685d5037088cf56d046fe99bf204edcfc57b1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Previously, we directly assigned the pointer of pool_name
and rbd_name, and this is not safe. After the rpc test,
we found the string value is not correct, so use strdup.
Change-Id: Ibadc57d3cb5b9869b7db5a22c2459769e92edebd
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
This requires a couple of related changes:
- I/O queue IDs are now allocated by using a bit array of free queue IDs
instead of keeping an array of pre-initialized qpair structures.
- The "create I/O qpair" function has been split into two: one to create
the queue pair at startup, and one to reinitialize an existing qpair
structure after a reset.
Change-Id: I4ff3bf79b40130044428516f233b07c839d1b548
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the transport ctrlr_construct callback responsible for allocating
its own controller.
Change-Id: I5102ee233df23e27349410ed063cde8bfdce4c67
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Four read/write functions share the same code for checking
IO len and offset. Extract this code into separate function.
Change-Id: I40f0021e70a60c591b048ad3a70b22eaa07af3b4
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
These are specific to local NVMe PCIe devices, so move them out of the
generic NVMe code into the PCIe transport.
Change-Id: Iea2056a4c438b7d3a303b4b5e977ce7aa9e58c05
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
