Imagine the following code flow:
1. `spdk_put_io_channel();`
2. `spdk_io_device_unregister();`
Since putting an io_channel is always deferred,
it is likely that spdk_io_device_unregister will
lock the io_channel mutex first. It will set
dev->unregistered flag and then quickly realize
there are still open channels for this device
(refcnt > 0) - so it'll return. All fine here.
However, if the deferred put_io_channel happens to
lock the mutex first from other thread, it will
decrement the refcnt and attempt to free the device.
Both the decrementation and freeing are done under
a mutex, but there is a slight window inbetween
where the mutex is re-locked. spdk_io_device_unregister
is already sleeping on a mutex_lock(), so it might
strike now. It'll see there are no more io_channels
for this device and free the device. Once
put_io_channels regains the lock again, it will attempt
freeing the device for a second time.
This patch removes the slight window mentioned above.
Related to #278
Change-Id: I6c0f6014353529028d658211135196d97f1d8547
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/408193
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Previously, there was a period of time where the user
had put the last reference to a channel, but when
calling to get a new channel would end up with the
previously created channel and channel destruction
would never execute.
This causes a number of unexpected issues to pop up,
as often the creation and destruction of a channel
is a key event.
Change-Id: Ie68135f6efdf45093a5a227f8c51cd11b971fcff
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/407602
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
The unregister callback is only invoked after all I/O channels related
to the I/O device have been freed. The last I/O channel to be freed
might be on a thread different than the thread where spdk_io_device_unregister
was called. If this is the case, send an event to the latter thread
so that the callback gets called on the same thread as spdk_io_device_unregister.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3c910198fbd83eae95e5c57d618284e64db46ff7
Reviewed-on: https://review.gerrithub.io/404414
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We must keep the lock held after removing the io_ch from the thread
before iterating the list of threads to check for other channels
referencing this thread. Without it, it is possible for two different
threads to think they are the last thread holding a reference to
an unregistered device, and both will call the unregister_cb and free
the dev memory.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ifcc18752937731722fefc6e2161ee41443c4fb86
Reviewed-on: https://review.gerrithub.io/404410
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Change-Id: If5e16c2ac148fd4818c942e8f57b02671e76ed62
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/403153
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
There existing some cases like in our UT code, the
start_poller_fn and stop_poller_fn is not configured.
Add a check here and properly handle these cases.
Change-Id: Iaac83d78547432584f6de0c54e9d2e06b9b35741
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/392996
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This requires exposing struct spdk_io_channel in the
public header - mark it as internal with Doxygen
comments to make it extra clear that applications
should not use the data structure directly.
This is a very hot function in the main I/O path,
so making this function inline has a significant
performance benefit. A bdevperf microbenchmark
using null bdevs shows a 11% improvement.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I70e30e184000705704bb004e8da1c7476a6aceeb
Reviewed-on: https://review.gerrithub.io/393824
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
While iterating, allow the user to perform asynchronous
operations. To continue iteration, the user is expected
to call spdk_for_each_channel_continue.
Change-Id: Ifd7d03d5fbf17cf13843704274b036d49ca0484a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/391309
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
This function will send a message to each allocated
thread asynchronously, then call a callback on
the originating thread.
Change-Id: I3ebe7c6c5b460a944a32487d1091b601a482a256
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/388041
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Make this generic and not directly dependent on
the event framework. That way our libraries can
register pollers without adding a dependency.
Change-Id: I7ad7a7ddc131596ca1791a7b0f43dabfda050f5f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/387690
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Change the return type of spdk_channel_msg from
void to int. If this msg executed in a failure,
we do not need continue sending the message to other
threads, we can just tell the original thread, and
let the orgiginal thread call the spdk_channel_for_each_cpl
call back.
Thus we can track the qpair creation/destroy case for bdev
reset in nvme bdev module;
Change-Id: Ide9dffd1f84a29fcf61d8339a9ece2a0245d968d
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-on: https://review.gerrithub.io/387284
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is the inverse operation of spdk_io_channel_get_ctx.
Change-Id: Ica6593240dfb05cb53dc7ec910bc0a78270e81c0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/387679
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fixes condition where blobstore was prematurely calling the
application callback on spdk_bs_unload(), if the application
tries to do something too quickly bad things happen.
To avoid application changes with how the g_devlist_mutex is
held, it is no longer held while calling
_spdk_io_device_attempt_free() because the app unload CB is
called from that function and may want to call
spdk_io_device_unregister() from its unload CB. So the lock
is now held and releases strictly around the list its
protecting which allows the CB from _spdk_io_device_attempt_free()
to be called without issue.
Change-Id: Ib451cfe6b33ea0c3f9e66c86785316f9d88837c7
Signed-off-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/377872
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
A channel may have been deleted between when a message was sent to
a thread and when it gets executed on that thread. So we must look
for the channel when the messages gets executed - if it's not found,
just continue to the next thread(s).
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib7e596f11f287c6be521ba729d33378f0a825c7e
Reviewed-on: https://review.gerrithub.io/378002
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Some callers may not require a callback function once
the specified function has been called on all channels.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I925751d221918810c2e966640ca636482bf6f866
Reviewed-on: https://review.gerrithub.io/377859
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
This moves the thread name setting code into the generic SPDK thread
setup code, so now all spdk_threads can be named, not just ones created
by the event framework.
Change-Id: I6c824cf4bcf12fe64a8e2fc7cdc2d6c949021e40
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/375220
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Patch afe860ae deferred freeing the io_device. However, for nvme, the
io_device context (spdk_nvme_ctrlr) is still being destructed before
io_channels are destroyed, causing segfaults on hotremove.
This patch defers io_device context destruction and fixes nvme
hotremove.
Fixes: afe860aeb1 ("channel: Correctly defer unregisters if channels exist")
Fixes: 5533c3d208 ("util: defer put_io_channel")
Change-Id: I7af699174cac0c6c6a6faa2cc65418c47347eb9a
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/370459
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Unregistering a device should be deferred if I/O
channels still exist. Those I/O channels are likely
undergoing a deferred unregister themselves.
Change-Id: I67186232a58f212b867f6ef894c3d37aae2a4d53
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/370351
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
As specified in the doc of spdk_for_each_channel:
```
Once 'fn' has been called on each channel, 'cpl' will be called
on the thread that spdk_for_each_channel was initially called from.
```
Fixes: ff87d29cc3 ("io_channel: Add mechanism to call a function on
each channel")
Change-Id: Ic60b061ec402672f510d99697943e96ff9a73417
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/369719
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
With vbdevs, there are many more cases where the last
(or only) channel for a given I/O device may be destroyed
in the context of the poller for that I/O device. To
avoid returning to the poller and having it check
for data on a resource that was just released, instead
defer the actual put_io_channel work via
spdk_thread_send_msg().
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic99776bddbe9a305f221f8c094ea97dce2d6df0b
Reviewed-on: https://review.gerrithub.io/368619
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is faster than spdk_get_thread(), which iterates over all of the
threads under a lock until it finds the current one.
Change-Id: I957a68438efb5b7d632d2b9b1b567fed7714e02a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/365079
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
For a given I/O device, call a user function on the correct
thread for each channel.
Change-Id: I568a443184ac834c80c5e36b80aa9b6f8bb0ac99
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362256
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This serves two purposes. First, some older compiler
chains don't support thread local storage. Second,
we're going to need a way to iterate the list of
threads in the future, so keep them in a list.
Change-Id: I80e709f4665afb03cf4bcf0db19ef8ea353acdc1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362255
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When a thread is registered, the user must provide
a function pointer that can pass a message to that thread.
Change-Id: I743b5e0d6e3b5118c0a68d2fcedbccdd6fb237f9
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362067
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
This will be used in the future to pass a message
to any given thread.
Change-Id: I3be5fe66244e360b7667427647fd8fdede110930
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362066
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This wasn't used anywhere and we currently believe there
are superior software-only techniques for controlling
quality of service.
Change-Id: Icdadd5870ed0629b338c307d2619bbc242c3e7a3
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362065
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is no longer used anywhere. For the places where we previously
used it, we've since found alternate solutions that do not
require it.
Change-Id: I738a80b95ef50348ce1c14969a3812b0a625b3fd
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/362064
Tested-by: <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Enforce exactly one trailing \n, and fix all of the existing cases.
Change-Id: I6218e4700e90aeb647eaee78089530c79993c8c8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The free function is missed for dev in spdk_io_device_
unregister function.
Change-Id: Id212344dfcde2ae4780c631e3443f530ef25cfd1
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Some subsystems may wish to create unique I/O channels
which are not shared across all users of the same I/O
device on the same thread.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3ade3675d57338cf85b6a301285e6f392bd6cd2e
This patch adds a basic framework for creating I/O channels
for I/O devices. An spdk_io_channel represents a one-to-one
mapping between a calling thread (represented by spdk_thread)
and an I/O device that the thread will perform I/O operations
on.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I658ab7f995cc962f4e2a204e058cdd3ad3fd735d
