Users can now specify a number of small/large buffers to be cached on
each iobuf channel. Previously, we relied on the cache of the
underlying spdk_mempool, which has per-core caches. However, since iobuf
channels are tied to a module and an SPDK thread, each module and each
thread is now guaranteed to have a number of buffers available, so it
won't be starved by other modules/threads.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I1e29fe29f78a13de371ab21d3e40bf55fbc9c639
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15634
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
The idea behind "iobuf" is to have a single place for allocating data
buffers across different libraries. That way, each library won't need
to allocate its own mempools, therefore decreasing the memory footprint
of the whole application.
There are two reasons for putting these kind of functions in the thread
library. Firstly, the code is pretty small, so it doesn't make sense to
create a new library. Secondly, it relies on the IO channel abstraction,
so users will need to pull in the thread library anyway.
It's very much inspired by the way bdev layer handles data buffers (much
of the code was directly copied over). There are two global mempools,
one for small and one for large buffers, and per-thread queues that hold
requests waiting for a buffer. The main difference is that we also need
to track which module requested a buffer in order to allow users to
iterate over its pending requests.
The usage is fairly simple:
```
/* Embed spdk_iobuf_channel into an existing IO channel */
struct foo_channel {
...
struct spdk_iobuf_channel iobuf;
};
/* Embed spdk_iobuf_entry into objects that will request buffers */
struct foo_object {
...
struct spdk_iobuf_entry entry;
};
/* Register the module as iobuf user */
spdk_iobuf_register_module("foo");
/* Initialize iobuf channel in foo_channel's create cb */
spdk_iobuf_channel_init(&foo_channel->iobuf, "foo", 0, 0);
/* Finally, request a buffer... */
buf = spdk_iobuf_get(&foo_channel->iobuf, length,
&foo_objet.entry, buf_get_cb);
...
/* ...and release it */
spdk_iobuf_put(&foo_channel->iobuf, buf, length);
```
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifaa6934c03ed6587ddba972198e606921bd85008
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15326
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This exercises the parts of spdk_spin_*() that are difficult to test in
unit tests. In particular, it tests multiple SPDK threads running on
different pthreads contending for a lock and it tests pollers and
messages going off CPU with a lock held.
Change-Id: I5cd6ce29c92c44ba63f47332fe339e59eed81553
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15534
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
This introduces an enhanced spinlock that adds safeguards compared to
the default pthread_spinlock_t. In particular:
- A pthread_spinlock_t is still used, but additional error checking is
performed to ensure there is no undefined behavior on relock,
unlocking when not the owner, or destoying a locked lock.
- The SPDK concurrency model allows an SPDK thread to be migrated
between pthreads. Releasing a pthread spinlock on a different thread
from where it is taken is undefined behavior. If an SPDK spinlock is
held at a time that a time when a poller or message returns control to
thread_poll(), the program will abort.
- SPDK spinlocks can only be obtained from an SPDK thread.
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I6dd6493ab5f5532ae69e20654546405a507eb594
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15277
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
If the calloc failed, the fd was left in the fd_group.
Change-Id: Ie68426a13d342756c20315656f0309440fda6e02
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15475
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: John Levon <levon@movementarian.org>
This function can be useful to query if a thread
had spdk_thread_exit() called on it yet. Internally
we have both EXITING and EXITED state - so
!spdk_thread_is_running() can be used to detect a
thread that is in either of those states.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2f6fb024a6b1bc895fdc5132c722abc10f5d30f9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15512
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
At end of spdk_thread_poll(), if thread is in EXITING state,
we call thread_exit() to see if the thread can move to
EXITED state. If there are any pollers, io_channels
or pending device unregistrations in progress, thread_exit()
will keep the thread in EXITING mode for this iteration.
But a thread may post messages to itself during this cleanup
process, so thread_exit() should also check if there are
any messages on its queue.
Found during testing of spdk_thread lifetime patch set.
rbd bdev module will send messages to itself like this
during cleanup. Without this change, rbd module testing
with bdevperf would cause an spdk_thread to move to
EXITED state prematurely.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie611026a67b7fa48640ae83be03e29a9c64883a2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15533
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The "app thread" will always be the first
thread created using spdk_thread_create(). There
are many operations throughout SPDK that implicitly
expect to happen in the context of this app thread,
so by formalizing it we can start to make assertions
on this to help clarify and simplify locking and
synchronization through the code base.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7133b58c311710f1d132ee5f09500ffeb4168b15
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15497
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Next patch will add a new caller to this function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I54374c0af3a4a0fdcc5ac9ca25e2c7ef03e99829
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15576
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.
git log --follow -C90% --format=%ad --date default <file> | tail -1
and then pull just the 4 digit year from the result.
Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc). Contribution date used
"--follow -C95%" to get the most accurate date.
Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line. Simply there for
consistency.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
For example, in the calling from spdk_bdev_get_current_qd(), if
spdk_for_each_channel() failed to allocate struct spdk_io_channel_iter,
it will just return and the ctx allocated in spdk_bdev_get_current_qd()
is not released.
Instead to change the public API of spdk_for_each_channel() to return
the failed status to let the caller properly handle the NOMEM case and
release the allocation, it just adds the assert here.
Change-Id: I6a95207dd390586bdae4e86e5d550cdac709e10a
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14657
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
There may be for_each operations outstanding on an
io_device when it is unregistered. Currently we just
return when this happens, not unregistering the
device but also not notifying the caller that this
happened (since it returns void, and the callback
function doesn't have a status parameter either).
We could just push this responsibility to the caller,
to never unregister an io_device if it knows it has
outstanding for_each calls waiting to complete. But
I think we can simplify this a lot by just handling
this inside of the thread library. Mark that the
device is pending registration, and unregister it
(on the original requesting thread!) when the
for_each count gets back to zero. Also don't
allow any new for_each operations either.
Note this requires a bit of refactoring on the
thread unit tests, since it is now possible to
unregister a device with outstanding for_each
operations.
Fixes issue #2631.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I978f2d99a25e65d2b7d71ce9b1926a79a6c94263
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13890
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
If spdk_for_each_channel is called on a device that doesn't
exist, we need to set a non-zero status (-ENODEV in this
case) to the completion function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I898ad5ea499fb6087338b621b2befcadd6a05414
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13889
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Here should return -ENOMEM, and other places are
changed.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: Id81cd7485733e66d996b1501061a45f774f2b51a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13863
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
spdk in interrupt, reactor dosen't correctly handle exited threads,
causing vhost threads still in reactor's lw_threads list. The fix
will do cleanup thread when it's state becomes EXITED. Though it's
exposed in v22.05.x, but the master branch also has the problem.
We will do this as below:
(1) When thread's state becomes SPDK_THREAD_STATE_EXITED, reactor
process thread exits first.
(2) Then reactor do remove lw_thread and destroy it.
Fix issue: #2574
Signed-off-by: Apokleos <oliverliyn@gmail.com>
Change-Id: I3ac2681d70480563db3a0aee4aff61c2f272b140
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13706
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
the underlying spdk_cpuset_copy() takes `const spdk_cpuset*` as the
`src` parameter. there is no need to take non-const spdk_cpuset*.
hence, in this change, let's relax the requirement of the pointer type.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Change-Id: I1f626c7fea45cf7250bf56b891bcba4a0f2a8917
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13443
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in every source file. This patch replaces
the bulk of SPDK .c, .cpp and Makefiles with the BSD-3-Clause
identifier.
Almost all of these files share the exact same license text,
and this patch only modifies the files that contain the
most common license text. There can be slight variations
because the third clause contains company names - most say
"Intel Corporation", but there are instances for Nvidia,
Samsung, Eideticom and even "the copyright holder".
Used a bash script to automate replacement of the license text
with SPDX identifier which is checked into scripts/spdx.sh.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaa88ab5e92ea471691dc298cfe41ebfb5d169780
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12904
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: <qun.wan@intel.com>
With these we can write a simple bpftrace script to identify work being done,
and in particular what woke us up from sleeping.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I8997d847625ee4558092dbd753e6fc1b17beca92
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9424
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The spdk_msg_mempool structure has a fixed size, which is not flexible
enough. The size of the memory allocation can now be changed in the
options given to the spdk_app_start function.
Signed-off-by: Alexis Lescouet <alexis.lescouet@nutanix.com>
Change-Id: I1d6524ab8cf23f69f553aedb0f5b0cdc9dde374b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11635
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Since there is a pthread_mutex_unlock() in normal condition, another
pthread_mutex_unlock() in the "tmp != NULL" branch should be removed,
otherwise will cause a double unlock.
Fixes issue #2378.
Signed-off-by: Weiguo Li <liwg06@foxmail.com>
Change-Id: I6c80a9527dd60e0b7c1d3c54b6da371b31118f02
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11642
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These tracepoints don't include this parameter in their definitions.
This patch fixes the following assertion when the traces are enabled in
the thread library:
```
_spdk_trace_record: Assertion `0 && "Unexpected number of tracepoint arguments"' failed
```
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I5159dbafd25c3150c90fa26c966dadb1fe239953
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11159
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Add/modify tpoints around io_device name in lib/bdev/bdev.c
and lib/thread/thread.c.
Deleted double spaces in commets of trace_defs.h.
Change-Id: I0e2f5118e68b1b329a422bde3400fd2273e7387e
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10687
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Issue: spdk_top tracked pollers by the poller name string and the
thread_id they are running on. This shows incorrect stats when
multiple pollers exist on the same thread with the same name.
Solution: Added a unique poller id for each poller on a thread and
to allow spdk_top to track pollers by thread_id and poller_id.
Signed-off-by: Michael Piszczek <mpiszczek@ddn.com>
Change-Id: I1879e2afc9a929d1df9e8e35510f0092c5443bdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5868
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
When intr mode is enabled, it will be common that
poller is unregistered during interrupt processing.
Since poller unregister is a delayed operation,
mark it in spdk_thread object, and reap unregistered
pollers out of poller execution.
Fixes#2143
Change-Id: Ieb61fc7685f85af5c15e833dd1dd56f8c97a3b12
Signed-off-by: Liu Xiaodong <xiaodong.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5770
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This is to help with binding trace objects together and
for the convenience (all trace definitions are in one place
instad of being scattered accross multiple files).
Change-Id: Ib15bc9c2eeee9c4d0816bcee509ab69f3f558e19
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9574
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
For debugging purposes, take a name for identifying fds added to a group.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If1654e56e19f7fa964446ef1b9e71debf74979d1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9731
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
POSIX defines PRId64/PRIu64/PRIx64 for printing 64-bit values in a
portable way. Replace a reference to %lu to remove the assumption
about the size of a long.
Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Change-Id: I622fd43e7acf2cb93d3ba4ba9e9367e6dd064a74
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9663
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
We report other errors in spdk_interrupt_register(), so add one if there is an
issue with adding the given efd to the fd group.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Idcd810d2b3f644bab1514063b399b9767797130f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9388
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Use the macros for red black tree provided by Free BSD to speed up
io_channel lookup.
Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Change-Id: Icfd87a8a2f60c082a17b8c501a03faba83edb762
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7895
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: GangCao <gang.cao@intel.com>
In current implementation, io_channel list will be accessed by
spdk_for_each_channel() and spdk_get_io_channel(). We will try to
accelerate spdk_get_io_channel() in the following change "thread: speed
up io_channel lookup by using rbtree" by changing io_channel from list
into RB tree.
To make it cleaner, we prefer to use ch->dev as the key for the
io_channel RB tree instead of ch->dev->io_device. This patch makes
spdk_for_each_channel() use the i->dev to find the expected io_channel.
And the io_device in structure spdk_io_channel_iter is not needed in
spdk_for_each_channel_continue() but we keep it for the compatibility of
spdk_io_channel_iter_get_io_device().
After this patch, spdk_for_each_channel() has to access both io_device
list and io_channel list, and spdk_for_each_channel_continue() still has
to access only io_channel list.
Both io_device list and io_channel list will become RB tree. Hence
performance degradation will be negligible. spdk_for_each_channel() is
not so performance critical than spdk_get_io_channel().
Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Change-Id: Idd486b0aa1b63b57ede90527dcd1631cbb008a1a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8749
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Use the macros for red black tree provided by Free BSD to speed up
io_device lookup.
This change was reverted once but is re-submitted because the critical
issue was fixed by the preceding patches.
In addition to the fix, add unit tests to verify the fix explicitly.
Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I97ed77f6e5ceacdf2593c9751b55a7d0b92c0b35
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8525
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Move the definition of structure spdk_io_channel into
lib/thread/thread_internal.h, so we don't have to update SO_VER for
other libraries in future when we need to change the internal details on
the structure.
Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Change-Id: I3d2ca7a8737972e0b33ce92e464da42c48f89dec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8189
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The traces record calls to spdk_(get|put)_io_channel() and saves the
reference count of the IO channel and its context. The context, instead
of an IO channel pointer, was selected because the same pointer is often
used in other traces (e.g. nvmf's poll group), so it makes it possible
to match these traces together.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I15fe982a89685d8f6e23d406d6d48f5c2d9d604b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7232
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This reverts commit 2246a93718.
We are seeing a lot of failure on io_device lookup in the test
pool. These only showed up after this patch was merged and sees
the most likely culprit. Reverting this patch for now while we
continue debug.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2ab098319dfae3a5356eb4fe0dbf9f4af2d2eea5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8199
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Use the macros for red black tree provided by Free BSD to speed up
io_device lookup.
Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Change-Id: Ib3bd382bbeb610503194e7d7bfd569f60a0d0121
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7894
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
(cd83ea4a)thread: Add SPDK internal APIs spdk_thread_get_first/next_active/timed/paused_poller()
Patch above by mistake iterates over active_pollers list
for function that lists paused pollers.
Fixes#1947
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1b69d942675f34f5f046ec46feacc8d81d89f015
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7952
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: <dongx.yi@intel.com>
Use the macros for red black tree provided by Free BSD to manage
timed pollers efficiently.
Allow RB_INSERT() to insert elements with duplicated keys by changing
the compare function to return 1 if two keys are equal.
Check the return code of RB_INSERT() because this is the first use case
for RB tree macros in SPDK. We did the same for RB_REMOVE() by
adding another temporary variable but we remove it from this patch
because it is not so important compared with RB_INSERT().
When a timed poller is inserted, update the cache for the closest (leftmost)
timed poller only if the tree was empty before or the closest (leftmost)
timed poller was actually changed. We do not have to use RB_MIN()
because all duplicated entries are inserted on the right side.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Ibe253ca8eecc10116548b5eedbcdba8fb961b88d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7722
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
We already hold thehe next closest timed poller in tmp. Inlining
poller_remove_timer() into thread_poll() makes the cache update
more efficient.
After this patch, poller_remove_timer() is called only in a single case
and the case is compiled only on Linux. So add it inside of a temporary
block is much clearner. However it will be used by spdk_poller_reschedule()
in the end of this patch series. So keep
the current position.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I2e6858223713eed84f5d70b160da6122edae6d03
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7910
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
This enable us to optimize the cache update when RB tree is supported.
Call poller_remove_timer() after getting the next element because
as TAILQ_FOREACH_SAFE() and RB_FOREACH_SAFE() do, TAILQ_NEXT() may
not be valid after the current element is removed.
Previously, the patch had called poller_remove_timer() before getting
the next element. However, thanks to the nice testing, this bug was
found.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I18afb4412115dc1696cc568610cbe3dc618c2357
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7909
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
This change is a preparation to first dequeue the closest timed poller
always when it is expired. Previously the poller_remove_timer() calls
were not consistent and difficult to follow.
spdk_poller_pause() sets poller to PAUSING even when it in RUNNING
and move it to PAUSED after returning from its context.
If spdk_poller_pause() and spdk_poller_resume() are called while poller
runs, it is moved to WAITING. Hence thread_execute_poller() and
thread_execute_timed_poller() ignore such cases.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I29340613a2ec0c3529d0886f4d81c0a0fdf8745d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7908
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When we introduce RB tree, getting the closest timed poller is not
O(1) but O(log N). To mitigate such delay, cache the closest timed
poller into thread, and update the cache when its content is changed.
Add unit test cases for this change. They will also clarify the current
behavior of spdk_poller_unregister() and spdk_poller_pause() for
timed pollers.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Ibb98a54c261859a3210034038d3953e5c93ef8aa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7720
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Community-CI: Mellanox Build Bot
This is the same effort as spdk_poller.
The following patches will move the definition of struct spdk_thread and
enum spdk_thread_state from include/spdk_internal/thread.h to
lib/thread/thread.c.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I7f7bdfdd7a7b1b834d16d79638a4fd2d63e9daf6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7800
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Most accesses to the struct spdk_poller outside lib/thread have been
done via functions but a few direct accesses remain.
Change these to indirect accesses by addinng a few helper functions
as SPDK internal APIs.
Add spdk_poller_get_name() to get the name of the poller.
Remove spdk_poller_state_str() and add spdk_poller_get_state_str().
Exposing enum spdk_poller_state outside lib/thread is not really
necessary.
This removal requires us to update major SO version.
Add spdk_poller_get_period_ticks() to get the period ticks of the
poller.
Add struct spdk_poller_stats and spdk_poller_get_stats() to get
the stats of the poller.
The next patch will move the definition of struct spdk_poller and
enum spdk_poller_state from include/spdk_internal/thread.h to
lib/thread/thread.c.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Id597dae074a15fcd8af09fd9d416a22ce2f403c3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7798
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
The following patches will update the cache to the closest timed
poller when removing it from the list. To do it easier, factor out
the operation to remove a timed poller from timed_pollers list into
a helper function.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I25016d86117b240a2651d1f06e23bea0342211f1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7719
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
