From b71c31335d2d63a981f5a6f7e759e9e0b643b205 Mon Sep 17 00:00:00 2001 From: Daniel Verkamp Date: Thu, 26 Jan 2017 16:10:25 -0700 Subject: [PATCH] event: move component descriptions to doc/ and rewrite Change-Id: I53b061933319204f9153423fa18532712676287a Signed-off-by: Daniel Verkamp --- doc/Doxyfile | 1 + doc/event/index.md | 71 ++++++++++++++++++++++++++++++++++++++++++++ doc/index.md | 1 + include/spdk/event.h | 44 ++++----------------------- 4 files changed, 79 insertions(+), 38 deletions(-) create mode 100644 doc/event/index.md diff --git a/doc/Doxyfile b/doc/Doxyfile index 56d502b8d..77446c793 100644 --- a/doc/Doxyfile +++ b/doc/Doxyfile @@ -762,6 +762,7 @@ INPUT = ../include/spdk \ index.md \ directory_structure.md \ porting.md \ + event/index.md \ ioat/index.md \ nvme/index.md \ nvme/async_completion.md \ diff --git a/doc/event/index.md b/doc/event/index.md new file mode 100644 index 000000000..ee53b183a --- /dev/null +++ b/doc/event/index.md @@ -0,0 +1,71 @@ +# Event framework {#event} + +SPDK provides a framework for writing asynchronous, polled-mode, shared-nothing server applications. +The event framework is intended to be optional; most other SPDK components are designed to be +integrated into an application without specifically depending on the SPDK event library. +The framework defines several concepts - reactors, events, and pollers - that are described +in the following sections. +The event framework spawns one thread per core (reactor) and connects the threads with +lockless queues. +Messages (events) can then be passed between the threads. +On modern CPU architectures, message passing is often much faster than traditional locking. + +The event framework public interface is defined in spdk/event.h. + +# Event Framework Design Considerations {#event_design} + +Simple server applications can be written in a single-threaded fashion. This allows for +straightforward code that can maintain state without any locking or other synchronization. +However, to scale up (for example, to allow more simultaneous connections), the application may +need to use multiple threads. +In the ideal case where each connection is independent from all other connections, +the application can be scaled by creating additional threads and assigning connections to them +without introducing cross-thread synchronization. +Unfortunately, in many real-world cases, the connections are not entirely independent +and cross-thread shared state is necessary. +SPDK provides an event framework to help solve this problem. + +# SPDK Event Framework Components {#event_components} + +## Events {#event_component_events} + +To accomplish cross-thread communication while minimizing synchronization overhead, +the framework provides message passing in the form of events. +The event framework runs one event loop thread per CPU core. +These threads are called reactors, and their main responsibility is to process incoming events +from a queue. +Each event consists of a bundled function pointer and its arguments, destined for +a particular CPU core. +Events are created using spdk_event_allocate() and executed using spdk_event_call(). +Unlike a thread-per-connection server design, which achieves concurrency by depending on the +operating system to schedule many threads issuing blocking I/O onto a limited number of cores, +the event-driven model requires use of explicitly asynchronous operations to achieve concurrency. +Asynchronous I/O may be issued with a non-blocking function call, and completion is typically +signaled using a callback function. + +## Reactors {#event_component_reactors} + +Each reactor has a lock-free queue for incoming events to that core, and threads from any core +may insert events into the queue of any other core. +The reactor loop running on each core checks for incoming events and executes them in +first-in, first-out order as they are received. +Event functions should never block and should preferably execute very quickly, +since they are called directly from the event loop on the destination core. + +## Pollers {#event_component_pollers} + +The framework also defines another type of function called a poller. +Pollers may be registered with the spdk_poller_register() function. +Pollers, like events, are functions with arguments that can be bundled and sent to a specific +core to be executed. +However, unlike events, pollers are executed repeatedly until unregistered. +The reactor event loop intersperses calls to the pollers with other event processing. +Pollers are intended to poll hardware as a replacement for interrupts. +Normally, pollers are executed on every iteration of the main event loop. +Pollers may also be scheduled to execute periodically on a timer if low latency is not required. + +## Application Framework {#event_component_app} + +The framework itself is bundled into a higher level abstraction called an "app". Once +spdk_app_start() is called, it will block the current thread until the application +terminates by calling spdk_app_stop(). diff --git a/doc/index.md b/doc/index.md index 4f6a29be1..804284be2 100644 --- a/doc/index.md +++ b/doc/index.md @@ -19,6 +19,7 @@ which avoids kernel context switches and eliminates interrupt handling overhead. ## Modules {#modules} +- @ref event - @ref nvme - @ref nvmf - @ref ioat diff --git a/include/spdk/event.h b/include/spdk/event.h index d59882147..c63674236 100644 --- a/include/spdk/event.h +++ b/include/spdk/event.h @@ -31,44 +31,12 @@ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ -/** \file -* Event framework public API. -* -* This is a framework for writing asynchronous, polled-mode, shared-nothing -* server applications. The framework relies on DPDK for much of its underlying -* architecture. The framework defines several concepts - reactors, events, pollers, -* and subsystems - that are described in the following sections. -* -* The framework runs one thread per core (the user provides a core mask), where -* each thread is a tight loop. The threads never block for any reason. These threads -* are called reactors and their main responsibility is to process incoming events -* from a queue. -* -* An event, defined by \ref spdk_event is a bundled function pointer and arguments that -* can be sent to a different core and executed. The function pointer is executed only once, -* and then the entire event is freed. These functions should never block and preferably -* should execute very quickly. Events also have a pointer to a 'next' event that will be -* executed upon completion of the given event, which allows chaining. This is -* very much a simplified version of futures, promises, and continuations designed within -* the constraints of the C programming language. -* -* The framework also defines another type of function called a poller. Pollers are also -* functions with arguments that can be bundled and sent to a different core to be executed, -* but they are instead executed repeatedly on that core until unregistered. The reactor -* will handle interspersing calls to the pollers with other event processing automatically. -* Pollers are intended to poll hardware as a replacement for interrupts and they should not -* generally be used for any other purpose. -* -* The framework also defines an interface for subsystems, which are libraries of code that -* depend on this framework. A library can register itself as a subsystem and provide -* pointers to initialize and destroy itself which will be called at the appropriate time. -* This is purely for sequencing initialization code in a convenient manner within the -* framework. -* -* The framework itself is bundled into a higher level abstraction called an "app". Once -* \ref spdk_app_start is called it will block the current thread until the application -* terminates (by calling \ref spdk_app_stop). -*/ +/** + * \file + * Event framework public API. + * + * See @ref event_components for an overview of the SPDK event framework API. + */ #ifndef SPDK_EVENT_H #define SPDK_EVENT_H