event: move component descriptions to doc/ and rewrite

Change-Id: I53b061933319204f9153423fa18532712676287a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>

doc/Doxyfile

@@ -762,6 +762,7 @@ INPUT                  = ../include/spdk \
                          index.md \
                          directory_structure.md \
                          porting.md \
+                         event/index.md \
                          ioat/index.md \
                          nvme/index.md \
                          nvme/async_completion.md \

doc/event/index.md

@@ -0,0 +1,71 @@
+# Event framework {#event}
+
+SPDK provides a framework for writing asynchronous, polled-mode, shared-nothing server applications.
+The event framework is intended to be optional; most other SPDK components are designed to be
+integrated into an application without specifically depending on the SPDK event library.
+The framework defines several concepts - reactors, events, and pollers - that are described
+in the following sections.
+The event framework spawns one thread per core (reactor) and connects the threads with
+lockless queues.
+Messages (events) can then be passed between the threads.
+On modern CPU architectures, message passing is often much faster than traditional locking.
+
+The event framework public interface is defined in spdk/event.h.
+
+# Event Framework Design Considerations {#event_design}
+
+Simple server applications can be written in a single-threaded fashion. This allows for
+straightforward code that can maintain state without any locking or other synchronization.
+However, to scale up (for example, to allow more simultaneous connections), the application may
+need to use multiple threads.
+In the ideal case where each connection is independent from all other connections,
+the application can be scaled by creating additional threads and assigning connections to them
+without introducing cross-thread synchronization.
+Unfortunately, in many real-world cases, the connections are not entirely independent
+and cross-thread shared state is necessary.
+SPDK provides an event framework to help solve this problem.
+
+# SPDK Event Framework Components {#event_components}
+
+## Events {#event_component_events}
+
+To accomplish cross-thread communication while minimizing synchronization overhead,
+the framework provides message passing in the form of events.
+The event framework runs one event loop thread per CPU core.
+These threads are called reactors, and their main responsibility is to process incoming events
+from a queue.
+Each event consists of a bundled function pointer and its arguments, destined for
+a particular CPU core.
+Events are created using spdk_event_allocate() and executed using spdk_event_call().
+Unlike a thread-per-connection server design, which achieves concurrency by depending on the
+operating system to schedule many threads issuing blocking I/O onto a limited number of cores,
+the event-driven model requires use of explicitly asynchronous operations to achieve concurrency.
+Asynchronous I/O may be issued with a non-blocking function call, and completion is typically
+signaled using a callback function.
+
+## Reactors {#event_component_reactors}
+
+Each reactor has a lock-free queue for incoming events to that core, and threads from any core
+may insert events into the queue of any other core.
+The reactor loop running on each core checks for incoming events and executes them in
+first-in, first-out order as they are received.
+Event functions should never block and should preferably execute very quickly,
+since they are called directly from the event loop on the destination core.
+
+## Pollers {#event_component_pollers}
+
+The framework also defines another type of function called a poller.
+Pollers may be registered with the spdk_poller_register() function.
+Pollers, like events, are functions with arguments that can be bundled and sent to a specific
+core to be executed.
+However, unlike events, pollers are executed repeatedly until unregistered.
+The reactor event loop intersperses calls to the pollers with other event processing.
+Pollers are intended to poll hardware as a replacement for interrupts.
+Normally, pollers are executed on every iteration of the main event loop.
+Pollers may also be scheduled to execute periodically on a timer if low latency is not required.
+
+## Application Framework {#event_component_app}
+
+The framework itself is bundled into a higher level abstraction called an "app". Once
+spdk_app_start() is called, it will block the current thread until the application
+terminates by calling spdk_app_stop().

doc/index.md

@@ -19,6 +19,7 @@ which avoids kernel context switches and eliminates interrupt handling overhead.
 
 ## Modules {#modules}
 
+- @ref event
 - @ref nvme
 - @ref nvmf
 - @ref ioat

include/spdk/event.h

@@ -31,43 +31,11 @@
  *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
-/** \file
+/**
+ * \file
  * Event framework public API.
  *
-* This is a framework for writing asynchronous, polled-mode, shared-nothing
+ * See @ref event_components for an overview of the SPDK event framework API.
-* server applications. The framework relies on DPDK for much of its underlying
-* architecture. The framework defines several concepts - reactors, events, pollers,
-* and subsystems - that are described in the following sections.
-*
-* The framework runs one thread per core (the user provides a core mask), where
-* each thread is a tight loop. The threads never block for any reason. These threads
-* are called reactors and their main responsibility is to process incoming events
-* from a queue.
-*
-* An event, defined by \ref spdk_event is a bundled function pointer and arguments that
-* can be sent to a different core and executed. The function pointer is executed only once,
-* and then the entire event is freed. These functions should never block and preferably
-* should execute very quickly. Events also have a pointer to a 'next' event that will be
-* executed upon completion of the given event, which allows chaining. This is
-* very much a simplified version of futures, promises, and continuations designed within
-* the constraints of the C programming language.
-*
-* The framework also defines another type of function called a poller. Pollers are also
-* functions with arguments that can be bundled and sent to a different core to be executed,
-* but they are instead executed repeatedly on that core until unregistered. The reactor
-* will handle interspersing calls to the pollers with other event processing automatically.
-* Pollers are intended to poll hardware as a replacement for interrupts and they should not
-* generally be used for any other purpose.
-*
-* The framework also defines an interface for subsystems, which are libraries of code that
-* depend on this framework. A library can register itself as a subsystem and provide
-* pointers to initialize and destroy itself which will be called at the appropriate time.
-* This is purely for sequencing initialization code in a convenient manner within the
-* framework.
-*
-* The framework itself is bundled into a higher level abstraction called an "app". Once
-* \ref spdk_app_start is called it will block the current thread until the application
-* terminates (by calling \ref spdk_app_stop).
  */
 
 #ifndef SPDK_EVENT_H