doc: Integrate concurrency and event framework docs

Make these two documents link to one another.

Also do the following:

1) Make "Event Framework" a Programmer Guide
2) Update some of the text about pollers to reflect recent changes.

Change-Id: I3dfbcde0dadcf69b7c165f7bad5bee00d3c10d1f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/397633
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

doc/concurrency.md

@@ -108,17 +108,16 @@ large portion of the code in each was implementing the basic message passing
 infrastructure required to call spdk_allocate_thread(). This includes spawning
 one thread per core, pinning each thread to a unique core, and allocating
 lockless rings between the threads for message passing. Instead of
-re-implementing that infrastructure for each example application, SPDK provides
+re-implementing that infrastructure for each example application, SPDK
-the SPDK [event framework](http://www.spdk.io/doc/event_8h.html). This library
+provides the SPDK @ref event. This library handles setting up all of the
-handles setting up all of the message passing infrastructure, installing signal
+message passing infrastructure, installing signal handlers to cleanly
-handlers to cleanly shutdown, implements periodic pollers, and does basic
+shutdown, implements periodic pollers, and does basic command line parsing.
-command line parsing. When started through spdk_app_start(), the library
+When started through spdk_app_start(), the library automatically spawns all of
-automatically spawns all of the threads requested, pins them, and calls
+the threads requested, pins them, and calls spdk_allocate_thread() with
-spdk_allocate_thread() with appropriate function pointers for each one. This
+appropriate function pointers for each one. This makes it much easier to
-makes it much easier to implement a brand new SPDK application and is the
+implement a brand new SPDK application and is the recommended method for those
-recommended method for those starting out. Only established applications with
+starting out. Only established applications with sufficient message passing
-sufficient message passing infrastructure should consider directly integrating
+infrastructure should consider directly integrating the lower level libraries.
-the lower level libraries.
 
 # Limitations of the C Language
 

doc/event.md

@@ -1,68 +1,70 @@
-# Event framework {#event}
+# Event Framework {#event}
 
-SPDK provides a framework for writing asynchronous, polled-mode, shared-nothing server applications.
+SPDK provides a framework for writing asynchronous, polled-mode,
-The event framework is intended to be optional; most other SPDK components are designed to be
+shared-nothing server applications. The event framework is intended to be
-integrated into an application without specifically depending on the SPDK event library.
+optional; most other SPDK components are designed to be integrated into an
-The framework defines several concepts - reactors, events, and pollers - that are described
+application without specifically depending on the SPDK event library. The
-in the following sections.
+framework defines several concepts - reactors, events, and pollers - that are
-The event framework spawns one thread per core (reactor) and connects the threads with
+described in the following sections. The event framework spawns one thread per
-lockless queues.
+core (reactor) and connects the threads with lockless queues. Messages
-Messages (events) can then be passed between the threads.
+(events) can then be passed between the threads. On modern CPU architectures,
-On modern CPU architectures, message passing is often much faster than traditional locking.
+message passing is often much faster than traditional locking. For a
+discussion of the theoretical underpinnings of this framework, see @ref
+concurrency.
 
-The event framework public interface is defined in spdk/event.h.
+The event framework public interface is defined in event.h.
 
 # Event Framework Design Considerations {#event_design}
 
-Simple server applications can be written in a single-threaded fashion. This allows for
+Simple server applications can be written in a single-threaded fashion. This
-straightforward code that can maintain state without any locking or other synchronization.
+allows for straightforward code that can maintain state without any locking or
-However, to scale up (for example, to allow more simultaneous connections), the application may
+other synchronization. However, to scale up (for example, to allow more
-need to use multiple threads.
+simultaneous connections), the application may need to use multiple threads.
-In the ideal case where each connection is independent from all other connections,
+In the ideal case where each connection is independent from all other
-the application can be scaled by creating additional threads and assigning connections to them
+connections, the application can be scaled by creating additional threads and
-without introducing cross-thread synchronization.
+assigning connections to them without introducing cross-thread
-Unfortunately, in many real-world cases, the connections are not entirely independent
+synchronization. Unfortunately, in many real-world cases, the connections are
-and cross-thread shared state is necessary.
+not entirely independent and cross-thread shared state is necessary. SPDK
-SPDK provides an event framework to help solve this problem.
+provides an event framework to help solve this problem.
 
 # SPDK Event Framework Components {#event_components}
 
 ## Events {#event_component_events}
 
-To accomplish cross-thread communication while minimizing synchronization overhead,
+To accomplish cross-thread communication while minimizing synchronization
-the framework provides message passing in the form of events.
+overhead, the framework provides message passing in the form of events. The
-The event framework runs one event loop thread per CPU core.
+event framework runs one event loop thread per CPU core. These threads are
-These threads are called reactors, and their main responsibility is to process incoming events
+called reactors, and their main responsibility is to process incoming events
-from a queue.
+from a queue. Each event consists of a bundled function pointer and its
-Each event consists of a bundled function pointer and its arguments, destined for
+arguments, destined for a particular CPU core. Events are created using
-a particular CPU core.
+spdk_event_allocate() and executed using spdk_event_call(). Unlike a
-Events are created using spdk_event_allocate() and executed using spdk_event_call().
+thread-per-connection server design, which achieves concurrency by depending
-Unlike a thread-per-connection server design, which achieves concurrency by depending on the
+on the operating system to schedule many threads issuing blocking I/O onto a
-operating system to schedule many threads issuing blocking I/O onto a limited number of cores,
+limited number of cores, the event-driven model requires use of explicitly
-the event-driven model requires use of explicitly asynchronous operations to achieve concurrency.
+asynchronous operations to achieve concurrency. Asynchronous I/O may be issued
-Asynchronous I/O may be issued with a non-blocking function call, and completion is typically
+with a non-blocking function call, and completion is typically signaled using
-signaled using a callback function.
+a callback function.
 
 ## Reactors {#event_component_reactors}
 
-Each reactor has a lock-free queue for incoming events to that core, and threads from any core
+Each reactor has a lock-free queue for incoming events to that core, and
-may insert events into the queue of any other core.
+threads from any core may insert events into the queue of any other core. The
-The reactor loop running on each core checks for incoming events and executes them in
+reactor loop running on each core checks for incoming events and executes them
-first-in, first-out order as they are received.
+in first-in, first-out order as they are received. Event functions should
-Event functions should never block and should preferably execute very quickly,
+never block and should preferably execute very quickly, since they are called
-since they are called directly from the event loop on the destination core.
+directly from the event loop on the destination core.
 
 ## Pollers {#event_component_pollers}
 
-The framework also defines another type of function called a poller.
+The framework also defines another type of function called a poller. Pollers
-Pollers may be registered with the spdk_poller_register() function.
+may be registered with the spdk_poller_register() function. Pollers, like
-Pollers, like events, are functions with arguments that can be bundled and sent to a specific
+events, are functions with arguments that can be bundled and executed.
-core to be executed.
+However, unlike events, pollers are executed repeatedly until unregistered and
-However, unlike events, pollers are executed repeatedly until unregistered.
+are executed on the thread they are registered on. The reactor event loop
-The reactor event loop intersperses calls to the pollers with other event processing.
+intersperses calls to the pollers with other event processing. Pollers are
-Pollers are intended to poll hardware as a replacement for interrupts.
+intended to poll hardware as a replacement for interrupts. Normally, pollers
-Normally, pollers are executed on every iteration of the main event loop.
+are executed on every iteration of the main event loop. Pollers may also be
-Pollers may also be scheduled to execute periodically on a timer if low latency is not required.
+scheduled to execute periodically on a timer if low latency is not required.
 
 ## Application Framework {#event_component_app}
 

doc/index.md

@@ -29,10 +29,11 @@
 - @ref bdev_module
 - @ref directory_structure
 - [Public API header files](files.html)
+- @ref event
 
 # Modules {#modules}
 
-- @ref event
+
 - @ref nvme
 - @ref nvmf
 - @ref ioat