doc: Finish overview page

This has been on the TODO list for a long time

Change-Id: Ibdd130f976e9f4436a9dc94919878e2ff187e873
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453452
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

doc/overview.md

@@ -2,32 +2,47 @@
 
 # Overview {#dir_overview}
 
-SPDK is primarily a collection of C libraries intended to be consumed directly by
+SPDK is composed of a set of C libraries residing in `lib` with public interface
-applications, but the repository also contains many examples and full-fledged applications.
+header files in `include/spdk`, plus a set of applications built out of those
-This will provide a general overview of what is where in the repository.
+libraries in `app`. Users can use the C libraries in their software or deploy
+the full SPDK applications.
+
+SPDK is designed around message passing instead of locking, and most of the SPDK
+libraries make several assumptions about the underlying threading model of the
+application they are embedded into. However, SPDK goes to great lengths to remain
+agnostic to the specific message passing, event, co-routine, or light-weight
+threading framework actually in use. To accomplish this, all SPDK libraries
+interact with an abstraction library in `lib/thread` (public interface at
+`include/spdk/thread.h`). Any framework can initialize the threading abstraction
+and provide callbacks to implement the functionality that the SPDK libraries
+need. For more information on this abstraction, see @ref concurrency.
+
+SPDK is built on top of POSIX for most operations. To make porting to non-POSIX
+environments easier, all POSIX headers are isolated into
+`include/spdk/stdinc.h`. However, SPDK requires a number of operations that
+POSIX does not provide, such as enumerating the PCI devices on the system or
+allocating memory that is safe for DMA. These additional operations are all
+abstracted in a library called `env` whose public header is at
+`include/spdk/env.h`. By default, SPDK implements the `env` interface using a
+library based on DPDK. However, that implementation can be swapped out. See @ref
+porting for additional information.
 
 ## Applications {#dir_app}
 
-The `app` top-level directory contains four applications:
+The `app` top-level directory contains full-fledged applications, built out of the SPDK
- - `app/iscsi_tgt`: An iSCSI target
+components. For a full overview, see @ref app_overview.
- - `app/nvmf_tgt`: An NVMe-oF target
- - `app/iscsi_top`: Informational tool (like `top`) that tracks activity in the
-    iSCSI target.
- - `app/trace`: A tool for processing trace points output from the iSCSI and
-    NVMe-oF targets.
- - `app/vhost`:  A vhost application that presents virtio controllers to
-    QEMU-based VMs and process I/O submitted to those controllers.
 
-The application binaries will be in their respective directories after compiling and all
+SPDK applications can typically be started with a small number of configuration
-can be run with no arguments to print out their command line arguments. For the iSCSI
+options. Full configuration of the applications is then performed using
-and NVMe-oF targets, they both need a configuration file (-c option). Fully commented
+JSON-RPC. See @ref jsonrpc for additional information.
-examples of the configuration files live in the `etc/spdk` directory.
 
-## Build Collateral {#dir_build}
+## Libraries {#dir_lib}
 
-The `build` directory contains all of the static libraries constructed during
+The `lib` directory contains the real heart of SPDK. Each component is a C library with
-the build process. The `lib` directory combined with the `include/spdk`
+its own directory under `lib`. Some of the key libraries are:
-directory are the official outputs of an SPDK release, if it were to be packaged.
+
+- @ref bdev
+- @ref nvme
 
 ## Documentation {#dir_doc}
 
@@ -59,8 +74,8 @@ directory and include the headers by prefixing `spdk/` like this:
 #include "spdk/nvme.h"
 ~~~
 
-Most of the headers here correspond with a library in the `lib` directory and will be
+Most of the headers here correspond with a library in the `lib` directory. There
-covered in that section. There are a few headers that stand alone, however. They are:
+are a few headers that stand alone, however. They are:
 
  - `assert.h`
  - `barrier.h`
@@ -74,41 +89,6 @@ There is also an `spdk_internal` directory that contains header files widely inc
 by libraries within SPDK, but that are not part of the public API and would not be
 installed on a user's system.
 
-## Libraries {#dir_lib}
-
-The `lib` directory contains the real heart of SPDK. Each component is a C library with
-its own directory under `lib`.
-
-### Block Device Abstraction Layer {#dir_bdev}
-
-The `bdev` directory contains a block device abstraction layer that is currently used
-within the iSCSI and NVMe-oF targets. The public interface is `include/spdk/bdev.h`.
-This library lacks clearly defined responsibilities as of this writing and instead does a
-number of
-things:
- - Translates from a common `block` protocol to specific protocols like NVMe or to system
-  calls like libaio. There are currently three block device backend modules that can be
-  plugged in - libaio, SPDK NVMe, CephRBD, and a RAM-based backend called malloc.
- - Provides a mechanism for composing virtual block devices from physical devices (to do
-  RAID and the like).
- - Handles some memory allocation for data buffers.
-
-This layer also could be made to do I/O queueing or splitting in a general way. We're open
-to design ideas and discussion here.
-
-### Configuration File Parser {#dir_conf}
-
-The `conf` directory contains configuration file parser. The public header
-is `include/spdk/conf.h`. The configuration file format is kind of like INI,
-except that the directives are are "Name Value" instead of "Name = Value". This is
-the configuration format for both the iSCSI and NVMe-oF targets.
-
-... Lots more libraries that need to be described ...
-
-## Makefile Fragments {#dir_mk}
-
-The `mk` directory contains a number of shared Makefile fragments used in the build system.
-
 ## Scripts {#dir_scripts}
 
 The `scripts` directory contains convenient scripts for a number of operations. The two most