diff --git a/doc/nvmf.md b/doc/nvmf.md index 966a0771f..eb1b4697b 100644 --- a/doc/nvmf.md +++ b/doc/nvmf.md @@ -231,3 +231,51 @@ Disconnect: ~~~{.sh} nvme disconnect -n "nqn.2016-06.io.spdk:cnode1" ~~~ + +## Enabling NVMe-oF target tracepoints for offline analysis and debug {#nvmf_trace} + +SPDK has a tracing framework for capturing low-level event information at runtime. The NVMe-oF +target is instrumented with tracepoints to enable analysis of both performance and application +crashes. (Note: the SPDK tracing framework should still be considered experimental. Work to +formalize and document the framework is in progress.) + +To enable the instrumentation, start the target with the -e parameter: + +~~~{.sh} +app/nvmf_tgt/nvmf_tgt -e 0xFFFF +~~~ + +Information about the shared memory file will appear in the log: + +~~~{.sh} +app.c: 527:spdk_app_setup_trace: *NOTICE*: Tracepoint Group Mask 0xFFFF specified. +app.c: 531:spdk_app_setup_trace: *NOTICE*: Use 'spdk_trace -s nvmf -p 24147' to capture a snapshot of events at runtime. +app.c: 533:spdk_app_setup_trace: *NOTICE*: Or copy /dev/shm/nvmf_trace.pid24147 for offline analysis/debug. +~~~ + +Note that when tracepoints are enabled, the shared memory files are not deleted when the application +exits. This ensures the file can be used for analysis after the applicatione exits. On Linux, the +shared memory files are in /dev/shm, and can be deleted manually to free shm space if needed. A system +reboot will also free all of the /dev/shm files. + +The spdk_trace program can be found in the app/trace directory. To analyze the tracepoints on the same +system running the NVMe-oF target, simply execute the command line shown in the log: + +~~~{.sh} +app/trace/spdk_trace -s nvmf -p 24147 +~~~ + +To analyze the tracepoints on a different system, first prepare the tracepoint file for transfer. The +tracepoint file can be large, but usually compresses very well. This step can also be used to prepare +a tracepoint file to attach to a GitHub issue for debugging NVMe-oF application crashes. + +~~~{.sh} +bzip2 -c /dev/shm/nvmf_trace.pid24147 > /tmp/trace.bz2 +~~~ + +After transferring the /tmp/trace.bz2 tracepoint file to a different system: + +~~~{.sh} +bunzip2 /tmp/trace.bz2 +app/trace/spdk_trace -f /tmp/trace +~~~ diff --git a/lib/event/app.c b/lib/event/app.c index 86b735bc3..3ed7f0a17 100644 --- a/lib/event/app.c +++ b/lib/event/app.c @@ -529,6 +529,9 @@ spdk_app_setup_trace(struct spdk_app_opts *opts) opts->name, opts->shm_id >= 0 ? "-i" : "-p", opts->shm_id >= 0 ? opts->shm_id : getpid()); +#if defined(__linux__) + SPDK_NOTICELOG("Or copy /dev/shm%s for offline analysis/debug.\n", shm_name); +#endif spdk_trace_set_tpoint_group_mask(tpoint_group_mask); } }