ivampiresp/Spdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nvmf: document process for offline tracepoint debugging

Browse Source 
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ieb6ff4f13aa6f2d3e6499ebe9365cd169d0618c1

Reviewed-on: https://review.gerrithub.io/424600
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
This commit is contained in:
Jim Harris 2018-09-04 12:07:34 -07:00
parent e8881867f8
commit 6eeb762f4c
2 changed files with 51 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

48
doc/nvmf.md
View File

@ -231,3 +231,51 @@ Disconnect:
~~~{.sh}
nvme disconnect -n "nqn.2016-06.io.spdk:cnode1"
~~~
## Enabling NVMe-oF target tracepoints for offline analysis and debug {#nvmf_trace}
SPDK has a tracing framework for capturing low-level event information at runtime. The NVMe-oF
target is instrumented with tracepoints to enable analysis of both performance and application
crashes. (Note: the SPDK tracing framework should still be considered experimental. Work to
formalize and document the framework is in progress.)
To enable the instrumentation, start the target with the -e parameter:
~~~{.sh}
app/nvmf_tgt/nvmf_tgt -e 0xFFFF
~~~
Information about the shared memory file will appear in the log:
~~~{.sh}
app.c: 527:spdk_app_setup_trace: *NOTICE*: Tracepoint Group Mask 0xFFFF specified.
app.c: 531:spdk_app_setup_trace: *NOTICE*: Use 'spdk_trace -s nvmf -p 24147' to capture a snapshot of events at runtime.
app.c: 533:spdk_app_setup_trace: *NOTICE*: Or copy /dev/shm/nvmf_trace.pid24147 for offline analysis/debug.
~~~
Note that when tracepoints are enabled, the shared memory files are not deleted when the application
exits. This ensures the file can be used for analysis after the applicatione exits. On Linux, the
shared memory files are in /dev/shm, and can be deleted manually to free shm space if needed. A system
reboot will also free all of the /dev/shm files.
The spdk_trace program can be found in the app/trace directory. To analyze the tracepoints on the same
system running the NVMe-oF target, simply execute the command line shown in the log:
~~~{.sh}
app/trace/spdk_trace -s nvmf -p 24147
~~~
To analyze the tracepoints on a different system, first prepare the tracepoint file for transfer. The
tracepoint file can be large, but usually compresses very well. This step can also be used to prepare
a tracepoint file to attach to a GitHub issue for debugging NVMe-oF application crashes.
~~~{.sh}
bzip2 -c /dev/shm/nvmf_trace.pid24147 > /tmp/trace.bz2
~~~
After transferring the /tmp/trace.bz2 tracepoint file to a different system:
~~~{.sh}
bunzip2 /tmp/trace.bz2
app/trace/spdk_trace -f /tmp/trace
~~~

3
lib/event/app.c
View File

@ -529,6 +529,9 @@ spdk_app_setup_trace(struct spdk_app_opts *opts)
opts->name,
opts->shm_id >= 0 ? "-i" : "-p",
opts->shm_id >= 0 ? opts->shm_id : getpid());
#if defined(__linux__)
SPDK_NOTICELOG("Or copy /dev/shm%s for offline analysis/debug.\n", shm_name);
#endif
spdk_trace_set_tpoint_group_mask(tpoint_group_mask);
}
}