From 835494b53a2ddaf02efede53280c455902bd315d Mon Sep 17 00:00:00 2001 From: Krzysztof Karas Date: Fri, 11 Mar 2022 08:29:51 +0000 Subject: [PATCH] docs: update scheduler documentation Updated scheduler documentation with information about thread and core workloads, scheduling process and dynamic scheduler parameters. Added descriptions of new dynamic scheduler parameters in jsonrpc.md. Change-Id: I567efe21c425c13c16235d7976aad6ae961381bb Signed-off-by: Krzysztof Karas Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11886 Reviewed-by: Tomasz Zawadzki Reviewed-by: Jim Harris Community-CI: Broadcom CI Tested-by: SPDK CI Jenkins --- doc/jsonrpc.md | 3 +++ doc/scheduler.md | 33 ++++++++++++++++++++++++++++----- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/doc/jsonrpc.md b/doc/jsonrpc.md index 00f5873da..39945e2b9 100644 --- a/doc/jsonrpc.md +++ b/doc/jsonrpc.md @@ -713,6 +713,9 @@ Name | Optional | Type | Description ----------------------- | -------- | ----------- | ----------- name | Required | string | Name of a scheduler period | Optional | number | Scheduler period +load_limit | Optional | number | Thread load limit in % (dynamic only) +core_limit | Optional | number | Load limit on the core to be considered full (dynamic only) +core_busy | Optional | number | Indicates at what load on core scheduler should move threads to a different core (dynamic only) #### Response diff --git a/doc/scheduler.md b/doc/scheduler.md index 0acba78f7..3090645f9 100644 --- a/doc/scheduler.md +++ b/doc/scheduler.md @@ -48,10 +48,11 @@ core has to be managed by the administrator. ## Scheduler implementations The scheduler in use may be controlled by JSON-RPC. Please use the -[framework_set_scheduler](jsonrpc.md/#rpc_framework_set_scheduler) RPC to -switch between schedulers or change their options. +[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler) RPC to +switch between schedulers or change their options. Currently only dynamic +scheduler supports changing its parameters. -[spdk_top](spdk_top.md#spdk_top) is a useful tool to observe the behavior of +[spdk_top](spdk_top.html#spdk_top) is a useful tool to observe the behavior of schedulers in different scenarios and workloads. ### static [default] @@ -65,11 +66,27 @@ the previous behavior of the SPDK event/application framework. The `dynamic` scheduler is designed for power saving and reduction of CPU utilization, especially in cases where workloads show large variations over -time. +time. In SPDK thread and core workloads are measured in CPU ticks. Those +values are then compared with all the ticks since the last check, which allows +to calculate `busy time`. + +`busy time = busy ticks / (busy tick + idle tick) * 100 %` + +The thread is considered to be active, if its busy time is over the `load limit` +parameter. Active threads are distributed equally among reactors, taking cpu_mask into account. All idle threads are moved to the main core. Once an idle thread becomes -active, it is redistributed again. +active, it is redistributed again. Dynamic scheduler monitors core workloads and +redistributes SPDK threads on cores in a way that none of them is over `core limit`. +In case a core utilization surpasses this threshold, scheduler should move threads +out of it until this condition no longer applies. Cores might also be in overloaded +state, which indicates that moving threads out of this core will not decrease its +utilization under the `core limit` and the threads are unable to process all the I/O +they are capable of, because they share CPU ticks with other threads. The threshold +to decide if a core is overloaded is called `core busy`. Note that threads residing +on an overloaded core will not perform as good as other threads, because the CPU ticks +intended for them are limited by other threads on the same core. When a reactor has no scheduled `spdk_thread`s it is switched into interrupt mode and stops actively polling. After enough threads become active, the @@ -80,3 +97,9 @@ not exceed the sum of all idle threads. When no active threads are present on the main core, the frequency of that CPU core will decrease as the load decreases. All CPU cores corresponding to the other reactors remain at maximum frequency. + +The dynamic scheduler is currently the only one that allows manual setting of +its parameters. + +Current values of scheduler parameters can be displayed by using +[framework_get_scheduler](jsonrpc.html#rpc_framework_get_scheduler) RPC.