ivampiresp/Spdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nvme: add NVME_QUIRK_DELAY_BEFORE_INIT quirk

Browse Source 
Currently we *always* wait 2 seconds before starting
controller initialization during attach.  This
works around an issue where some older Intel NVMe SSDs
could not handle MMIO writes too soon after a PCIe
FLR (which would be triggered when VFIO was enabled).

After further discussion with Intel experts, we know
the SSD models that exhibit this issue.  So we can
quirk this so that only the older SSDs incur the extra
delay.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ieb408c24f6afd5bd5147d1c87239aa20f2d13511

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466064
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
This commit is contained in:
Jim Harris 2019-08-22 03:09:53 -07:00
parent d1d69a169c
commit 32e22643ef
4 changed files with 27 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
lib/nvme/nvme_ctrlr.c
View File

@ -2064,18 +2064,18 @@ nvme_ctrlr_process_init(struct spdk_nvme_ctrlr *ctrlr)
switch (ctrlr->state) {
case NVME_CTRLR_STATE_INIT_DELAY:
nvme_ctrlr_set_state(ctrlr, NVME_CTRLR_STATE_INIT, ready_timeout_in_ms);
/*
* Controller may need some delay before it's enabled.
*
* This is a workaround for an issue where the PCIe-attached NVMe controller
* is not ready after VFIO reset. We delay the initialization rather than the
* enabling itself, because this is required only for the very first enabling
* - directly after a VFIO reset.
*
* TODO: Figure out what is actually going wrong.
*/
SPDK_DEBUGLOG(SPDK_LOG_NVME, "Adding 2 second delay before initializing the controller\n");
ctrlr->sleep_timeout_tsc = spdk_get_ticks() + (2000 * spdk_get_ticks_hz() / 1000);
if (ctrlr->quirks & NVME_QUIRK_DELAY_BEFORE_INIT) {
/*
* Controller may need some delay before it's enabled.
*
* This is a workaround for an issue where the PCIe-attached NVMe controller
* is not ready after VFIO reset. We delay the initialization rather than the
* enabling itself, because this is required only for the very first enabling
* - directly after a VFIO reset.
*/
SPDK_DEBUGLOG(SPDK_LOG_NVME, "Adding 2 second delay before initializing the controller\n");
ctrlr->sleep_timeout_tsc = spdk_get_ticks() + (2000 * spdk_get_ticks_hz() / 1000);
}
break;
case NVME_CTRLR_STATE_INIT:

6
lib/nvme/nvme_internal.h
View File

@ -123,6 +123,12 @@ extern pid_t g_spdk_nvme_pid;
*/
#define NVME_QUIRK_SHST_COMPLETE 0x200
/*
* The controller requires an extra delay before starting the initialization process
* during attach.
*/
#define NVME_QUIRK_DELAY_BEFORE_INIT 0x400
#define NVME_MAX_ASYNC_EVENTS (8)
#define NVME_MAX_ADMIN_TIMEOUT_IN_SECS (30)

6
lib/nvme/nvme_quirks.c
View File

@ -43,13 +43,15 @@ static const struct nvme_quirk nvme_quirks[] = {
NVME_INTEL_QUIRK_READ_LATENCY |
NVME_INTEL_QUIRK_WRITE_LATENCY |
NVME_INTEL_QUIRK_STRIPING |
NVME_QUIRK_READ_ZERO_AFTER_DEALLOCATE
NVME_QUIRK_READ_ZERO_AFTER_DEALLOCATE |
NVME_QUIRK_DELAY_BEFORE_INIT
},
{ {SPDK_PCI_VID_INTEL, 0x0A53, SPDK_PCI_ANY_ID, SPDK_PCI_ANY_ID},
NVME_INTEL_QUIRK_READ_LATENCY |
NVME_INTEL_QUIRK_WRITE_LATENCY |
NVME_INTEL_QUIRK_STRIPING |
NVME_QUIRK_READ_ZERO_AFTER_DEALLOCATE
NVME_QUIRK_READ_ZERO_AFTER_DEALLOCATE |
NVME_QUIRK_DELAY_BEFORE_INIT
},
{ {SPDK_PCI_VID_INTEL, 0x0A54, SPDK_PCI_ANY_ID, SPDK_PCI_ANY_ID},
NVME_INTEL_QUIRK_READ_LATENCY |

6
test/unit/lib/nvme/nvme_ctrlr.c/nvme_ctrlr_ut.c
View File

@ -1793,8 +1793,12 @@ test_nvme_ctrlr_init_delay(void)
g_ut_nvme_regs.cc.bits.en = 0;
g_ut_nvme_regs.csts.bits.rdy = 0;
/* Delay initiation and default time is 2s */
SPDK_CU_ASSERT_FATAL(nvme_ctrlr_construct(&ctrlr) == 0);
/* Test that the initialization delay works correctly. We only
* do the initialization delay on SSDs that require it, so
* set that quirk here.
*/
ctrlr.quirks = NVME_QUIRK_DELAY_BEFORE_INIT;
ctrlr.cdata.nn = 1;
ctrlr.page_size = 0x1000;
ctrlr.state = NVME_CTRLR_STATE_INIT_DELAY;