From 019cbb9335c016b86af0e1fa68303e4f34a27e5a Mon Sep 17 00:00:00 2001 From: Jacek Kalwas Date: Wed, 22 Feb 2023 06:39:57 -0500 Subject: [PATCH] nvmf: disable data buf mempool cache Depending on the number of cores there are sporadic issues getting elements of that pool although free elements are there during poll group creation. Operation returns -ENOBUF. It results in odd notice msg. "nvmf_transport_poll_group_create: *NOTICE*: Unable to reserve the full number of buffers for the pg buffer cache. Decrease the number of cached buffers from 455 to 1366" In this case 1366 is the actual number of available elements in the pool. Few poll groups suceeds and few are ending up with the buffer cache size set to 0. Issue has been rootcaused as bug or behaviour change in DPDK v22.01. Consider example: We create DPDK mempool with 4K buffers, cache of 256. When first poll group requests 512 buffers, DPDK mempool first looks in its per-core cache, sees no buffers (mempool buffer cache doesn't get prepopulated) and then requests 512 + 256 buffers from the backing pool. It returns 512 of the buffers to the user, and puts the other 256 buffers in the cache ...it should only request 512 buffers total. For 8 cores and 512 buffers requested only 5 cores will get their buffers. Disabling mempool cache seems to workaround the issue. More effective cache is already implemented on nvmf generic layer. Signed-off-by: Jacek Kalwas Change-Id: I3149dea95a4f24a75dd0074eda9468c4856d901d Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16913 Tested-by: SPDK CI Jenkins Community-CI: Mellanox Build Bot Reviewed-by: Jim Harris Reviewed-by: Tomasz Zawadzki --- lib/nvmf/transport.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/nvmf/transport.c b/lib/nvmf/transport.c index 78041a7c3..67325a3b9 100644 --- a/lib/nvmf/transport.c +++ b/lib/nvmf/transport.c @@ -197,11 +197,8 @@ nvmf_transport_create_async_done(void *cb_arg, struct spdk_nvmf_transport *trans } if (ctx->opts.num_shared_buffers) { - transport->data_buf_pool = spdk_mempool_create(spdk_mempool_name, - ctx->opts.num_shared_buffers, - ctx->opts.io_unit_size + NVMF_DATA_BUFFER_ALIGNMENT, - SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, - SPDK_ENV_SOCKET_ID_ANY); + transport->data_buf_pool = spdk_mempool_create(spdk_mempool_name, ctx->opts.num_shared_buffers, + ctx->opts.io_unit_size + NVMF_DATA_BUFFER_ALIGNMENT, 0, SPDK_ENV_SOCKET_ID_ANY); if (!transport->data_buf_pool) { if (spdk_mempool_lookup(spdk_mempool_name) != NULL) { SPDK_ERRLOG("Unable to allocate poll group buffer pull: already exists\n");