ivampiresp/Spdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nvmf: disable data buf mempool cache

Browse Source 
Depending on the number of cores there are sporadic issues getting
elements of that pool although free elements are there during poll
group creation. Operation returns -ENOBUF. It results in odd notice
msg.

"nvmf_transport_poll_group_create: *NOTICE*: Unable to reserve the
full number of buffers for the pg buffer cache. Decrease the number of
cached buffers from 455 to 1366"

In this case 1366 is the actual number of available elements in the
pool. Few poll groups suceeds and few are ending up with the buffer
cache size set to 0.

Issue has been rootcaused as bug or behaviour change in DPDK v22.01.

Consider example:
We create DPDK mempool with 4K buffers, cache of 256. When first poll
group requests 512 buffers, DPDK mempool first looks in its per-core
cache, sees no buffers (mempool buffer cache doesn't get prepopulated)
and then requests 512 + 256 buffers from the backing pool. It returns
512 of the buffers to the user, and puts the other 256 buffers in the
cache ...it should only request 512 buffers total. For 8 cores and 512
buffers requested only 5 cores will get their buffers.

Disabling mempool cache seems to workaround the issue. More effective
cache is already implemented on nvmf generic layer.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I3149dea95a4f24a75dd0074eda9468c4856d901d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16913
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This commit is contained in:
Jacek Kalwas 2023-02-22 06:39:57 -05:00 committed by Tomasz Zawadzki
parent b9298bf908
commit 019cbb9335
1 changed files with 2 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
lib/nvmf/transport.c
View File

@ -197,11 +197,8 @@ nvmf_transport_create_async_done(void *cb_arg, struct spdk_nvmf_transport *trans
}
if (ctx->opts.num_shared_buffers) {
transport->data_buf_pool = spdk_mempool_create(spdk_mempool_name,
ctx->opts.num_shared_buffers,
ctx->opts.io_unit_size + NVMF_DATA_BUFFER_ALIGNMENT,
SPDK_MEMPOOL_DEFAULT_CACHE_SIZE,
SPDK_ENV_SOCKET_ID_ANY);
transport->data_buf_pool = spdk_mempool_create(spdk_mempool_name, ctx->opts.num_shared_buffers,
ctx->opts.io_unit_size + NVMF_DATA_BUFFER_ALIGNMENT, 0, SPDK_ENV_SOCKET_ID_ANY);
if (!transport->data_buf_pool) {
if (spdk_mempool_lookup(spdk_mempool_name) != NULL) {
SPDK_ERRLOG("Unable to allocate poll group buffer pull: already exists\n");