ivampiresp/Spdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Set low watermark in NVMe/TCP target to a more appropriate value.

Browse Source 
In SPDK NVMe/TCP target, when initializing the socket, the low
watermark is set to sizeof(struct spdk_nvme_tcp_common_pdu_hdr),
which is 24 bytes. In our testing, some times there might be very
small data packet (as small as 16 bytes) be sent to wire. After
this, if there is no more data sent to the same socket, this small
data packet won’t be received by NVMe/TCP controller qpair thread
because the size hasn’t reached the low watermark. Because of this,
the qpair thread is waiting for more data come in and the initiator
is waiting for the IO request to be completed. Hence the delay
happens.

As the minimum data that allows target to determine the PDU type is
sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 8 bytes, we
changed low watermark setting as below. With the change, the problem
was gone immediately.

Change-Id: I14ccc4c84b77e33a617726e7455304aca29d5d57
Signed-off-by: Wenhua Liu <liuw@vmware.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2138
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This commit is contained in:
Wenhua Liu 2020-05-01 21:40:40 +00:00 committed by Tomasz Zawadzki
parent cfb65ba611
commit a4340e4501
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
lib/nvmf/tcp.c
View File

@ -861,7 +861,7 @@ spdk_nvmf_tcp_qpair_sock_init(struct spdk_nvmf_tcp_qpair *tqpair)
int rc;
/* set low water mark */
rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_c2h_data_hdr));
rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_common_pdu_hdr));
if (rc != 0) {
SPDK_ERRLOG("spdk_sock_set_recvlowat() failed\n");
return rc;