If the qpair is part of a poll group, the socket will get
flushed as part of polling that group already. We only need
to explicitly flush to handle the case where the qpair is
not in a poll group.
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: Ib2a510b6d26d1622950437d81e0a40f6b15d6b54
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7049
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
There was a fix for this that went into the posix layer, but the
underlying problem is the logic in the nvme/tcp transport. Attempt to
fix that instead.
Change-Id: I04dd850bb201641d441c8c1f88c7bb8ba1d09e58
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6751
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Remove the polling group check. Because at this moment,
the qpair is not added into a polling group. If we do
not remove it, we will never enable zcopy feature for
I/O qpair.
And in sock implementmentation, we already fixed the zero copy
handling if a socket is not in a polling group. See
posix_sock_flush function. So we can fix this issue if we directly
remove this check.
Reported by: Aleksey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I969936c4b6c7f13cbfa4d6eb479010c53f3e384a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7056
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
This patch refactor the pdu sending logic with the async manner,
then if the group contains the accel engine, we can use it.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I2d669c0a3255d7a8898441e406906add2f3a3556
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6759
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
It is invalid to try to delete a NULL qpair, so do
not check for it in nvme_tcp_ctrlr_delete_io_qpair and
return an error when NULL. Just change it to an
assert instead. This makes it consistent with pcie
and rdma.
While here, add an assert in rdma as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic2f76deecb21b78749dac85e33fb1fa0d14a1239
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6917
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: <dongx.yi@intel.com>
Previously the callback parameter for this function is NULL, this will
cause segment fault, so pass the correct parameter here.
Fix#1817
Change-Id: Ie768b7bf4a72862d16a44742ab3032803d0939a2
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6690
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Community-CI: Mellanox Build Bot
Purpose: To get the optimal group, we need the socket information.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I17b048a402fbf002307dd225f64b20a9f876d642
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3324
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Community-CI: Mellanox Build Bot
Add the real support in nvme tcp transport.
Change-Id: I2aa9b0284d6fe009925e67f602a055e787f77987
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5734
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
No need these returns at the end of void functions.
So remove them.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I8889745f3ef82af513d03259a77a33c1f4f536cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6015
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
In the next patch this member will be used to track
both read and write offsets
Change-Id: I852125ff35257f9821ddf4a641d96afb29ebf0a0
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5924
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
There is a problem with TCP zcopy enabled:
1. TCP initiator sends icreq and start polling a qpair. Polling of qpair
actively calls nvme_tcp_read_pdu function
2. nvme_tcp_read_pdu: qpair is in NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_CH state,
it reads 8 bytes of common PDU header. It determines the type of the PDU
and finds the size of PDU_PSH header.
3. nvme_tcp_read_pdu: qpair is in NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_PSH state.
It should read 120 bytes of icresp PDU. The number of bytes which needs to be
read is pdu->psh_len - pdu->psh_valid_bytes. qpair receives 120 bytes
(the full PDU) and calls nvme_tcp_pdu_psh_handle -> nvme_tcp_icresp_handle.
Here we check that we haven't yet received buffer reclaim notification and
simply return from this function. At the same time we continue to poll the qpair.
4. nvme_tcp_read_pdu: qpair is in NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_PSH state
and tries to read data from a socket again. The number of bytes is
pdu->psh_len - pdu->psh_valid_bytes. But now pdu->psh_len == pdu->psh_valid_bytes,
so we call nvme_tcp_read_data with zero length.
readv with zero length is commonly used to check errors on the socket,
but in our case there is no errors and readv returns 0.
5. nvme_tcp_read_data treats zero as error and return NVME_TCP_CONNECTION_FATAL.
Fix is to handle icresp, but leave qpair in INITIALIZING state until
we receive acknowledgement for icreqsend_ack. We also move qpair to
NVME_TCP_PDU_RECV_STATE_AWAIT_PDU_READY recv_state so recv_pdu
will be zerofied and qpair will try to read a common PDU header.
But since it is not initialized yet, it won't receive anything
from the target.
Fixes issue #1633
Change-Id: I22cedefe530a8ac3b51495988ed6265d8fad15bb
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4969
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
According to the SPEC we should support up to 8192 bytes
of ICD for admin and fabric commands. Transport configuration
parameter in_capsule_data_size is applied to all qpair types -
admin and IO. Also we allocate resources when we get a connection
request, so we don't know qpair type at this moment.
Create a list of buffer in TCP poll group to support ICD up
to 8192 bytes when configuration ICD is less than this value.
The number of elements in this pool is hardcoded, it is planned
to add a new configuration parameter later.
Fixes issue #1569
Change-Id: I8589e3e2ea95d515f5503c6de7c1ee40aaf7b6da
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4754
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The current approach checks "rc == 0". It worked before adding
polling of poll group since a single qpair should return 1
completion for its own icreq while poll group can return
several completions for all qpairs attached to this poll
group (but .e.g not for those qpair who is waiting for the
completion).
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I60d05d8d6640e4e2bbaf3cd533d2f5a3637adea1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4768
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch removes the string from register component.
Removed are all instances in libs or hardcoded in apps.
Starting with this patch literal passed to register,
serves as name for the flag.
All instances of SPDK_LOG_* were replaced with just *
in lowercase.
No actual name change for flags occur in this patch.
Affected are SPDK_LOG_REGISTER_COMPONENT() and
SPDK_*LOG() macros.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I002b232fde57ecf9c6777726b181fc0341f1bb17
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4495
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Mellanox Build Bot
Reviewed-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI
We can receive buffer reclaim notifications only when a qpair is
attached to a poll group (so qpair's socket is connected to a socket
poll group).
The previous assumption that we enable zcopy only for IO qpairs was
wrong since IO qpair might not use poll groups too (e.g. abort
application).
Fixes issue #1607
Change-Id: I67329d755d81da6606e65eddfeceb20839346d87
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4476
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This seems to be causing some CI test failures. So
disable zero copy in all cases for now for client
sockets.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iceea09fe65fb90c7df15f500878a473f1ad4152c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4473
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In NVME TCP initiator zero copy is enabled for IO qpairs
and disabled for admin qpairs
Change-Id: Ibdf521dccde9b95ec5dd15a5eb2baed8fcf8b88e
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4211
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
A preparation step for enabling zero copy in NVMEoF TCP initiator.
With zero copy enabled, some requests might be completed out
of "process_completions" call and we should take them into
account to return the correct number of completions.
Change-Id: Iba7973f6da815645bbfad0334619d46b66379226
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4209
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
A preparation step for enabling zero copy in NVMEoF TCP initiator.
We should wait for both events to occur before continue qpair
initialization.
Add a new bit to nvme_tcp_qpair::flags to track receiving of icreq ack
since icreq is sent without tcp_req and there is no way to apply
existing synchronization mechanisms.
Move tcp qpair to initializing state if we receive icresp before icreq ack,
this state will be checked during handling of icreq ack to continue
qpair initialization
Change-Id: I7f1ec710d49fb1322eb0a7f133190220b9f585ab
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4207
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
A preparation step for enabling zero copy in NVMEoF TCP initiator.
Since nvme_tcp_qpair_process_completions doesn't process poll
group, we can't get asycn notification from kernel.
1. Add a qpair to poll group before we send icreq in order to be able
to process buffer reclaim notification.
2. Check if qpair is connected to a poll group and call
nvme_tcp_poll_group_process_completions instead of
nvme_tcp_qpair_process_completions when waiting for icresp
3. Add processing of poll group to nvme_wait_for_completion_timeout
and nvme_wait_for_completion_robust_lock since they are used to
process FABRIC_CONNECT command
Change-Id: I38d2d9496bca8d0cd72e44883df2df802e31a87d
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4208
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Currently host/data digest are bool members of nvme_tcp_qpair
structure. Change the type of this members to bitfield, reserved
bits will be used in the next patches to support zero copy.
Change-Id: If0659bf2445901e45fe0816af5f4fca5f494b154
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4206
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
A preparation step for enabling zero copy in NVMEoF TCP initiator.
Make sure that we complete a request (call user's callback)
when all acknowledgements are received. For write operation -
when we received send cmd ack, h2c ack and response from target.
For read operation - when we received send cmd ack and c2h
completed
Since we can receive send ack after resp command, store
nvme completion received in resp command in a new field
added to tcp_req structure
Change-Id: Id10d506a346738c7a641a979e1c8f86bc07465a4
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4204
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
A preparation step for enabling zero copy in NVMEoF TCP initiator.
Some NVMEoF TCP targets can send several R2T requests. We should
check that we finished the previous H2C (received buffer reclaim
notification from kernel) before sending the next H2C.
This patch adds a new ordering bit indicating the described case
and 2 fields to nvme_tcp_req to store the values from the last R2T
request which will be applied when send ack is received.
Change-Id: Iaa5ad846712ca18a8382680baa02413c18c4eb37
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4203
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Fix some spelling and make the message clearer
Change-Id: Ib291542a9735d6409db84f16c530e78567123f67
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4249
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Rename ordering bit r2t_recv to h2c_send_waiting_ack, that is more
descriptive name.
Change-Id: I6d6143ff4c1cccc74e11226b7974706808092f9a
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4202
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This makes it easier to zerofy ordering bits.
Change-Id: If5696bfedfff1bf75e41c1449eac7fccb469e98b
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4201
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
We should make nvme_tcp_ctrlr_connect_qpair always return
negative value if this function fails.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I457e704e39d7a3acd298fd48e89e8ea51e2ed4ad
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3809
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
According to page35 in recent NVMe-oF spec (
NVMe-over-Fabrics-1.1-2019.10.22-Ratified), ioccsz is used
to restrict the incapsule size of I/O command, so do not
restrict the NVMe-oF OPC command and also the admin command.
We accidently trigger an bug in kernel since we do not send
the fabrics command with the incapsule and make the kernel
coredump, though the kernel has bugs.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I869a2c8ab7b9c2ac1e5cc5b603920662591c2c64
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3837
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Purpose: To make the pdu management consistent with other PDUs, then
we can easily adapt our code into some hardware offloading solution.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: Ic4a2847fd1b6cacda4cbaa52ff12c338f0394805
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3588
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
1 Change the default factor from 4 to 8, which can be used
to improve the performance.
2 Change the base buffer size in nvme_tcp.c,
we should not use sizeof(struct spdk_nvme_tcp_cmd),
it is 72 bytes. Normally, the initiator will receive
C2h pdus and R2T Pdus by most, so set the size of using
sizeof(struct spdk_nvme_tcp_c2h_data_hdr) is enough.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I384f4cb026cb8d83e75b639f7256ee8cb8ed1df1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3283
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Previous we fix the same issue in this commit:
cb98b2ab3e
But we forget to fix it here. And we also need to update here,
otherwise we will still face the same issue described in
commit: cb98b2ab3e
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I3660dbb6e97c92ea4cb347cfce4bf23c6dfe97ab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3242
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
To abort requests whose cb_arg matches, add child abort request greedily.
Iterating all outstanding requests is unique for each transport but
adding child abort is common among transports, and adding child abort
is replaceable by other operations.
Hence add qpair_iterate_requests() function to the function pointer table
of transport, and pass the operation done in the iteration by a
parameter of it.
In each transport, the implementation of qpair_iterate_requests() uses
TAILQ_FOREACH_SAFE() for potential future use cases.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Ic70d1bf2613fce2566eade26335ceed731f66a89
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2038
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Each request has a callback context as cb_arg, and the callback to
nvme_complete_request() for the completed request may reuse the context
to the new request.
On the other hand, TCP transport dequeues tcp_req from
tqpair->outstanding_reqs after calling nvme_complete_request() for
the request pointe by tcp_req.
Hence while nvme_complete_request() is executed, tqpair->outstanding_reqs
may have two requests which has the same callback context, the
completed request and the new submitted request.
The upcoming patch will search all requests whose cb_arg matches to
abort them. In the above case, the search may find two requests by
mistake.
To avoid such error, move dequeueing tcp_req from tqpair->outstanding_reqs
before calling nvme_request_complete(). One exception is the case that
only nvme_tcp_req_put() is called. For the case remove tcp_req from
tqpair->outstanding_reqs before calling nvme_tcp_req_put().
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I5f2ac292c60431ac1e27b8657db92b220860a0a8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2865
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
As is well known, we may also handle the r2t data pdu receving earlier
before calling the cb function of send_cmd due to the outof order execution
of the lower layer uring socket interface.So we need to fix
this issue, otherwise the data of the sending_pdu will be placed with the
wrong data.
And it will cause the issue shown in
https://github.com/spdk/spdk/issues/1473Fixes#1473
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: Idac1ad65761695f3a655b85003861c1d1f4f3875
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3215
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Without this patch, we will face the following warning code
when compiled with (--with-uring --enable-debug) while testing
big I/O size: 256KB, e.g.,
"nvme_qpair.c: 474:nvme_qpair_resubmit_requests: *ERROR*:
Unable to resubmit as many requests as we completed"
The reason is because the nvme_tcp_request structure is not freed yet
if send_ack is not set, so there will be no entries when there
are other requests submit again.
And this patch can mitigate such issue.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I4c7616fbd3c82a883b4e9facd257a1a4f66e876d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3123
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
When using uring socket, we see following assert
nvme_tcp.c:1018: nvme_tcp_capsule_resp_hdr_handle:
Assertion `tcp_req->state == NVME_TCP_REQ_ACTIVE' failed.
Detailed info is in
https://ci.spdk.io/results/autotest-per-patch/builds/19205/archive/nvmf-tcp-vg-autotest/build.log
We face this issue, because there is also code execution ordering
between "sending callback function" and "pdu receving function".
We did not find it in physical machine testing, but finding it
in vagrant machine in CI.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I5eb241d564c0fc42ce0601b7c85999a2550f0de3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3046
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Purpose: It will be used to leverage the uring acceleration
later when we use io_uring_prep_write_fixed.
Because for using the Registered buffers feature in I/O uring,
we currently can register all the huge memories. And if we
allocate send_pdus in DMA memory, we can leverage such feature.
Change-Id: Id0ba5f7fe43202027c0378e9cbe74d861aad21e5
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3002
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch tries to solve the out of order
call back handling for cmd sending and the incoming pdu handling.
Normally, the cmd call back will be called before
receving the next PDU from the target if the application
uses the sync manner.
With the uring implementation, after sending the
cmd to the target, we may have the following scenerio:
(1) Firstly receive the incoming pdu(e.g., CapsuleResp pdu, C2hdata pdu)
due to the group polling read event.
(2) Secondly execute the callback function related with NVMe command sending.
This means that the data from the initiator is really sent out to the target,
and the target receives, then sending back the data to the initiator. But the
uring io_uring_cqe event is not handled, thus if we execute
(1) first, it will clean the data structures related with nvme_tcp_req, and the
nvme_tcp_req will be used for other purpose. Then causes wrong behaviour like
the following:
"Rand Write test failed at QD=128 because fio hangs with the following error:
nvme_tcp.c: 971:nvme_tcp_capsule_resp_hdr_handle: *ERROR*:
no tcp_req is found with cid=66 for tqpair=0x7f23d8001710".
And this patch can address this issue.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I5043aaa8adf5033d93dedac15f633f0850e0b9f2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2818
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
remove only the spdk_ prefix from static functions in
the above libraries.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I59ce032c3312fa73f30c133fd62e603c1eee2859
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2365
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Purpose: To set the priority of the NVMe-oF connection especially
for TCP connection.
For example, the previous example can be:
trtype:TCP adrfam:IPv4 traddr:10.67.110.181 trsvcid:4420
With the change, it could be:
trtype:TCP adrfam:IPv4 traddr:10.67.110.181 trsvcid:4420 priority:2
The priority is optional. We try to change
spdk_nvme_transport_id but not in spdk_nvme_ctrlr_opts since
the opts in spdk_nvme_ctrlr_opts will reflect in every nvme ctrlr,
this is short of flexibility.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Sudheer Mogilappagari <sudheer.mogilappagari@intel.com>
Change-Id: I1ba364c714a95f2dbeab2b3fcc832b0222b48a15
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1875
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This makes more sense within the context of the nvme driver and
helps us avoid the awkward situation of getting a failed_qp callback
on a qpair that simply hasn't been connected.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ibac83c87c514ddcf7bd360af10fab462ae011112
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1734
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
By aborting all requests from every qpair when it is disconnected,
we can completely avoid having to abort requests when we enable the
qpair since nothing will be left enabled.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Iba3bd866405dd182b72285def0843c9809f6500e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1788
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The state should be changed and checked by the transport
layer. All transports should follow the same list of steps
when disconnecting/reconnecting.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: If2647624345f2c70f78a20bba4e2206d2762f120
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1853
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The qpair states should be maintained at the generic level.
Always going through the transport disconnect function is
one step in that direction.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I019b2b4a14fe192eff5293f918d633dde2c5400a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1851
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This variable really indicates when a qpair is
no longer connected. So NVME_QPAIR_DISCONNECTED is
actually much more accurate.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ia480d94f795bb0d8f5b4eff9f2857d6fe8ea1b34
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1850
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
