rdma_reg_msgs() was replaced by ibv_reg_mr() recently to support
persistent PD per RDMA device. The difference between rdma_dereg_mr()
and ibv_dereg_mr() is only return value and errno. For consistency,
replace rdma_dereg_mr() by ibv_dereg_mr().
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I55e0743690e74f9510863bfa122a75d0632dce4e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13949
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Get a PD for the device from the PD pool managed by the RDMA provider
when creating a QP, and put the PD when destroying the PD.
By this change, PD is managed completely by the RDMA provider or the hooks.
nvme_rdma_ctrlr::pd was added long time ago but is not referenced
anywhere. Remove nvme_rdma_ctrlr::pd for cleanup and clarification.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: If8dc8ad011eed70149012128bd1b33f1a8b7b90b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13770
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
SPDK NVMe RDMA initiator used the default PD per RDMA device. Default PD
may be changed when all QPs for the RDMA device are destroyed and created
again.
For multipath, the RDMA zero copy feature require the PD per RDMA device
to be persistent when all QPs for the RDMA device are destroyed and
created again.
Maintain such persistent PDs in this patch.
Add two APIs, spdk_rdma_get_pd() and spdk_rdma_put_pd().
In each call of two APIs, synchronize RDMA device list with
rdma_get_devices().
Context may be deleted anytime by rdma-core. To avoid such deletion,
hold the returned array by rdma_get_devices().
RDMA device has PD, context, ref. count, and removed flag. If context
is missing in rdma_get_devices(), set the removed flag to true. Then,
if the ref count becomes zero, free the PD and the RDMA device.
The ref. count of a RDMA device is incremented when spdk_rdma_get_pd()
is called and decremented when spdk_rdma_put_pd() is called.
To simplify synchronization, sort the returned array by
rdma_get_devices().
To avoid resource leakage, add destructor function and free all PDs
and related data at termination.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I093cb4ec2c7d8432642edfbffa270797ccf3e715
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13769
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
An earlier commit added ctrlr_ready into struct
spdk_nvme_transport_ops. However, the major SO
version was not increased.
Fixes: 3dd0bc9e (nvme: Add transport controller ready step)
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: Id903634f9aaf5bdaa62fd30e92a4fb39a985b86f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13981
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Instead of passing each parameter to create a module, just have the user
make one and pass it in. This makes it easier to change the module
definition later.
Change-Id: I3a29f59432a6f0773129d7b210fbc011175b2252
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13909
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Masked by how accel_perf was doing decomp verificiation which is
changed in the next few patches and verifies these fixes.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Icb03fc169bf8d2f05396addaf1db56d6de1827d1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13038
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Docs explaining how to use the RPC are in the next patch in the
series.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I7dab8fdbeb90cdfde8b3e916ed6d19930ad36e66
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12848
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
submit_inflight_desc() actually do some meaningful work, so when it really process tasks, the poller should return BUSY status.
Signed-off-by: YafeiWangAlice <yafei.wang@samsung.com>
Change-Id: I2103cea6d28e8b355dad4ddd603d917f10e44c08
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13486
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
There may be for_each operations outstanding on an
io_device when it is unregistered. Currently we just
return when this happens, not unregistering the
device but also not notifying the caller that this
happened (since it returns void, and the callback
function doesn't have a status parameter either).
We could just push this responsibility to the caller,
to never unregister an io_device if it knows it has
outstanding for_each calls waiting to complete. But
I think we can simplify this a lot by just handling
this inside of the thread library. Mark that the
device is pending registration, and unregister it
(on the original requesting thread!) when the
for_each count gets back to zero. Also don't
allow any new for_each operations either.
Note this requires a bit of refactoring on the
thread unit tests, since it is now possible to
unregister a device with outstanding for_each
operations.
Fixes issue #2631.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I978f2d99a25e65d2b7d71ce9b1926a79a6c94263
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13890
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
If spdk_for_each_channel is called on a device that doesn't
exist, we need to set a non-zero status (-ENODEV in this
case) to the completion function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I898ad5ea499fb6087338b621b2befcadd6a05414
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13889
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This has been implicitly required before, and all
in-tree apps (except accel_perf) set it, so let's
explicitly require it. This name gets used for
things like the shm name for spdk trace event file.
While here, add the name for accel_perf.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I47a22466550d4b31bacafee58d30339b4f22f4b4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13876
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
When decoupling a snapshot from its parent, we need to clear its parent.
So we should remove the xattr BLOB_SNAPSHOT. Modifying the xattrs of a blob
only works if its metadata are not in read-only mode.
By default, a snapshot is in read-only mode so this operation fails. When we
later want to delete the snapshot, we will see that it has a parent, so we will
try to remove the snapshot from its parent's clones list. This will cause a
crash.
The fix is to remove the BLOB_SNAPSHOT xattr only after setting the snapshot's
metadata in rw mode.
Signed-off-by: Alex Michon <amichon@kalrayinc.com>
Change-Id: I80efa6dd3dcb38b4c738ce2e97aa2ffc281cefa5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13723
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
DPDK may use this NULL pointer to access its member,
And then got segmentation fault. But we only need it
exit or report normal error.
To minimize the impact, and to prevent these going on,
we add check the error return for creating NULL mempool
in spdk_thread_lib_init_ext in spdk_reactors_init.
when error returning from spdk_thread_lib_init_ext in spdk_reactors_init.
It contains thread_lib_init which reports error for failed mempool.
Thus, codes will return and will not cause segmentation fault.
Fixes issue #2620.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I63369fdaeb231196e8f8daa826eb5b057ed829b8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13842
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Here should return -ENOMEM, and other places are
changed.
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: Id81cd7485733e66d996b1501061a45f774f2b51a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13863
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Parameter `num_queues` for virtio_scsi PCI device means
maximum number of queues, it SHOULD include the `eventq`
and `controlq`, while for `vhost_user` RPC call, it means
the number of IO queues, so here we use it as `max_queues`
in lib/virtio and add the fixed number queues for `vhost_user`
SCSI device.
Also fix `vhost_fuzz` to get `num_queues` earlier than
negotiate the feature bits.
Change-Id: I41b3da5e4b4dc37127befd414226ea6eafcd9ad0
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13791
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The `vhost_user` socket transport APIs are already in the
same source file, so just call the function directly.
No code logic changes in this commit.
Change-Id: If471b9b0166d43591fb8614e95a17473c964e87c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13789
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Similar with NVMe device driver, here `virtio` is a specification
abstraction library, `pci` and `vhost_user` are transports layer,
here we merge vhost_user.c and virtio_user.c into one new source
file `virtio_vhost_user.c` so that to make code more clear.
No logic change, just code movement in this commit.
Change-Id: I8e3e5c477e7c45e6eeebad240b8cc3c9476b86d1
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13788
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
While running into this function, even the subsystem can't be
destroyed due to error subsystem state, it's better to continue
the execution.
Continue to fix#2590, QEMU is stuck for the failure case, and
nvmf target should process such error because it may support other
normal subsystems at the same time.
Change-Id: Ib05e24996378b52070d2b760519f476f9b2d7e76
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13839
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The RPC provides a list of initialized engine names along with
that engine's supported operations.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I59f9e5cb7aa51a6193f0bd2ec31e543a56c12f17
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13745
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
In prep for upcoming patch that will provide an RPC to override
and automatic assignment of an op code to an engine.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I17d4b962fb376a77f97ce051a513679d0fba698e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12829
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
fix: If the first six characters of two scsi lun's name are the same,
such as aaaaaa0 and aaaaaa1, so do theirs naa identifier
Signed-off-by: Bin Yang <bin.yang@jaguarmicro.com>
Change-Id: I4e0541b372a0e20e95e0a24d62dd3d85b7abe230
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13824
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This is another preparation to create and use ibv_context and pd.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: Id594fa1ccb2daf535b1aaaef0a397bda2ec98578
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13710
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The following patches will create and use ibv_context and pd
explicitly instead of using default ibv_context and pd created
by rdmacm.
As a preparation, pass pd instead of cm_id to nvme_rdma_reg_mr().
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: Denis Nagorny <denisn@nvidia.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: Ifdcd18ed363b8ba4a23a920bf3559237e38821c6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13599
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
spdk in interrupt, reactor dosen't correctly handle exited threads,
causing vhost threads still in reactor's lw_threads list. The fix
will do cleanup thread when it's state becomes EXITED. Though it's
exposed in v22.05.x, but the master branch also has the problem.
We will do this as below:
(1) When thread's state becomes SPDK_THREAD_STATE_EXITED, reactor
process thread exits first.
(2) Then reactor do remove lw_thread and destroy it.
Fix issue: #2574
Signed-off-by: Apokleos <oliverliyn@gmail.com>
Change-Id: I3ac2681d70480563db3a0aee4aff61c2f272b140
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13706
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
If Controller Fatal Status (CFS) bit is set, there's no point in waiting
for CSTS.RDY and the only way to move forward with the initialization is
to perform a controller reset.
This fixes issues with test/nvme/sw_hotplug.sh when running under qemu.
It seems that during that test, qemu marks the emulated NVMe drives as
fatal, so if we didn't check CSTS.CFS, the initialization would time
out.
Fixes#2201.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I97712debc80c3dd6199545d393c0f340f29d33b2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13820
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Sometimes VM may get a kernel panic when starting, and SPDK CI will kill
`nvmf_tgt` after 60 seconds, and for this exception, SPDK will raise an
assertion when destroying the subsystem, while here, we remove this
assertion and print the error information.
CI will still mark this case as a failed case, then we can use this error
information to understand error subsystem state in vfio-user.
Fix issue #2590.
Change-Id: I20b16f9e96a566730eca2dd9ea165645bd9160bd
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13773
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
this is a prework for further changes - with lock on generic layer
lock on specific transport (e.g. tcp, rdma) layer becomes optional
possibly it won't be required if some contract introduced on public
interfaces (to be considered)
- spdk_nvmf_poll_group_[create|destroy]
- spdk_nvmf_tgt_listen_ext, spdk_nvmf_tgt_stop_listen
- spdk_nvmf_get_optimal_poll_group
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: Ib132babf9e7022342129fe795991cdad834e7f53
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13665
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When there are not enought transport buffers for
multi SGL request in state NEED_BUFFER, WRs
received from the data_wr_pool are returned back
to the pool. However rdma_req->data.wr.next pointer
still points to the first WR from the pool. Usually
it doesn't cause any problems since rdma_req will
try to fill buffers again, but when qpair is being
destroyed, all requests are completed forcefully.
When the request is completed and data.wr.next
pointer is not NULL, we'll try to put already
released WRs into the pool one more time.
That corrupts the pool and leads to undefined
behavior.
Fixes#2541
Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: I238b92eec132d8d845330362af6f335421177454
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13760
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
The function `nvme_ctrlr_init_ana_log_page` is exactly
same with `nvme_ctrlr_update_ana_log_page`, so remove it.
Change-Id: I1ad51635f47cf95cfa6de217e3b9144885c3b74e
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13652
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
allow DSA device to async offload crc32 calculation in nvmf-TCP
This patch can use DSA to accelerate crc32 computation, making
the io performance of TCP paths using crc32 approach the io
performance of TCP paths that do not use crc32.
Using SLIST to minimize the performance drop. SLIST has less
operation compared to TAILQ.
Thinking about memory thrashing, we should use the same memory as
possible to receive new PDUs. So, insert newly freed PDU in to head
is better.
The performance drop is within 1% compared to the TCP path without
crc32.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I480eb8db25f0e730cb198ca5ec19dbe3b4d38440
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11708
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We have to process whole QoS queue on each QoS poll. It may contain
IOs that still have quota or not affected by QoS rules at all. If we
stop on the first queued IO, all IOs will be limited by the minimum
QoS rule even if they're not affected by this rule.
Here is an example and simple test. We have a NVMf target with Null
bdev and QoS configured with read bandwidth limited to 10 MB/s and
write bandwidth limited to 100 MB/s. First we start nvme_perf with
only write IOs and we see that reported bandwidth is 100 MB/s. Then we
start another instance of nvme_perf with only read IOs. We see that
reported read bandwidth is 10 MB/s but we also see that write
bandwidth also drops to 10 MB/s.
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I1edf09d038e65f873deef19ecb0f4bf9725a5ca5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13767
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Original implementation creates pollers and CQs for all discovered
devices at poll group creation. Device (ibv_context) that has no
references, i.e. has no QPs, may be removed from the system and
ibv_context may be closed by rdma_cm. In this case we will have a CQ
that refers to closed ibv_context and it may crash in ibv_poll_cq.
With this patch pollers are created on demand when we create the first
QP for a device. When there are no more QPs on the poller, we destroy
the poller. This also helps to avoid polling CQs that don't have any
QPs attached.
Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I46dd2c8b9b2902168dba24e139c904f51bd1b101
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13692
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Both PCIE and VFIO-USER can use the same APIs to get IO queue
pair statistic data, so merge them here.
Change-Id: Iadf9ead2bd5abaf11d2ef5d1884acb67369f85bb
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13538
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
When a bdev is registered, it is examined by the bdev modules before the
bdev register even is notified.
Examination may be asychronous, e.g. when the bdev module has to perform
I/O on the new bdev.
This causes a race condition where the bdev might be destroyed while
examination is not finished. Then, once all modules have signaled that
examination is done, `bdev_register_finished` makes an invalid access to
the freed bdev pointer.
To fix this, defer the unregistration until the examine is completed by
opening a descriptor on the bdev.
Change-Id: I79a2faa96c1c893fc1cee645fbe31f689b03ea4a
Signed-off-by: Nathan Claudel <nclaudel@kalray.eu>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13630
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When a secondary process exit without deleting allocated IO
queue pair, then a new secondary process will do cleanup for
previous allocated queue pair, then segment fault will happen
due to `stat` inside IO queue pair data strucutre can't be
accessed in this cleanup process.
Fix issue #2565.
Change-Id: I01a037642683901941b5268ac20d17b78b6c6350
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13537
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
This avoids conflict with public vhost_user.h header
file which can cause problems with abidiff.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia258b4621eda9f6855d46bbf67d8369a053a7116
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13732
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This avoids conflict with public vmd.h header which
can cause problems with abidiff.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2f00c07226dec273516868f5fa9d7aa384378308
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13731
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Avoids conflict with public tree.h that can cause
problems with abidiff.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3ccf4c0198f7975d8ebbee57f50c52f9f2e96fc0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13730
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This avoids confusion with the public idxd.h
header file which causes problems with abidiff.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7910c93d9d95b99c82f4dfdba845e6804e1b6568
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13729
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When doing DIF insert and strip, we will reserve extra
buffer in block device layer to save DIF information,
so when attaching one device to Namespace, we will
check the value first so that the reserved buffer
size isn't smaller than metadata size.
Change-Id: Id9272886ce8a7c01271279686730af4e5b24f35a
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12188
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
When `dif_insert_or_strip` is enabled, NVMf library will do
DIF insert and strip automatically, client isn't aware of
it, when `dif_insert_or_strip` is disabled, we will report
Namespace E2E Protection Capabilities to client, but we
don't process PRACT and PRCHK flags in NVMf library, so
here we don't report the capabilities to client and leave
the use of extended LBA buffer to users.
Change-Id: Ic610dc65fef210a7799c6ab693d89138b99e1193
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12165
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Some of the options in sock_impl_opts could be different for different
sockets (even if they're using the same impl). However, outside of a
few selected options (recv_buf_size, send_buf_size), there was no
interface to change them.
This change will allow users to change impl_opts on a per-socket basis
when creating a socket. Sockets created through accept() inherit
impl_opts from the listening socket.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I7628ae19def25cef6ffa62aa54bd34e446632579
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13661
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Now that spdk_sock has impl_opts, we no longer need to store a copy of
impl_opts.zerocopy_threshold in spdk_sock.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I96377e330351b1afb57811578acfadf05d53f49c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13660
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Also remove ftl_mngt_get_status() because it won't be necessary now.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I335831cb1c506379e9afeb0bf87f1f873033073d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13668
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
it is impl and used only in nvmf.c source file
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I1236f9ede28c5da313d118ce73e1da64381379c5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13664
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
That is possible to get/set registers from any thread,
during regs processing we are polling admin qpair to
get a completion. At the same time, another thread
can also poll admin qpair and that can lead to
undefined behavior.
This patch fixes an issue when bdev_nvme is configured
with io_timeout. If remote target becomes unresponsive
(e.g. due to link down), IO timeout occurs and bdev_nvme
tries to get csts registers in timeout_cb. At the same
time another thread can process adminq, so we may have
2 simultaneous adminq polls. If admin qpair is disconnecting
at that time (RDMA transport) we may destroy resources
twice from different threads.
We don't see a problem with set_regs function but it
won't be redundant to lock mutex in set_regs as well.
Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: I7ec3984d25d0249061005533d13b22315b44ddf2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13687
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We cannot count AERs as outstanding IO for purposes
of subsystem pause, because we cannot expect them
to be completed. Previously we would account for this
in nvmf_ctrlr_async_event_request() by decrementing
the counter, but this did not consider cases in the
calling function (nvmf_ctrlr_process_admin_cmd) where
an AER might complete with error before this function,
resulting in the counter getting stuck indefinitely
with a >0 value.
Rather than adding a decrement in all of those
error cases, do a single check at the beginning
of nvmf_ctrlr_process_admin_cmd, and remove the
one from nvmf_ctrlr_async_event_request.
Fixes issue #2215.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ica969f116d80dfba0168369ff2fba9a4a42fc076
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13678
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
This patch defines a new function, spdk_sock_readv_async(), which allows
the user to send a readv request and receive a callback once the
supplied buffer is filled with data from the socket. It works simiarly
to asynchronous writes, but there can only be a single outstanding read
request at a time.
For now, the interface isn't implemented and any calls will return
-ENOTSUP. Subsequent patches will add support for it in the uring
module and as well as emulation in the posix module.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I924e2cdade49ffa18be6390109dc7e65c2728087
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12170
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
In NVMe/TCP target, the socket low water mark is set to
sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 8 bytes.
In corner test, there might be 4 bytes data packet sent to
NVMe/TCP target, after that, if there is no more data sent to
the same socket, the 4 bytes won't be read by NVMe/TCP target
qpair thread. Because of this, there is a IO request didn't
complete in initiator. Then, if manual call the readv function to
read the 4 bytes for the pdu in target, the io request complete
normally in initiator. It seems like the pdu might be split,
and in the situation, the IO request will not complete until
new IO request reach.
After set low water mark in NVMe/TCP target to 1 byte, just
like iscsi target done, the issue disappear immediately.
Signed-off-by: BinYang0 <bin.yang@jaguarmicro.com>
Change-Id: I59d3d900f0b25632d786ef25ab096eabe43476bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13633
Reviewed-by: <chuanwei.ji@jaguarmicro.com>
Reviewed-by: Qingmin Liu <qingmin.liu@jaguarmicro.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
It is a nicer API to allow users to use an
md-related API such as spdk_bdev_read_blocks_with_md
passing md_buf as NULL to mean "don't read metadata".
This avoids the need for an if-statement in the users
code to check if the md buffer is NULL before deciding
which API needs to be called.
This basically requires two changes:
1) only check if the metadata is separate for the bdev
if the md_buf != NULL
2) do not fail if the buffer is specified but the
md buffer is not (we only need to fail the case where
the md buffer is specified but the data buffer is not)
Note that spdk_bdev_readv/writev_blocks_ext was already
allowing the metadata buffer to be NULL, but change
those functions too to match the others on how we check
if the data buffer isn't allocated.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I764cf49b9f573fccb19e73876a376fd231cc3580
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13612
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This will allow the code calling this RPC to interpret the error and
check whether the transport already exists (-EEXIST) or some other error
occurred.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I8c4af84763ddba908c59ff881b09834a439186a8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13577
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
In the multi-process case, a process may call `spdk_nvme_ctrlr_free_io_qpair` on
a foreign I/O qpair (i.e. one that this process did not create) when that qpairs
process exits unexpectedly.
The variable `qpair->poll_group` isn't multi-process safe, we can't use it
in `spdk_nvme_ctrlr_free_io_qpair` and related transport poll group APIs.
Change-Id: Ic13a6a2c7d760477be5be5a56a45caa2b5518717
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13573
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
It will not find the h2c related reqs in the tailq now.
We can get it from tqpair->reqs directly.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I25f0900e875b054d7617450477e9719e7a59aa18
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12861
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Currently we initialize pending_bytes only in pre-copy state. This is
pointless since we don't generate any migration data at this state, so
if the vfio-user client reads migration data it will be garbage. Even
worse, we don't re-initialize pending_bytes in stop-and-copy state, so
if the vfio-user client reads the entire migration data in pre-copy state
then there will be nothing left to read in the stop-and-copy state,
which is where we actually produce the migration data. This results in
corruption of the controller's state (e.g. queues).
This patch ensures that migration data are available in the
stop-and-copy state, by setting pending_bytes accordingly only in that
state.
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
Change-Id: I0b215e64cd1f58f254e1079f06402d196f984099
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11718
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The read_data, write_data, and data_written migration callbacks assume
that the migration data are accessed in one go. Until this is fixed,
with this patch we ensure we don't ignore unsupported ranges.
Change-Id: I640415858b8c374ffc9e487cd20f5130e0be9305
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11717
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
All of the callers immediately put the req right
after the nvme_rdma_req_complete call, so just move
the put into that function instead.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic370cf689850924e0c902a6071af8b3a7ed58c0b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13527
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
This follows similar logic in the pcie and tcp
completion paths, including omitting error
messages when aborting aers by adding a print_on_error
parameter to the completion function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id558d0af2cdd705dfb60abb842bd567a0949ccce
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13525
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
By default, the SPDK nvmf target reports vid==INTEL,
which results in the SPDK nvme driver trying to enable
Intel vendor-specific log page. Fix this by trying to
enable those log pages only for PCIE transport
controllers.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I78ebf365d4fa6295d1f610697266c3ead765988d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13524
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
We can only get to this code path if the controller
has vid==INTEL, so make that more clear by changing
the check to an assert.
Remove unit test that calls
nvme_ctrlr_construct_intel_support_log_page_list()
for a controller that is not VID==INTEL - this is
no longer valid.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3b58451bc95992bf641e7452f0ac4c2bac9fe31c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13523
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
This follows similar logic in the pcie completion
path, including omitting error messages when aborting
aers by adding a print_on_error parameter to the
completion function.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I96df72280bb8fcbee3847fdc27f38e14a1bf3251
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13522
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
nvme_tcp_req_complete_safe caches values on
the request, so that we can free the request *before*
completing it. This allows the recently completed
req to get reused in full queue depth workloads, if
the callback function submits a new I/O.
So do this nvme_tcp_req_complete as well, to make
all of the completion paths identical. The paths
that were calling nvme_tcp_req_complete previously
are all non-fast-path, so the extra overhead is
not important.
This allows us to call nvme_tcp_req_complete from
nvme_tcp_req_complete_safe to reduce code duplication,
so do that in this patch as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I876cea5ea20aba8ccc57d179e63546a463a87b35
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13521
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
All callers of nvme_tcp_req_complete call
nvme_tcp_req_put immediately afterwards, so move
this call into nvme_tcp_req_complete.
This will help enable some improvements in later
patches.
Note that nvme_tcp_req_complete_safe has this same
functionality open coded right now, but that will
get changed in the next patch. It calls
nvme_tcp_req_put immediately after the TAILQ_REMOVE,
so do that in nvme_tcp_req_complete as well.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I368122bc49a7f0772e3011e5427e3c43618380eb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13520
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
nvme_qpair_abort_all_queued_reqs() aborts error injections, queued
requests, aborting queued requests, and outstanding requests. (Aborting
outstanding requests depends on transports.) However, it did not abort
queued aborts.
Include nvme_ctrlr_abort_queued_aborts() into
nvme_qpair_abort_all_queued_reqs() to do really the name of the
function indicates.
nvme_ctrlr_abort_queued_aborts() has been called in a few cases, but
we do not care duplication.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I19102cc6603a72ce5c398a7947cb4d606b692991
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12849
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Vasuki Manikarnike <vasuki.manikarnike@hpe.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
QEMU may exit due to some exceptions which mean the socket
connection may be disconnected at any time, so for asynchronous
callbacks especially the subsystem pause/resume callbacks, they
all run in asynchronous way, the controller pointer may become
invalid before the callbacks are called.
Fix#2530.
Change-Id: I6d73597d75761e28844e83bfee7f8a446d85fa49
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12831
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Fix issue: #2561
The issue here is that in the bdev_set_qd_sampling_period RPC
command, the QD sampling period has been set. Then later the
related Desc is closed and in the bdev_close() function the
QD sampling period is reset to 0.
A new QD desc is added as the QD sampling period update could
be handled properly.
Meanwhile, a new QD Poll In Progress flag is also added so as
to indicate there are ongoing events of QD sampling and the
Bdev unregister will be handled in the proper way.
Related test case and unit test also updated for this change.
Change-Id: Iac86c2c6447fe338c7480cf468897fc8f41f8741
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13016
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
The bdev_lvol_grow_lvstore will grow the lvstore size if the undering
bdev size is increased. It invokes spdk_bs_grow internally. The
spdk_bs_grow will extend the used_clusters bitmap. If there is no
enough space resereved for the used_clusters bitmap, the api will
fail. The reserved space was calculated according to the num_md_pages
at blobstore creating time.
Signed-off-by: Peng Yu <yupeng0921@gmail.com>
Change-Id: If6e8c0794dbe4eaa7042acf5031de58138ce7bca
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9730
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reserve space for used_cluster bitmap. The reserved space is calculated
according to the num_md_pages. The reserved space would be used when
the blobstore is extended in the future.
Add the num_md_pages_per_cluster_ratio parameter to the
bdev_lvol_create_lvstore API. Then calculate the num_md_pages
according to the num_md_pages_per_cluster_ratio and bdev total size, then
pass the num_md_pages to the blobstore.
Signed-off-by: Peng Yu <yupeng0921@gmail.com>
Change-Id: I61a28a3c931227e0fd3e1ef6b145fc18a3657751
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9517
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
SPDK was previously incorrectly requesting log levels such as
LOG_NOTICE. Update libvfio-user so it is in fact supported, and check
that setting up the callback actually worked.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I41c2a8cf683868c3c2e40470f78e1af3dba29de4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12839
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Swapnil Ingle <swapnil.ingle@nutanix.com>
Update libvfio-user such that the SGL access APIs can be used
concurrently. We are guaranteed that the guest memory remains mappable
now that the vfio-user transport has implemented quiescence.
This is currently only really useful (for a single controller) in poll
mode, but shouldn't break interrupt mode, as we still ensure all a
controller's queues are on the same poll group in that case.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I0988e731558e9bf63992026afc53abc66ec2a706
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12349
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
In SPDK, declarations have the return type on the same line. Definitions
have the return type on a separate line. Astyle has an option for
enforcing this. Unfortunately, it seems to have two bugs:
1) It doesn't work correctly at all on C++ files.
2) It often fails on functions that return enums, or long type names
Deal with 1) by adjusting the check_format.sh script to only tell astyle
to fix return type line breaks for C files and not C++. Deal with 2) by
adding a few typedefs to work around the problem.
Change-Id: Idf28281466cab8411ce252d5f02ab384166790c6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13437
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
the underlying spdk_cpuset_copy() takes `const spdk_cpuset*` as the
`src` parameter. there is no need to take non-const spdk_cpuset*.
hence, in this change, let's relax the requirement of the pointer type.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Change-Id: I1f626c7fea45cf7250bf56b891bcba4a0f2a8917
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13443
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
We had not incremented ctrlr->outstanding_aborts when aborting a
request in the ctrlr->queued_aborts, and ctrlr->outstanding_aborts
became negative. Fix the bug in this patch. Additionally add assert
to check if ctrlr->outstanding_aborts is not negative.
Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: I58090286f070ba854bdea87f0f8ecb7810890338
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13452
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
While we are quiesced, we're not allowed to access guest memory via the
SGL APIs. Refuse to process any commands unless we're in RUNNING state.
We need to synchronize with each poll group via a message before we can
call vfu_device_quiesced(), otherwise we could still be processing
commands via nvmf_vfio_user_sq_poll().
For interrupt mode, we then might miss processing commands in a
corresponding interrupt callback, so make sure we process them when we
return to RUNNING state.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ieae5a9ae8d9de722e0bdf4bb8d61e7e678159f1f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12912
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
An oversight meant that quiesce was in fact only pausing the admin
queue, and not ensuring no I/O was ongoing. Fix this by passing the
right flag to spdk_nvmf_subsystem_pause().
Change-Id: I930c616d1170ac0299339b04928da57f6a7489ab
Signed-off-by: John Levon <john.levon@nutanix.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13441
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
If the broadcast NSID is supplied, every namespace is paused.
Change-Id: I40cc3e04b5a75b731ab0c8946ed8146275cc8ee4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13394
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
It's useful to add these APIs.
spdk_copy_iovs_to_buf and spdk_copy_buf_to_iovs.
It prepares that other ones can call these.
We don't need to define them in static state
repeatedly.
And add corresponding unit tests.
Change-Id: Ife40fec8d047a48af67b04e6c055e4932282abfb
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12075
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Profiling data showed the deference of the CQ head in cq_is_full() was a
significant contributor to the CPU cost of post_completion(). Use the
cached ->last_head value instead of a doorbell read every time.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ib8c92ce4fa79683950555d7b0c235449e457b844
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11848
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Fully asynchronous ctrlr detach (b6ecc3729) introduce a register
operation state machine that waits for operation to complete. When
controller failed to initialize, `nvme_ctrlr_fail` set qpair state to
`DISCONNECTED` immediately, causing qpair process completions to
never complete register operations therefore prevent async detach exit.
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I205c5157b8ea7b4535f98ff4052414310e421446
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12858
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
reactor_run() decides whether to start gather_metrics
based on non-zero scheduler period.
The default of 1 sec was set during initialization,
in scheduler_subsystem_init().
This resulted in unessecary operations each second,
even if only 'static' scheduler is used.
This patch moves setting default scheduling period to
respective schedulers.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I953aee271a959b6314c8e83434c922dba9638de4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9492
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
When calculating the bar_addr which is used to access SPARSE MMAP area, we should use the
(offset - region->mmaps[i].offset) as the increment to get the valid access address.
Signed-off-by: Jun Zeng <jun1.zeng@intel.com>
Change-Id: Ie5d0c63cf572847d15dc92f0995fddecf35f1cdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13021
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
rte_power was added to DPDK long time ago,
but some of the DPDK packages do not include it.
For those cases just skip building components that depend on in.
This change still allows to use dynamic scheduler, since
the dpdk_governor usage is optional.
Fixes#2534
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ied88edc8d58aae07d1384c1c40203fc80b919d80
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12993
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
dpdk_governor and gscheduler use rte_power,
which is only available on Linux and when
DPDK env is used.
Rather than repeat those checks in each mk or Makefile,
added DPDK_POWER flag directly to DPDK env.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I438caad8d333a4df697a79aa45de2930cce71d23
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12992
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
rte_net is a dependency for both rte_vhost and rte_power.
Next patch will simplify the checks to include rte_power,
and keeping this depenency next to component that directly
depends on it will make it easier to understand.
Since DPDK_LIB_LIST is sorted by the end of the env.mk,
it shouldn't be a problem to include the rte_net twice.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If2bb2aa5d972148ca8143023657b0aec45306a08
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12991
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When running perf test, sometimes after CONNECT req's resp was
received and processed, the qpair still failed to change from state
CONNECTING to CONNECTED. For when it goes to nvme_fabric_qpair_connect_poll
-> nvme_wait_for_completion_robust_lock_timeout_poll to process the
CONNECT req's resp, the req may have not been finished in sock_check_zcopy,
although its resp has been received and processed, which means the
tcp_req->ordering.bits.send_ack is still 0 and the status->done still
is false. And after the req is completed in sock_check_zcopy, we need
to poll this qpair again to make the state enter CONNECTED.
And if icreq's resp received and processed before nvme_tcp_send_icreq_complete
is called by _sock_check_zcopy, the qpair will be stuck in CONNECTING
and it never proceed to send the CONNECT req. We also need to put it
in pgroup->needs_poll to fix it.
I can reproduce this bug with the following configuration.
target: 16NVMe SSD, running on 20 cores;
initiator: randread test using nvme perf with 32 cpu cores and
zerocopy enabled.
The error doesn't always occur. CONNECT failure is about 1 failure in
ten with the following log. And icreq failure is less frequent with
only target side's "keep alive timeout" log.
Error reported in initiator side:
Initialization complete. Launching workers.
[2022-05-23 14:51:07.286794] nvme_qpair.c: 760:spdk_nvme_qpair_process_completions:
*ERROR*: CQ transport error -6 (No such device or address) on qpair id 2
ERROR: unable to connect I/O qpair.
ERROR: init_ns_worker_ctx() failed
And target side shows:
Disconnecting host from subsystem nqn.2016-06.io.spdk:cnode2 due to keep alive timeout
Change-Id: Id72c2ffd615ab73c5fc67d36c3ff8b730cebcef7
Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12975
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Patch below changed the struct spdk_nvmf_ctrlr_data by inserting
spdk_nvme_cdata_fuses. This affects large number of nvmf interfaces.
(cbfd581) nvmf: Add NVMe fused operations to spdk_nvmf_ctrlr_data
Unfortunately was missed due to lack of rebase after ABI update on
CI machines.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifd06d0ddbefe9ea6c9715adae9881d4606e34b44
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13013
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Kamil Godzwon <kamilx.godzwon@intel.com>
Reviewed-by: Michal Berger <michallinuxstuff@gmail.com>
Community-CI: Mellanox Build Bot
We already list the libraries with their explicit
pathnames, so the -rpath-link serves no purpose.
Our Makefile was actually specifying this option
without an = sign - i.e:
-Wl,-rpath-link /path/to/lib
On the submitter's system, this resulted in an error:
cc: Missing argument for -Wl,-rpath-link
I have no idea why no one has ever run into this
error, except for this one submitter. But removing
the -rpath-link is the right thing to do here, since it
is not needed - so do that rather than adding the =
sign and continuing to figure out differences in
-Wl option processing on these different systems..
Fixes issue #2540.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4f6176e55701a5dea5b10bba1ad621250cb5cb51
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12984
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Add an option to stop nvmf transport advertising support for both the
compare command and the fused compare_and_write operation in vfio_user
transport.
Signed-off-by: Alexis Lescouet <alexis.lescouet@nutanix.com>
Change-Id: I3900218c0e9884f86a5c8698a030f8106b64f2f7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12919
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Compare command, when not supported natively by the underlying bdev
is emulated by the bdev layer.
Change nvmf ctrlr data to advertise compare command by default.
Signed-off-by: Alexis Lescouet <alexis.lescouet@nutanix.com>
Change-Id: I88646e6c1a7d7a2829be813ff0241661724bd127
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12918
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Fused compare_and_write operation is always advertised by the nvmf
transport.
Add the fuses structure to spdk_nvmf_ctrlr_data to make advertising
fused operation configurable.
Signed-off-by: Alexis Lescouet <alexis.lescouet@nutanix.com>
Change-Id: I73ee03dc8948f1d250cc0a8f0b8a3bde042a45e7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12917
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: John Levon <levon@movementarian.org>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
There are a few places we can replace existing license
text with SPDX license identifiers, that did not match
the auto-replacement script in the previous patch.
Make those replacements manually in this patch instead.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I258720c03bc2153d1c56a8adf6357f224b911c0b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12913
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in every source file. This patch replaces
the bulk of SPDK .c, .cpp and Makefiles with the BSD-3-Clause
identifier.
Almost all of these files share the exact same license text,
and this patch only modifies the files that contain the
most common license text. There can be slight variations
because the third clause contains company names - most say
"Intel Corporation", but there are instances for Nvidia,
Samsung, Eideticom and even "the copyright holder".
Used a bash script to automate replacement of the license text
with SPDX identifier which is checked into scripts/spdx.sh.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaa88ab5e92ea471691dc298cfe41ebfb5d169780
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12904
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: <qun.wan@intel.com>
Reflect that we are kicking the entire controller.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If5723a5f485745ef0a2456942b6df1d54133815b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12665
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
This function is really about re-arming all SQs for a poll group;
refactor to reflect this.
This is necessary ground-work before we can support multiple reactors in
vfio_user.c in interrupt mode.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I170fae2076fc80e742926cf448973671ac9e3bd9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12664
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Play it safe and add the same memory barrier in
nvme_pcie_qpair_process_completions() as for ppc64.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Change-Id: I7079b4769d30106387ef4549495a72b7fea6a77a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12879
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Prepare for the later patch, and make the later patch code clean
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I12b175c86a5245f38dc76fe2d3918ec4b30a475a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12830
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
If accel_tasks are used up, we should not directly return but give
an another chance to calc it directly.
Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I983b65d7dfff0fea3974682e886d2dcf309cd2c3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12841
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
For a C2HTermReq PDU, there's no associated tcp_req, so we need to check
it for NULL before dereferencing it.
Also, while here, moved some of the assignments to the declarations to
reduce the number of boilerplate lines.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iac05ef0ba605e2f40d0026ad1b131c28d29f7314
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12845
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
The timeout poller might still be registered when a qpair is destroyed
if we send C2HTermReq and then destroy the qpair before host terminates
the connection.
Fixes#2527
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I21acc147fdba3aaac66b0c6ed54e155195fe9816
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12844
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
We need to check that the given SQ is active (i.e. is currently mapped
into the process), so make the check the same as that in
poll_group_poll().
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ibd3babd7520f611f596f3bab15765fa13b4d6b99
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12663
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is better represented under the name vfio_user_ctrlr_intr().
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: Ic3fa0fe238fd8ce4930bfd3e34b9dbc1b935aa6e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12662
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
There's a non-zero cost to looking up the CQ; only call this function in
the poll path if we need to.
While here, we'll streamline the ctrlr-level check.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: I6bf123f759fcd856196f6613cb6c7d0219550136
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12660
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Rui Chang <rui.chang@arm.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This SGL type was missed in the original commit
that added the pretty printing.
Fixes: 4d9ab1e9a1 ("nvme: pretty print dptr")
Reported-by: Ramanjaneya Burugula <burugula@gmail.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ibc655db4e65009071f39f55f691c94a094cea0bc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12705
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Use the conventional huge-pages based spdk allocation scheme for the initiator
data-structures unconditionally.
Change-Id: I5baee7614e3ac9b5497b3d771dfddfbaa7fdf65b
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12687
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: I7d3804a84851753992af4a3a37b60dc6de0d22cb
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12780
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: Ie50c7421f991ad0474edba0e0f339180f7afee00
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12778
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Note that without ISAL or IAA a call to compress/decompress
will fail.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id20a08f6e61b9a51fa4a1634a5314e6ca18fa504
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12310
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Previously an error would have been completed twice.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ief645fc30754433398531c50357876e92804e4b5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12789
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Provide an interface to allow the caller to provide a proprely
formatted descriptor.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I5c397761f556361040ec962d61169459150b6494
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12703
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: Ia09368e426a83274d9c7fc90ed8b0391f4d0b67c
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12774
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch adds virtio_blk abstraction for custom transports,
with the 'vhost_user_blk' first one being used.
Added spdk_virtio_blk_transport_ops describing the nessecary
callbacks to be implemented by each transport.
Please use SPDK_VIRTIO_BLK_TRANSPORT_REGISTER to register the transport.
Transports can use virtio_blk_process_request() to process the
incoming I/O from their queues.
virtio_blk_create_transport RPC was added to create one of the
registered transports, possibly with custom JSON arguments.
Added 'transport' argument to vhost_create_blk_controller RPC,
to specify which transport should create the controller.
By default the vhost_user_blk transport is used.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic9d93a6e0f483796eb56b7174a678e41a6ea4808
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9540
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
These were deprecated in 2019, it's time to remove
support for them now.
Change-Id: I56dbaef56ff793e48441219e07dc6b02dda0b470
Signed-off-by: wanghailiangx <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12777
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
