Commit Graph

19729 Commits

Author SHA1 Message Date
Anil Veerabhadrappa
831773b220 nvmf/fc: delegate memory object free to LLD
'args' object in nvmf_fc_adm_evnt_i_t_delete() is actually allocated in
the FC LLD driver and passed to nvmf/fc in nvmf_fc_main_enqueue_event() call.
So this object should be freed in the LLD's callback function.

Change-Id: I04eb0510ad7dd4bef53fc4e0f299f7226b303748
Signed-off-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17836
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2023-04-28 09:08:48 +00:00
Michal Berger
abf212ea6a test/spdkcli: Wait until spdkcli_clear_config settled
Similarly to https://review.spdk.io/gerrit/c/spdk/spdk/+/17559, this
part may also be affected by:

Error in cmd: load_config /root/spdk/test/spdkcli/config.json (
load_config /root/spdk/test/spdkcli/config.json
request:
{
  "action_on_timeout": "none",
  "timeout_us": 0,
  "timeout_admin_us": 0,
  "keep_alive_timeout_ms": 10000,
  "transport_retry_count": 4,
  "arbitration_burst": 0,
  "low_priority_weight": 0,
  "medium_priority_weight": 0,
  "high_priority_weight": 0,
  "nvme_adminq_poll_period_us": 10000,
  "nvme_ioq_poll_period_us": 0,
  "io_queue_requests": 1024,
  "delay_cmd_submit": true,
  "bdev_retry_count": 3,
  "transport_ack_timeout": 0,
  "ctrlr_loss_timeout_sec": 0,
  "reconnect_delay_sec": 0,
  "fast_io_fail_timeout_sec": 0,
  "generate_uuids": false,
  "transport_tos": 0,
  "io_path_stat": false,
  "method": "bdev_nvme_set_options",
  "req_id": 31
}
Got JSON-RPC error response
response:
{
  "code": -1,
  "message": "RPC not permitted with nvme controllers already attached"
}

so make sure the nvme controller was fully detached.

Change-Id: Iaebed7b640d96fbabfc694dfebc5a725902caad2
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17850
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-28 09:05:36 +00:00
Michal Berger
18dfb63389 test/nvme: Lock FDP test to FDP-capable nvme only
Change-Id: I199394c54914c99448153134ceb6e67a9c003f94
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17773
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-28 09:03:36 +00:00
Michal Berger
258b7fbff2 test/nvme: Add helper functions to detect FDP-capable nvme
Change-Id: I817ffbfcb3bca154cad86ca70465a923610cbabb
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17772
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-28 09:03:36 +00:00
Michal Berger
29ba5b1b43 scripts/vagrant: Add support for configuring FDP per nvme
Change-Id: Id647a02b82f7ede25496bbbbc420ef7d13d8f9af
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17771
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-28 09:03:36 +00:00
Michal Berger
8b8a7a80f6 pkgdep/git: Bump QEMU to latest 8.0.0 release
Needed for testing latest FDP feature.

Change-Id: I83a0b46c716d6658efa4f2723c4c40e617f40cf7
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17770
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-28 09:03:36 +00:00
Michal Berger
e9d44aa39b scripts/vagrant: Remove support for spdk-5.0.0 fork ns config
But leave the shortcut for configuring nvme with a single namespace.

Change-Id: I0e5745db481b24ab813ec1e98426d709cde216fd
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17769
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Pawel Piatek <pawelx.piatek@intel.com>
2023-04-28 09:03:36 +00:00
Michal Berger
fa3f818b4e tests: Skip block devices marked as hidden
These devices don't come with their major:minor dev, hence they won't
pop up under /dev, i.e. are not really usable.

Change-Id: I49b39ccbedcdd1bfe37964819e15b769af22cab6
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17774
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2023-04-28 09:03:36 +00:00
Ankit Kumar
c976353be8 test/nvme: add test application to verify fdp functionality
TP4146 introduced support for flexible data placement, which is
a data placement directive.

This application will test the new I/O management commands,
write with directives, log pages and set/get features.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Change-Id: I2d68625d9a180afb5a6e85e59738c2713ce965a8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16521
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
2023-04-28 09:03:36 +00:00
Richael Zhuang
df4600f4c9 nvme_tcp: fix memory leak when resetting controllor
In failover test, it reports memory leak about tqpair->stats when
detaching a tcp controller and it failover to the other controller.

Because during resetting the controller, we disconnect the controller
at first and then reconnect. when disconnecting, the adminq is not
freed which means the corresponding tqpair and tqpair->stats are not
freed. But when reconnecting, nvme_tcp_ctrlr_connect_qpair will
allocate memory for tqpair->stats again which causes memory leak.

So this patch fix the bug by not reallocating memory for tqpair->stats
if it's not NULL. We keep the old stats because from user perspective,
the spdk_nvme_qpair is the same one.

Besides, when destroying a qpair, the qpair->poll_group is set as
NULL which means if qpair->poll_group is not NULL, it should be a
new qpair. So there's no need to check if stats is NULL or not if
qpair->poll_group is not NULL. So adjusting the if...else... in
_nvme_pcie_ctrlr_create_io_qpair.

Change-Id: I4108a980aeffe4797e5bca5b1a8ea89f7457162b
Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17718
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-27 11:00:03 +00:00
Ziye Yang
cb97b86081 env_dpdk: optimizing spdk_call_unaffinitized
Purpose: Reduce unnecessary affinity setting.

For some usage cases, the app will not use spdk
framework and already call spdk_unaffinitize_thread
after calling spdk_env_init().

Change-Id: I5fa8349913c4567ab63c5a01271e7b2755e53257
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17720
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2023-04-27 09:38:49 +00:00
Ziye Yang
c441e27023 bdev/rbd: Do not submit IOs through thread sending.
Currently, we send IOs to the main_td thread.
It is not needed, because all the read/write functions
provided by librbd are thread safe, so we can eliminate the
thread send messaging policy for read/write related functions.

And with this patch, users can observe the load balance
distribution of I/Os on each CPU core owned by spdk applications
through spdk_top tool.

In this patch, we did the following work:

1 Move rbd_open when create the bdev since we will create once.
2 Simplify the channel management.
3 Do not use thread send messaging to do the read/write I/Os.

According to our experiment results showed in
https://github.com/spdk/spdk/issues/2204

There will be more than 15% performance improvment in IOPS aspect
for different write I/O patterns, and it also addresses the I/O Load
balance issues.

Fixes issue: #2204

Change-Id: I9d2851c3d772261c131f9678f4b1bf722328aabb
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17644
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-27 09:35:04 +00:00
Anil Veerabhadrappa
3a5ebfb06d ut/fc: Cleanup transport cleanup tests
This change fixes the following assert in FC UT,

   transport.c:329: spdk_nvmf_transport_create: Assertion `nvmf_get_transport_ops(transport_name) && nvmf_get_transport_ops(transport_name)->create' failed.

Change-Id: I57a9c6e83f07656b207d74bbadeb82e5efb5fa32
Signed-off-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17717
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-27 09:33:39 +00:00
Anil Veerabhadrappa
665e3805f8 ut/fc: add missing spdk_mempool_lookup stub
This patch fixes the following error:
        fc_ut.o: In function `nvmf_transport_create_async_done':
        spdk/lib/nvmf/transport.c:203: undefined reference to `spdk_mempool_lookup'
        collect2: error: ld returned 1 exit status

Change-Id: I6e81a8d62cfcc70bed6efe6ac807739d77ef89aa
Signed-off-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17716
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-27 09:33:39 +00:00
Krzysztof Karas
3edc534216 vhost_blk: make sure to_blk_dev() return value is not NULL
Assert that return pointer of to_blk_dev() is not NULL,
before dereferencing it.

Change-Id: I15adeac0926f23f84fdb3af88fc15ac07c580d91
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17536
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2023-04-27 09:30:42 +00:00
Krzysztof Karas
50e3b7bf31 nvme_transport: return NULL if transport does not exist
spdk_nvme_ctrlr_get_registers() calls nvme_get_transport()
to get a reference for a transport, whose registers should
be returned, but nvme_get_transport() explicitly returns
NULL, if the transport does not exist. This would result
in dereferencing a NULL pointer on line 862.

To remedy that, if no transport was found, return NULL.

Additionally change "THis" to "This" on line 46.

Change-Id: I3944925659991e9424e2177b5c940b2e2626d1f4
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17532
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2023-04-27 09:29:59 +00:00
Jim Harris
4fafd3fe65 test/nvmf: fully re-enable host/timeout.sh
Now that the bug with the remove_listener path has been
fixed, we can re-enable this part of the test.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I249011b20ffe468ed499766e4333e7bf9007a962
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17797
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>
2023-04-27 09:28:38 +00:00
Jim Harris
baf250e5e4 nvmf: initialize trid param in get_***_trid paths
When removing a listener, for example with
nvmf_subsystem_remove_listener RPC, we use the concept of a
"listen trid" to determine which existing connections
should be disconnected.

This listen trid has the trtype, adrfam, traddr and trsvcid
defined, but *not* the subnqn.  We use the subsystem pointer
itself to match the subsystem.

nvmf_stop_listen_disconnect_qpairs gets the listen trid
for each qpair, compares it to the trid passed by the
RPC, and if it matches, then it compares the subsystem
pointers and will disconnect the qpair if it matches.

The problem is that the spdk_nvmf_qpair_get_listen_trid
path does not initialize the subnqn to an empty string,
and in this case the caller does not initialize it either.
So sometimes the subnqn on the stack used to get the
qpair's listen trid ends up with some garbage as the subnqn
string, which causes the transport_id_compare to fail, and
then the qpair won't get disconnected even if the other
trid fields and subsystem pointers match.

For the failover.sh test, this means that the qpair doesn't
get disconnected, so we never go down the reset path
on the initiator side and don't see the "Resetting" strings
expected in the log.

This similarly impacts the host/timeout.sh test, which is
also fixed by this patch.  There were multiple failing
signatures, all related to remove_listener not working
correctly due to this bug.

While the get_listen_trid path is the one that caused
these bugs, the get_local_trid and get_peer_trid paths
have similar problems, so they are similarly fixed in
this patch.

Fixes issue #2862.
Fixes issue #2595.
Fixes issue #2865.
Fixes issue #2864.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I36eb519cd1f434d50eebf724ecd6dbc2528288c3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17788
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Mike Gerdts <mgerdts@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: <sebastian.brzezinka@intel.com>
2023-04-27 09:24:18 +00:00
Mike Gerdts
c0ea96cf5e vbdev_lvol: allow degraded lvols to be deleted
An esnap clone is now deletable when its external snapshot is missing.
Likewise, the tree of degraded lvols rooted at a degraded esnap clone
can also be deleted, subject to the normal restrictions.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I711ae25d57f5625a955d1f4cdb2839dd0a6cb095
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17549
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
5b250c0836 vbdev_lvol: load esnaps via examine_config
This introduces an examine_config callback that triggers hotplug of
missing esnap devices.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I5ced2ff26bfd393d2df4fd4718700be30eb48063
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16626
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
5e79e84e78 include: add libgen.h to stdinc.h
A subsequent patch will need to use dirname(3), declared in libgen.h.
Because libgen.h is a POSIX header, the SPDK build requires that it is
defined in spdk/stdinc.h, not in the file that needs it.

libgen.h also declares basename() which has a conflicting declaration in
string.h. A small change is required in bdev_uring_read_sysfs_attr() to
accommodate this.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: Ib4ded2097881668aabdfd9f1683f933ce418db2e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17557
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
d453aaa360 vbdev_lvol: degraded open of esnap clones
If an esnap clone is missing its snapshot the lvol should still open in
degraded mode. A degraded lvol will not have a bdev registered and as
such cannot perform any IO.

Change-Id: I736194650dfcf1eb78214c8896c31acc7a946b54
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16425
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
a045d8d2fc vbdev_lvol: early return in _vbdev_lvs_remove
This replaces nested if statements with equivalent logic that uses
early returns. Now the code fits in 100 columns and will allow the next
patch in this series to avoid adding a fifth level of indentation.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: Ief74d9fd166b2fe1042c78e12fe79d5f325aa502
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17548
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
f3c14b8dee vbdev_lvol: add bdev_lvol_get_lvols RPC
This provides information about logical volumes without providing
information about the bdevs. It is useful for listing the lvols
associated with specific lvol stores and for listing lvols that are
degraded and have no associated bdev.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I795161ac88d9707831d9fcd2079635c7e46ecc42
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17547
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
a67e0eb37e vbdev_lvol: external snapshot rpc interface
Add RPC interfaces for creation of esnap clone lvols. This also
exercises esnap clone creation and various operations involving
snapshots and clones of esnap clones to ensure that bdev_get_bdevs
reports state correctly.

Change-Id: Ib87d01026ef6e45203c4d9451759885a7be02d87
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14978
Reviewed-by: Michal Berger <michal.berger@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
3f52d2659e test/common: allow tests to use set -u
Now autotest_common.sh is tolerant of tests that use "set -u" so that
they quickly generate useful errors when variables are used but not set.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I5d7709f3029fa8f52affecf68a4b9da97a84589d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16703
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
9b8f2ef354 test/lvol: test esnap clones with real bdevs
This adds test/lvol/esnap for functional tests lvol esnap clone bdevs
without RPCs or reactors.

Change-Id: If62b1bde2b19343af51ba4c11599623556484b0d
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16705
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
54b4f4dd4b vbdev_lvol: allow creation of esnap clones
This adds the ability for create esnap clone lvol bdevs.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: Ifeef983430153d84d896d282fe914c6671283762
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16590
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
0c31b86a6f vbdev_lvol: create esnap blobstore device
Register an spdk_bs_esnap_dev_create callback when initializing or
loading an lvstore. This is the first of several commits required to add
support enable lvol bdevs to support external snapshots and esnap
clones.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I35c4e61fdbe5b93d65b9374e0ad91cb7fb94d1f4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16589
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
0cea6b57f6 lvol: add spdk_lvol_get_by_* API
spdk_lvol_get_by_uuid() allows lookup of lvols by the lvol's uuid.

spdk_lvol_get_by_names() allows lookup of lvols by the lvol's lvstore
name and lvol name.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: Id165a3d17b76e5dde0616091dee5dff8327f44d0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17546
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
b7d84562cb lvol: add spdk_lvol_iter_immediate_clones()
Add an interator that calls a callback for each clone of a snapshot
volume. This follows the typical pattern of stopping iteration when the
callback returns non-zero.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: If88ad769b72a19ba0993303e89da107db8a6adfc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17545
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
30399f312c lvol_ut: test esnap hotplug
This exercises spdk_lvs_esnap_notify_hotplug() under a variety of happy
and not-so-happy paths.

Change-Id: I1f4101a082b113dacc7d03f81ca16069acfb458d
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17602
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
712f9aa452 lvol: hotplug of missing esnaps
This introduces spdk_lvs_notify_hotplug() to trigger the lvstore to call
the appropriate lvstore's esnap_bs_dev_create() callback for each esnap
clone lvol that is missing the device identified by esnap_id.

Change-Id: I0e2eb26375c62043b0f895197b24d6e056905aa2
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16428
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
f2dbb50516 lvol: keep track of missing external snapshots
If an lvol is opened in degraded mode, keep track of the missing esnap
IDs and which lvols need them. A future commit will make use of this
information to bring lvols out of degraded mode when their external
snapshot device appears.

Change-Id: I55c16ad042a73e46e225369bfff2631958a2ed46
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16427
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
87666f5286 blob: esnap clones are not clones
spdk_blob_is_clone() should return true only for normal clones. To
detect esnap clones, use spdk_blob_is_esnap_clone(). This also clarifies
documentation of spdk_blob_is_esnap_clone() to match the implementation.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I9993ab60c1a097531a46fb6760124a632f6857cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17544
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
8b3dcd6191 blob: add is_degraded() to spdk_blob_bs_dev
The health of clones of esnap clones depends on the health of the esnap
clone. This allows recursion through a chain of clones so that degraded
state propagates up from any back_bs_dev that is degraded.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: Iadd879d589f6ce4d0b654945db065d304b0c8357
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17517
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
09bf2b2092 blob: add spdk_blob_is_degraded()
In preparation for supporting degraded lvols, spdk_blob_is_degraded() is
added. To support this, bs_dev gains an optional is_degraded() callback.
spdk_blob_is_degraded() returns false so long as no bs_dev that the blob
depends on is degraded. Depended upon bs_devs include the blobstore's
device and the blob's back_bs_dev.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: Ib02227f5735b00038ed30923813e1d5b57deb1ab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17516
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-04-26 17:32:13 +00:00
Mike Gerdts
1db33a8f74 blob: add spdk_blob_get_esnap_bs_dev()
While getting memory domains, vbdev_lvol will need to be able to access
the bdev that acts as the lvol's external snapshot. The introduction of
spdk_blob_get_esnap_bs_dev() facilitates this access.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I604c957a468392d40b824c3d2afb00cbfe89cd21
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16429
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-04-26 17:32:13 +00:00
Konrad Sztyber
e3babb2be1 accel_perf: use accel stats when dumping results
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iae1128ce01c16731bced8f97c08f44e1b0bc83f2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17626
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-04-26 11:15:40 +00:00
Konrad Sztyber
55d6cc0eae accel: add method for getting per-channel opcode stats
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ic3cc0ddc5907e113b6d9d752c9bff0f526458a11
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17625
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2023-04-26 11:15:40 +00:00
Konrad Sztyber
d7b29fb9d5 accel: collect stats on the number of processed bytes
For operations that have differently sized input/output buffers (e.g.
compress, decompress), the size of the src buffer is recorded.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I1ee47a2e678ac1b5172ad3d8da6ab548e1aa3631
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17624
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-04-26 11:15:40 +00:00
Konrad Sztyber
7c621ff206 accel: specify number of events when updating stats
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I5b611c8978b581ac504b033e1f335a2e10a9315b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17623
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 11:15:40 +00:00
Konrad Sztyber
0de931dc6b accel: move accel_get_iovlen() up
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I6117057a1e3812386a0fb7a10e07978415a48261
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17622
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2023-04-26 11:15:40 +00:00
Konrad Sztyber
9a377ecb22 accel: append support for crc32c
It is now possible to append an operation calculating crc32c to an accel
sequence.  A crc32c operation needs special care when it's part of a
sequence, because it doesn't have a destination buffer.  It means that
we can remove copy operations following crc32c only when it's possible
to change the dst buffer of the operation preceding crc32c.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I29204ce52d635162d2202136609f8f8f33db312d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17427
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-04-26 11:15:40 +00:00
Konrad Sztyber
2b1ad70c4c accel: check operation type in accel_task_set_dstbuf()
This will reduce the amount of changes in the following patch which
makes this function recursive.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: If8da6ae52d78358b66b2d9303413a9723687a767
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17568
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-04-26 11:15:40 +00:00
Mike Gerdts
b0c93eb3fb accel: destroy g_stats_lock during finish
g_stats_lock is an spdk_spin_lock that is initialized as the module is
loading. With this change, it is destroyed as the module finishes.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I5263547f6d0e8981765d59665bd826cf07a6f83e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17681
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-04-26 11:06:02 +00:00
Konrad Sztyber
bade2d8db5 accel: delay finish until all IO channels are released
This ensures that there are no more outstanding operations, so we can
safely free any global resources.

Fixes #2987

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iac423b4f2a1183278d1db20f96c1a3b1bb657f85
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17767
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Mike Gerdts <mgerdts@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-26 11:06:02 +00:00
Jim Harris
e407385e03 env_dpdk: add ERRLOGs to help debug issue #2983
Issue #2983 shows a case where we seem to get a
device remove notification from DPDK (via vfio
path) after we have already detached the device
explicitly by SPDK.

This issue has proven difficult to reproduce
outside of the one observed failure so far, so
adding a couple of ERRLOGs into this path to help
confirm the this theory should it happen again.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0fda4229fe150ca17417b227e8587cd7fbda6692
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17631
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-25 16:54:59 +00:00
Michal Berger
aadd13f444 scripts/pkgdep: Add support for rocky|centos 9
Also, shuffle DAOS pieces a bit to keep repo handling in one place.
Also, also switch ceph repo to an actively supported release, common
and available for both centos|rocky 8|9 (i.e. pacific).

Change-Id: Idb19e4a5ff80770c7d6f9e6db85f983e163958e6
Signed-off-by: Michal Berger <michal.berger@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17661
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2023-04-25 11:26:35 +00:00
Pawel Piatek
64c27c8dcc scripts/vagrant: upload optional directories
Sometimes we need to copy additional directories with
sources into VM. Currently, two cases are known:
- spdk-abi
- dpdk (for CI vs-dpdk jobs)

Signed-off-by: Pawel Piatek <pawelx.piatek@intel.com>
Change-Id: I242838364d649b29a5a9dc720c6920493b061fa7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17645
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-04-25 10:13:18 +00:00