Some of the unit tests were only closing a blob,
without deleting it at the end of unit tests.
This is valid behaviour, but to later simplify
the setup/cleanup - a delete blob was added.
This change was made to unit tests where,
it was not focus of particular UT.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I7e338ba6fcb63af26658e8968971a6d56f70807b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1203
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Operation of closing and deleting a blob is common
throughout the tests.
This patch creates helper function ut_blob_close_and_delete(),
to do exactly that with the right asserts.
Throughout the UT this function replaces all instances where
blob is closed and deleted, when it is not the subject of
particular unit test (focus is on other functionality).
It will allow to later use it in setup/cleanup functions
for UT suite - similar to bs init.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic56594da8cd4c442b62890d62961b44771059679
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1202
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Operation of creating a blob (with or w/o opts),
then opening it through out the tests is quite common.
This patch creates helper function ut_blob_create_and_open(),
to do exactly that with the right asserts.
Throughout the UT this function replaces all instances where
blob is created and opened, when it is not the subject of
particular unit test (focus is on other functionality).
It will allow to later use it in setup/cleanup functions
for UT suite - similar to bs init.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ia541b855d154503f824bfa3909bcfeac649c5853
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1201
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Always call API on blobid that was received from blob
create, instead of relying on global one.
Where necessary added declaration and assignment.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I3529b4d5e86c1301bb64268029a57873f9f600cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/944
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Always call API on pointer that was received from blob
open, instead of relying on global one.
Where necessary added declaration and assignment.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I15531111201d071c471f5be79ce18ff508631677
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/943
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
A lot of unit tests require only creation of a blobstore
then unload it at the end.
This code can be made common and invoked automatically by
CUnit with CU_add_suite_with_setup_and_teardown().
Those functions are called before and after each test in suite.
This should simplify the unit tests by removing the common
code.
Following patches will add further suites with different
setup/cleanup to reduce common code even more.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I984cde2381f4461e188e06050355024a88ebfca9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/932
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Next patch in series adds setup/cleanup functions
that use g_bs for its operation.
Particular unit tests should not leave the g_bs
unassigned, if cleanup is supposed to unload it.
It applies to helper functions to reload/dirty_load.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6e022b6c762981c2c70ebf31f4981a17f54f3591
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1182
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Some of the unit tests initalized bs opts,
event if they were left unmodified.
There is no need to perform this operation,
as spdk_bs_init() can accept NULL as bs_opts parameter
to just initialize default values.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iacea1fa9d53796b8c359b611092cc63c6bce4094
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1181
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This patch adds better check if the xattr was removed.
It will identify potential bugs when first md sync
takes precedence over the second md sync.
As per suggestion in
https://review.spdk.io/gerrit/c/spdk/spdk/+/774/6/test/unit/lib/blob/blob.c/blob_ut.c#7655
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I463f2455614ef11cc1512dc2fb1972ba1f024337
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1109
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Operation of dirty shutdown of blobstore and reloading
it occurs often enough in UT to provide a common function
to perform it.
Added ut_bs_dirty_load() to facilitate this.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I0079dfabd64eaec6495db02fe200be1d6116f0c7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1092
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Operation of unloading and loading blobstore
occurs often enough in UT to provide a common function
to perform it.
Added ut_bs_reload() to facilitate this.
Couple occurences in this patch actually fix
test cases where mistakenly opts were not passed.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib46ed395cd134feaa540e00a334ae861872b3ef4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1091
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Couple unit tests were still refering to g_bs pointer,
rather than the one local to particular UT.
This patch fixes this.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: If343eb561a48c4547d499139d76e1db6b2d7f3bf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1090
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Always call API on pointer that was received from blobstore
initialization or load, instead of relying on global one.
Where necessary added declaration, assignment and asserts.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic4198ac984c13a8b77428deaa32ce9864e6750c1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/942
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
CUnit provides a helper macro CU_ADD_TEST() that
simplifies using CU_add_test() function.
Test name no longer needs to be provided,
instead test is named after test function name.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I9f5d9e0d8e72ab36b2a5c04e33c336e6c0a1fa6a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/931
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Failure to prepare CUnit for tests is always a fatal error,
and application exits.
Most test managment functions in CUnit set some CU_ErrorCode,
that can be retrived by return code from a function or CU_get_error().
Thus usual structure of UT in SPDK checked every output from every
test managment function.
There is a helper function CU_set_error_action() that when set to
CUEA_ABORT, will abort the application run with proper error return code.
Along with nice error message.
For example if one were to add same UT twice:
"Aborting due to error #32: Test having this name already in suite."
Application return code set to 32.
Using CU_set_error_action(CUEA_ABORT), removes the need for error checking
on test managment functions in CUnit.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ida7de5e040b714509e79957d47a006ee518a42b8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/930
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Expands blob_simultaneous_operations to cover case
when second md sync is perfomed without waiting for the first
one to complete.
The blob state is forced in UT, but in normal usage could be
changed by modifying xattr, allocating thin clusters or other.
Related to #1170
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I880d289cdf957d4513c080efdcb7a5d8c6b3b447
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/775
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This new UT is checking validity of md sync being run
along another md sync with changed metadata.
At this time added case where first md sync is being
interrupted by removal of xattr with second md sync.
This interruption is perfomed at increasing number of
poller executions, until the number is enough to
complete first md sync.
There are two expected states of used_md_pages array,
either with xattr or without. The state is verified
after md syncs.
This UT will be expaned in similar manner by other
operations (than xattr) that casue changes in persisted metadata.
Related to #960
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I251dca92ffc3080d8dc503a7f1ff342aa59adef9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/774
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Making use of recently added utility to poll exact number of times.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I23a9dc702c899285626522ace8ec77a9ba0f7757
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/773
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
spdk_put_io_channel() was designed to be called on the same thread
that called spdk_get_io_channel(). spdk_put_io_channel() sends a
message to its own thread, to allow the context to unwind before
releasing the resources. This had the side effect to allow an
incorrect thread to call spdk_put_io_channel(). This patch will fix
that.
Bdevperf tool had a design flaw that needed the side effect, but
it was fixed recently. We do not know if we have any other case.
Hence add assert to spdk_put_io_channel() to find other case.
We found that unit test for blobstore had called
spdk_put_io_channel() and fix it together in this patch.
Besides, correct the comment for spdk_put_io_channel() in
include/spdk/thread.h not to create any other case in future.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I6ec7bf074818abef43b23ca40bc9385adac70a75
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479390
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Extent table and extent page descriptors are now
set to be default way clusters are serialized on disk.
With this patch UT are ran with and without
extent table.
Changed two asserts in test, since amount is dependent on
which type of serialization is used.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ica58fce6a4effd014d7dd40ee26edd0fa3196d0f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/481901
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
This will be used to add another run of whole UT suite
with extent pages on/off.
Next patch in series will be enabling both types of
extent serialization for all UT.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ia8b4b8822edefb90ffc13cf777885f9af95e4545
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482170
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
All the non-ext version of the call is doing, is calling
ext with NULL as opts. Then default opts are used in
its place.
This change was facilitated by next on in series,
where all blob opts will be initalized in UT
with parameter use_extent_table either set to
true or false.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I62b642c1808b38a5f7c94a5900f25f4978a4ec39
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482859
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
After creation of blobs in both tests, only clusters indexed from
0 to 10 are supposed to be used. Index 0 for md and 1-10 for data
of single blob since it was create thick provisoned.
Cluster allocations are done in order so if there was a bug for
overflow amount of clusters claimed, first in order would be
one with index 11.
This patch adds asserts after each bs load for first data cluster
that is supposed to be used and for first data cluster that is
not.
During the tests those should remain constant.
When creating/deleting snapshots, the blobs are affected
by changing their type to/from thin_provisioned.
Added asserts to verify their state at every blob open.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I38418da55850d5b8468e578b3c42c5b817ae8045
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482661
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
g_bserrno from blob deletion or snapshot creation,
should not be checked. It is implementation
dependent whether the error (or success) from those
calls actually means that enough data was persisted
on disk.
This test case should work even if we set the threshold
high enough that no failed opperations occur.
On the other hand some parts of those calls do cleanup
in them, meanwhile there is enough metadata data on disk already.
Such as cleaning up unused clusters or pages issue
writes, but at that point the blobs already are in expected
state.
Thus removed assert for g_bserrno, as failure is not indicative
of impossibility to recover.
While here, removed the spdk_bs_unload(). This UT are for
testing power fail safety. Never should it be the case that
enough writes occured in create/delete, but blobs are not
in the expected state.
When such bug would be introduced, it could be covered up
by spdk_bs_unload() cleanly closing up the blobstore.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic69c3061f2cc1fe04bf895632cdb11efb2fe6912
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/482660
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Extent Pages claim and insertion can be asynchronous
when cluster allocation happens due to writing to a new cluster.
In such case lowest free cluster and lowest free md page
is claimed, and message is passed to md_thread.
Where inserting both into the arrays and md_sycn happens.
This patch adds parameters to pass the Extent Page offset
in such case.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I46d8ace9cd5abc0bfe48174c2f2ec218145b9c75
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479849
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Accept a clear method option on blob create by adding clear_method
to the opts structure passed in to _spdk_bs_create_blob(). Store
these 2 bits in md_ro_flags so that earlier versions without an
understanding of these bits can not alter metadata.
The new metadata values will be used later in the series.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I5440645ca20b426778d13b2e544b65dc2b3b83c7
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472204
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Previously this UT was only checking the snapshot counts.
This was correct, but since the blobs in this tests are
clones of each other - it would make sense to check it as well.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id9dec7f0a292cbb52da8196bfc77679fc0f49d8b
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478976
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
For deletion to actually deliver results, poll_threads()
has to be called.
g_bserrno came from previous API call, not delete itself.
This patch fixes it so that g_bserrno is from the delete itself.
Next patch in series adds checks for number of clones.
Having such checks in first place,
would allow to notice lack of poll_threads() here before.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I3477f5c5b99106b70ee3d5706906a4dc94283180
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478975
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch fixes minor unit test deallocations around io_channels.
It has not exposed any issues in blobstore code, but should always be right
to prevent covering up any issues later.
1) Two spdk_bs_free_io_channel() were missing
2) Dirty shutdown should still free bs resources with _spdk_bs_free()
Sample log at the end of UT, before this patch:
thread.c: 200:_free_thread: *ERROR*: thread 0x617000000080 still has channel for io_device blobstore
thread.c: 200:_free_thread: *ERROR*: thread 0x617000000080 still has channel for io_device blobstore
thread.c: 200:_free_thread: *ERROR*: thread 0x617000000080 still has channel for io_device blobstore
thread.c: 180:spdk_thread_lib_fini: *ERROR*: io_device blobstore not unregistered
thread.c: 180:spdk_thread_lib_fini: *ERROR*: io_device blobstore not unregistered
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I187bc61bb6e094c9c740a987e7d14760551a0503
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475872
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Previously blob_insert_cluster_msg UT specified the cluster
which will be used to store LBA. So for blob the cluster [1],
was always stored as cluster 0xF in blobstore.
This patch changes _spdk_bs_claim_cluster() with preselected
cluster in blobstore to _spdk_bs_allocate_cluster() that
will choose first free one in the map.
It will help in further patches that add more logic
on extent table and extent pages in _spdk_bs_allocate_cluster().
Otherwise this patch 'hard coded' values might get more complex
than needed.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I238329b2191e7ebd3f73c1fe85d7e1da0249979e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475493
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
When serializing extents, run-length encoding is supposed to
1) RLE all sequential LBAs
2) RLE zero LBAs (unallocated)
There is one special case, with sequential LBAs that start
with 0 LBA. This is RLE as 1) case, but results in descriptor
matching case 2). Which causes loss of allocated clusters.
This requires following conditions to be met:
- blobstore has just a single cluster reserved for MD
- blob is thin provisioned
- first allocation occurs on cluster_num=1
For last part to be true, very first write for blob has to be
issued to LBA between cluster_size and 2*cluster_size.
Causing allocation of second cluster in blobstore and assiging
it LBA equal to number of LBAs per cluster.
To fix this, case 1) disallows to RLE zeroes.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I136282407966310c882ca97c960e9a71c442c469
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475494
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This was inconsistent on all CU_ASSERT macros,
so now it is fixed.
Change-Id: I6d20f98528c4041baf02cb419ab9644a45b807c4
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459760
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Blob id only is matched to the very first page of md for
that particular blob.
During loading blobstore, we shouldn't verify
further pages in chain against the blobid.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifc7863ddcb403aedc264c14e6b4c3915bd30dc41
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460607
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Length of xattr descriptor is equal to length of xattr struct,
xattr name and the len of stored value.
There is no limit to how much can be stored in memory for xattr.
On disk xattr size is limited to single page and within that to
max descriptors that can fit in it.
This size is known at compile time.
Before this patch it was possible to add xattr exceeding
what was possible to be written to disk. This caused issues
when serializing the metadata during spdk_blob_sync_md()
or spdk_blob_close(). Making those fail without specific info
to the user and not actually writting such descriptor.
Since maximum length of xattr descriptor is known at compile time,
this patch compares against this value when setting the xattr.
It will immediately report back to user with error, and will
not store xattr in memory (thus not serialize it).
This patch should not affect any backward compatibility for blobs.
Too large xattrs weren't written to disk before,
API for blobstore stays the same - only reporting ENOMEM
when it should.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6f4af4d079e47f084e20d7a4969d9a78ec1f8610
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460450
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Currently we are missing cleanup routine for case when
power failure interrupts creating snapshot. This patch
add such routine.
For the case where we find blob with a parent snapshot ID
matching newly created snapshot we can finish whole process
during recovery by processing forward with setting snpashot
as read only, removing xattr and syncing. We should remove
snapshot only if there is no blob with parent pointing at
snapshot.
Fixes github issue #760
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I2f0e298164e07a2b4dfa5367e8878facef640702
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455216
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This is unit test for power failure event during snapshot
deletion. At the same time this is an example how to
use new power failure event functionality in unit tests.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I9212392f665576fa16edd28c609199d0e02dc434
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454608
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Starting with this patch it is possible to remove a
snapshot if there is only a one clone created from it.
In such case snapshot can be removed without any data
copying. This is achieved with following steps (case
with only one clone):
1. Open snapshot (Snapshot1) that shall be removed
2. Check if the Snapshot1 has no more than 1 clone (Clone1)
3. Remove Clone1 entry from Snapshot1
4. If the Snapshot1 has a parent snapshot (Snapshot2):
4a. Add Clone1 entry to the Snapshot2 clones list
4b. Remove Snapshot1 entry from Snapshot2 clones list
5. Open Clone1 blob
6. Freeze I/O operations on Clone1
7. Temporarily override md_ro flag for Snapshot1 and Clone1
for MD modification
8. Merge Snapshot1 and Clone1 clusters maps into Clone1
clusters map
9a. If Snapshot2 is present switch parent ID and backing
bs_dev on Clone1
9b. If Snapshot2 is not present set parent ID to
SPDK_BLOBID_INVALID and backing bs_dev to zeroes_dev
10. Sync MD on Clone1
11. Sync MD on Snapshot1
12. Restore MD flags for Clone1 and Snapshot1
13. Unfreeze I/O on Clone1
14. Close Clone1 blob
15. Remove Snapshot1
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I800724b981af894e01e1912d0077c5b34a2ae634
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/445576
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
In some cases user may want to flag blob for removal
then do some operations (before removing it) and while
it happens there might be power failure. In such cases
we should remove this blob on next blobstore load.
Example of such usage is delete snapshot functionality
that will be introduced in upcoming patch.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I85f396b73762d2665ba8aec62528bb224acace74
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453835
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This patch verifies that functionality of only releasing cluster
that was originaly claimed in race thread condition.
Change-Id: I4d60ffdb4fd6612d61a761835aae2fd1bf81b7fd
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452626
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Block operation that should not be done simultaneously.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I3cab510377a49be4e5847ba37a6218f0025c0db6
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450014
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
1,add a judgement to ensure spdk_blob_close() runs correctly.
2,make sure the command has been completed before _spdk_bs_free()
which would avoid heap-use-after-free.
Change-Id: I698521d8d4d8e1d8a7374665d27d6eec6d0b85f5
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449517
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
spdk_thread_poll()
This is an optimization if the calling function already knows the
current time.
Change-Id: I1645e08e7475ba6345a44e0f9d4b297a79f6c3c2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/443634
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Now that spdk_thread_poll can be called from within
a thread, some of the unit tests can be a bit simpler.
Change-Id: Ib7b92c3f59862e5c128f5e0cc8a08f65e2b89e93
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/441160
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
When last clone of a snapshot is being deleted
we remove that snapshot from snapshots list.
We should not do that as it still works as a
snapshot and it is read-only, but it does not list
as a snpashot from get_bdevs. Instead remove snapshot
entry from the list when blob that represents that
snapshot is being removed.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I8d76229567fb0d9f15d29bad3fd94b9813249604
Reviewed-on: https://review.gerrithub.io/436971
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I994b5d46faffd34430cb39e66225929c4cba90ba
Reviewed-on: https://review.gerrithub.io/414935
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Valgrind found this on my system. The unit test itself
was passing correctly (we wrote uninitialized data and then
successfully read back the same uninitialized data) - but
Valgrind doesn't like code that makes any decisions based
on reading uninitialized memory.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I762f5dd0cb368328bd5eba9cecf7db404b457fad
Reviewed-on: https://review.gerrithub.io/424109
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This patch adds UT for IOV split operation.
Next patch will refactor split of vectored read/write a little.
Change-Id: I77fbaf042fd894b23ad5ae846f54c28498e8e62b
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/422768
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Unify the regular mocks and the pointer mocks. Simplify
several of the #defines.
Change-Id: Ica8c69dbb70a685a55b5961b73fd7872f451c305
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/418884
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Previously cluster sizes only set as default were tested,
this patch adds test for load when different cluster_size is used.
Change-Id: I585adff25c81de809435ef235314cd4b8b116261
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/420607
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
It is not an error if bdev is smaller than cluster size so convert this
error to infolog. This fixes false error message dring examine process.
To return proper error message when creating blobstore using RPC the
_spdk_bs_alloc was adjusted to return errocode that is propagated up and
converted to "No space left on device".
Fixes#316
Change-Id: Ic9803720a55125fcfa34263346f2d9e1aae03a53
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-on: https://review.gerrithub.io/420054
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This patch prevents to copy cluster data when there is not
backing blob to improve cluster allocation performance
in thin provisioned blobs.
Change-Id: Ie766d2e5274daa74c2b13b2198a20205e3417467
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/417938
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Before this patch when serializing extents,
unallocated clusters were treated as separate lba.
This caused metadata to grow without need.
Change-Id: I5d66466dda5f5e6d4d53f4ed5bd0bac18c74be96
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/419180
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This patch adds more intensive unit tests for reading and
writing splitted operations.
Change-Id: I64921fa31689e660281674e8b2ad4253faa7cf04
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/410031
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Save the original size of the disk to metadata when it is first created.
On load verify that the disk did not change size.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I535940ee188425ee3b394effd99653cc073d541e
Reviewed-on: https://review.gerrithub.io/410896
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch adds an API to decouple blobs parent removing dependency
on it. Blob stays thin after this operation.
Also unit tests for blobstore inflate are improved and reused with
decouple parent functionality.
Change-Id: I96dfee467c78cf4f4d929ec7bc05263f7a23a8aa
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/410829
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When snapshot is created from a clone, clones parent is not
inherited.
This patch also updates unit tests covering this case.
Change-Id: I42eb00fe2f33504c5b5c5beded20c4ea65eaff67
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/414804
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I6182eb3a77d23db7088703492d71349e3a4b6460
Reviewed-on: https://review.gerrithub.io/399366
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The patch disables writing dirty bit during blobstore loading.
Instead, dirty bit is written prior to the first metadata update.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I7be81009a99f09048bf23749c8f6ef5e9f7b3751
Reviewed-on: https://review.gerrithub.io/410884
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I26e9d1e86c93f0b96b5bff7c9af5a453199c3e29
Reviewed-on: https://review.gerrithub.io/408138
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Inflate call can be used on thin provisioned blob or clones.
Function allocates all unallocated clusters on specified blob and:
- For clones, copies data from backing blob.
- For thin provisioned blobs, clusters are zeroed.
After this call all dependency from specified blob is removed
what allows deletion i.e. snapshots.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Change-Id: Ibff569e45b12068b2fb46557156be348b36c252b
Reviewed-on: https://review.gerrithub.io/399367
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Thin provisioned blobs mark unallocated clusters with
cluster ID 0. During recovery from a dirty shutdown,
we must not try to claim cluster 0 - we should ignore
them instead.
Fixes issue #291.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If0dd42416f5de8d9972073bf6ed44eb8bc655415
Reviewed-on: https://review.gerrithub.io/410065
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Change-Id: I3f9e54774951e691378e0659dcf15cecacd77e2c
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/405026
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
strncpy is going to be added to the list of banned functions because
it does not guarantee strings are null terminated.
Change-Id: I73adb5d3eea5c95ea705c67de4fc967e926eb6ce
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/407025
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Change-Id: Iad67be79d0ddd8c498950c4f7b1b3203e47a7a41
Reviewed-on: https://review.gerrithub.io/393936
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch adds new feature of blobstore.
New call creates a read-only snapshot of specified blob with provided options.
NOTE:
This patch doesn't cover recovery operation if snapshotting fails. This operation
will be implemented and added later.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Change-Id: I470ca13525638fa6df485d508b3adf71b6b69c0b
Reviewed-on: https://review.gerrithub.io/393935
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
To support resize operations during I/O, we will need
to send messages to each thread to quiesce I/O while
the resize operation is in progress to guard against
the cluster map memory changing while another thread
is accessing the cluster map.
Therefore, spdk_blob_resize needs to be asynchronous.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ida037334739b4b80a1dbc76e8f1c70bca8b73582
Reviewed-on: https://review.gerrithub.io/404616
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This file is accessed by files in multiple directories under test.
Change-Id: I634481fb58eab5c097aaece5289f88e531954fcb
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/404976
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Some error paths before _spdk_bs_alloc did not
destroy bs_dev.
After succesfull _spdk_bs_alloc, destroying is done
in _spdk_bs_free.
Change-Id: Ib69ae9707e12a646af80f7892af49cc4f79c199e
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/405223
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Blobstore supports only block lengths that are less than
SPDK_BS_PAGE_SIZE or when multiplication of block len
results in SPDK_BS_PAGE_SIZE.
This was checked only on spdk_bs_init(), but not spdk_bs_load().
When not checked, it caused issues with lvol store tasting.
During tasting, there is an attempt to perform spdk_bs_load()
on a given device.
It was possible to hit asserts in blobstore by creating
malloc with block size 8192.
Change-Id: I30b62bebad405b581eb2158925884adc616d9b92
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/404537
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
BlobFS shutdown path needs to be investigated more with these
changes.
This reverts commit a137b9afd0.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8b04b24e178945d62db20668b9e500f278ae955b
Reviewed-on: https://review.gerrithub.io/403600
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
For any given blob, if an spdk_blob_sync_md() operation
is already in progress, queue additional spdk_blob_sync_md()
operations until the previous one completes.
This ensures proper ordering of writing metadata to
disk.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2051e8cb5b8d1a033ec1238cb4811232110aa0f4
Reviewed-on: https://review.gerrithub.io/401257
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This patch adds possibility to set internal xattrs on blob
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I2d0f14558e4a1af7071ee1a4f59aaf9f14bea2d9
Reviewed-on: https://review.gerrithub.io/396418
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Unit tests implemented in following patches.
This is rebased patch from https://review.gerrithub.io/#/c/396648
merged as commit c1174e6895
and reverted in 0847f27b54.
Change-Id: I3d152bf7847c83bf75149edd61564c1f393927d8
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Reviewed-on: https://review.gerrithub.io/402529
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Before this patch super_blob id for blobstore was persisted
only during spdk_bs_unload. If power fail occurred after creating and
syncing blob, super_blob id was lost within blobstore.
Lvol store metadata would be lost, if proper shutdown
didn't occur in first SPDK instance run since creation of lvs.
This fix changes setting super blob to be instantly persisted
on disk in super block. Without affecting clean bit in super block.
Change-Id: I578f1fc8717e2d7968ad506fa4dead7507a5e0b4
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/398804
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Automatically detect more whitespace errors.
All existing cases are fixed; only whitespace change (verify with
diff -w) except for one comment style fixup in include/spdk/nvme.h.
Change-Id: If750e54b9c8e3421ea6feda5f20184a31431631e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/402360
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This change wasn't correctly rebased and needs to be updated to compile
against the current blobstore.
This reverts commit c1174e6895.
Change-Id: I529608bee7323cb626d8c36dff15adc9ba24ad26
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/402352
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Unit tests implemented in following patches.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Ib18c9060f527bd22bfdbed74e96871a6e0551ead
Reviewed-on: https://review.gerrithub.io/396648
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
blobfs and lvol can now use this to automatically iterate
all existing blobs during spdk_bs_load. Changes to blobfs
and lvol will come in future patches.
This will also be used in some upcoming patches which need
to iterate through blobs during load to determine
snapshot/clone relationships.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic7c5fac4535ceaa926217a105dda532517e3e251
Reviewed-on: https://review.gerrithub.io/400177
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Unit tests that use this will still immediately execute
the messages by default. But upcoming changes to
queue persists will need to leverage testing asynchronous
behavior so these changes enable that.
While here, fix some bugs where _bs_flush_scheduler()
did not fully flush the queue of scheduled ops.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I009e8277eef41d7e3677d9f135db34d8eaf2f071
Reviewed-on: https://review.gerrithub.io/401256
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Finish the sequence first, before calling _spdk_bs_free().
Otherwise synchronous bs_devs (like we use in the unit
tests) cause the sequence memory to get freed via
_spdk_bs_free() and then we try to finish the sequence.
This eliminates the need for g_scheduler_delay and
_bs_flush_scheduler() in the blob unit tests. But don't
remove them - they will be useful in upcoming unit tests
for queued persist operations.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I09aac3ae4d3a56ff8e04a5b822fcd6746f13afc3
Reviewed-on: https://review.gerrithub.io/401267
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
These new names are much more clear and are aligned with other
functions such as spdk_blob_close.
Keep the old names around for now but deprecate them. We will
remove them in next release.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idc60fd0b19fa2a8b0247a1f5835774d342e721f9
Reviewed-on: https://review.gerrithub.io/400884
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is needed for an upcoming change which will
prevent metadata functions from being called on
threads other than the metadata thread. Without
this change, there was no way for this function
to return an error if it was called from the wrong
thread.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I67e591140194ff6ad250878168f6b166a1ff2282
Reviewed-on: https://review.gerrithub.io/400883
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
There was some thinking that we would need to allocate
I/O channels on a per-blob basis to handle dynamic
resizing during I/O. Making spdk_blob an opaque handle,
with the existing spdk_blob structure renamed to
spdk_blob_data was a first step towards making that
happen. But more recent work on blobstore has
simplified the resizing approach, so this spdk_blob_data
is no longer needed. So revert it.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I22e07008faceb70649ee560176ebe5e014d5f1a3
Reviewed-on: https://review.gerrithub.io/400881
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I06d33ff7f6df0724768f5a23236485c26e0568f4
Reviewed-on: https://review.gerrithub.io/399729
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Patch adds internal version of xattr functions to allow
operations on internal xattrs, which are not visible to
upper layers.
When there is at least one internal xattr set, also
SPDK_BLOB_INTERNAL_XATTR flag is set in invalid_flags to prevent
loading this blob in previous spdk versions.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Iec918ec858f069f7cd9f36d5e8f0495ffa4a42d8
Reviewed-on: https://review.gerrithub.io/395122
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
This change will allow reusing this structure for both internal
and external xattrs as well as in functions having optional xattr,
but missing other options (i.e. snapshot, clone implemented in next patches)
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Ia6619a75efa0a100168a6f8317be274823af04ab
Reviewed-on: https://review.gerrithub.io/396417
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Iddbc490d1bf311f6e4b6e3ea3b7bdb72889bb2d3
Reviewed-on: https://review.gerrithub.io/394972
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Ibc9609ad36188006e9454e5c799bccd8a92d7991
Reviewed-on: https://review.gerrithub.io/391422
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This will be needed for thin provisioning, since a write
I/O may result in needing to insert a cluster into the
blob and that write I/O may not have been performed
on the metadata thread.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I84b0cb6e7af87b1f9c6cab4e2c24fa26b12e2c06
Reviewed-on: https://review.gerrithub.io/396737
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
PAGE_SIZE just happens to resolve to same value as SPDK_BS_PAGE_SIZE
on x86-64, but at least some ARM systems do not define PAGE_SIZE
in headers included by blob unit tests, which is only reason this
discrepency was not identified until now.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie4afbc35263d6c17eafa1abcbf3d342942c80ab0
Reviewed-on: https://review.gerrithub.io/396320
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Currently, there is no possibility to save read only blob to disk.
This patch modifies behaviour so that read only flags are applied after syncing blob.
This is analogy to resize, set xattr and remove xattr operations.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Iffed601c78cb83231bb20e7ef05b73847dc3c95a
Reviewed-on: https://review.gerrithub.io/394243
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Iadb29a8ce8dcebfea68d4feeb5f3de1bb3124f16
Reviewed-on: https://review.gerrithub.io/392286
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Ib3470fbac49e92308ed14e20ccde6655354f2580
Reviewed-on: https://review.gerrithub.io/389577
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch provides logic for returning errors instead of
assert when size is larger than blobstore size.
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I16d12338e2b682c39bd33d507d57ea126501a0d7
Reviewed-on: https://review.gerrithub.io/392749
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Recovery code did not claim clusters taken by metadata.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: If6726eddd22f4e1a3f9814b2348243155fb0fdb9
Reviewed-on: https://review.gerrithub.io/394173
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
