Commit Graph

45 Commits

Author SHA1 Message Date
Konrad Sztyber
3fbe74fd82 accel: don't modify user iovs when allocating buffers
It is quite common for a user to use the exact same iovec (in memory) to
describe buffers for two different operations.  If that iovec was
describing accel buffer, accel would modify it replacing it with an
actual buffer.  This is broken if that iovec was used by some other task
in a sequence, as accel wouldn't be aware that it has been changed too.

To address this, accel will use a new iovec from the aux_iovs array.  It
means that accel buffers always *must* be passed using a single iovec.
Theoretically, users could chunk that buffer into several iovecs, but
spdk_accel_get_buf() always returns a single buffer, so, in practice,
this should never happen, and therefore is unsupported.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I25271bc032987dd6028fb7b3adde061657759b4b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17039
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-03-13 21:02:27 +00:00
Konrad Sztyber
4832235759 accel: re-enable iobuf thread caches
They were disabled before the v23.01 release, because none of the other
libraries were using the new spdk_accel_append_* API.  But now, they
will be used in the bdev layer and bdev modules, so they need to be
re-enabled.  We're using the same values as we do in the bdev layer.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ibda86ca5619e4104e107048ce0965171501fdc5a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16968
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-03-13 21:02:27 +00:00
Konrad Sztyber
e6a8401a1d accel: execute accel sequences using a driver
If a driver is registered and selected, it'll now be used to execute
sequences of accel operations.  The driver has priority over accel
modules, so the modules will only be used to execute operations that the
driver cannot perform.

Once driver completes a task (or a number of tasks), it notifies accel
using standard spdk_accel_task_complete().  To let accel continue
processing a sequence, driver can call spdk_accel_sequence_continue().
This can be done when the driver executes all tasks (1), an error occurs
(2), or the driver doesn't know how to execute a given opcode (3).  In
case of (3), that operation will be executed using appropriate accel
module and, while the rest of the sequence will be sent back to the
driver.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: If414c02073ffc731454e03d25c7ee02bef58463b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16548
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-03-01 09:46:40 +00:00
Konrad Sztyber
0473a78165 accel: clear out domain/step_cb pointers in encrypt/decrypt
They  aren't cleared before a task is submitted and might store pointers
from a previous operation.  This can lead to issues if the previous
operation was using memory domains and we submit the task to a module
also supporting memory domains.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Icafb924c2e936ee6a83d921ae48e953b98f00841
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16848
Community-CI: Mellanox Build Bot
Reviewed-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-02-21 13:13:10 +00:00
Artur Paszkiewicz
b41efae2c6 accel: add support for xor
Change-Id: I3c7461a7abfc64402929c0bb24f8458814d9c706
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16394
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-02-21 08:33:18 +00:00
Konrad Sztyber
45b7fb23b1 accel: assert that we never modify seq's ERROR state
This should help catching bugs when a failed sequence gets cleared its
failed state.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I9389a2610e94e766aaf4185445c36442c4d4a1f7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16545
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-02-13 13:52:06 +00:00
Konrad Sztyber
914007c1c6 accel: functions to iterate over tasks in a sequence
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I97f011b7eaa7db7e9a9df583aff23321d60bb72f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16378
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-02-13 13:52:06 +00:00
Konrad Sztyber
02e8b81bb6 accel: expose method for allocating accel buffers
This will allow a platform driver to allocate a buffer in case it cannot
execute the whole sequence and the destination buffer of the last
operation is a "virtual" accel buffer.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia947cf553619828a170c5d0563b4c355d7b5ead5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16377
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-02-13 13:52:06 +00:00
Konrad Sztyber
efcae1bdee accel: method for getting accel memory domain
This will allow drivers to check if a task is using buffers from accel
domain.  This is just a helper, since the same can be achieved by
calling `spdk_memory_domain_get_first("SPDK_ACCEL_DMA_DEVICE")`, but
there's only a single accel domain and it is a bit special, so it makes
sense to have a dedicated helper function for getting it.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I07db7445ed9b109e66ecdbc0483a6a158a551070
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16376
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-02-13 13:52:06 +00:00
Konrad Sztyber
b2454dfbcc accel: introduce platform drivers
The goal of a platform driver is to execute chained accel operations in
the most efficient way possible.  A driver is aware of the hardware
available on a platform and can execute several operations as a single
one.  For instance, if we want to do DMA and then encrypt the data, the
driver can do both at the same time, if the hardware is capable of doing
that.

Platform drivers aren't required to support all operations.  If a given
operation cannot be executed, the driver should notify accel to continue
processing a sequence, via spdk_accel_sequence_continue(), and that
operation will processed by a module assigned to its opcode.

It is required however, that all platform drivers support memory
domains, including the "virtual" accel domain.  A method for allocating
those buffers will be added in the following patches.

This patch only adds methods to register and select platorm drivers, but
doesn't change the way a sequnce is executed (i.e. it doesn't use the
driver to execute it).

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I97a0b07e264601ab3cf980735319fe8cea54d38e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16375
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-02-13 13:52:06 +00:00
Konrad Sztyber
01608f1af2 accel: temporarily disable iobuf thread caches
The iobuf buffers are only used in accel when executing chained
operations.  Since none of the components in SPDK are using chaining
yet, there's little point in having per-thread iobuf caches, as they
only reduce the number of available buffers in other libraries.

This change will be reverted once bdev layer and bdev modules are
updated to support chaining.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ibad19ea92f2218a8dec01e802a736cfdd357dfc6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16398
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2023-01-20 17:56:44 +00:00
paul luse
d6d0ef6c16 lib/accel: remove 2 lines of dead code
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I3c4b04109a4c2f8319408f2ba5b805db05626535
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16051
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-01-20 10:26:21 +00:00
paul luse
91f3063b14 lib/accel: add output_size to decompress API
We had it for compress but simply didn't think of a use case for
decompress.  During the develpoment of the compressdev accel_fw
module it was discovered that compressdev does indeed provide the
uncompressed length on completion of decompress and the reducelib
uses it.  So, add it here.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I2f6a8bbbe3ef8ebe0b50d6434845f405afa7d37d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16035
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-19 11:16:01 +00:00
Konrad Sztyber
9cdbd9e4f3 accel: support appending encrypt/decrypt operations
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I7bbe90936ff11b50a7cca7b15eade2025daac83b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16292
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
3de19b0b55 accel: allow modules to report memory domain support
Accel modules can now implement the get_memory_domains() callback to
indicate the types of memory domains they support.  If unimplemented, a
module is assumed not to support memory domains and accel will take care
of pulling/pushing data to local buffers prior to passing a task to be
executed by a module.

For now, similarly to the bdev layer, we only check if a module supports
memory domains, but we don't verify the types of the domains.  That
could be easily added in the future, if necessary.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia513f4f31124672b705b6dd33a2624f0ae94d3ce
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16027
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
a6fef9b194 accel: store in-use modules in an extra structure
It allows accel to store private data per each opcode/module without
having to change externally visible structures or allocate anything when
a module is registered. Since a single module can service multiple
opcodes at the same time, so some of these values might be duplicated.
However, there are only a handful of opcodes, so it shouldn't be a
problem.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I609a6ccc2d241cb9b8273cc2c6d1933d2bc25e0e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16026
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
81fe7ef0af accel: push data if dstbuf is in remote memory domain
If the destination buffer is in remote memory domain, we'll now push the
temporary bounce buffer to that buffer after a task is executed.

This means that users can now build and execute sequence of operations
using buffers described by memory domains.  For now, it's assumed that
none of the accel modules support memory domains, so the code in the
generic accel layer will always allocate temporary bounce buffers and
pull/push the data before handing a task to a module.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia6edf266fe174eee4d28df0ca570c4d825436e60
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15948
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
316f9ea3f5 accel: pull data if srcbuf is in remote memory domain
If the source buffer is from a remote memory domain, we will now pull it
to the temporary bounce buffer before a task is executed.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I476684a4359410c69dd69a2b425b9e61d4c55a7e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15947
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
78186df493 accel: put completed tasks on the completed queue in process_sequence
The first task on a sequence's task queue is the one that we're
currently executing.  By moving the place where we remove it from that
queue and place it on the completed queue to process_sequence(), we'll
be able to perform some extra steps (e.g. memory domain push) after a
task has been completed.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia98f491eb52be0156954372461e05c198c070e3b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15946
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
fb67fa548f accel: sequence state machine
Processing a sequence consists of multiple steps and we call
accel_process_sequence() mutliple times, so we need to check various
things to verify if some of those steps have already been done.  Having
a state machine allows us to reduce the number of such checks and makes
it easier to add additional steps.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I254819fee0893866de395193041b319cbad228ec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15945
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
9a562043a7 accel: alloc buffers for data in remote memory domains
If a task has buffers in a remote memory domains, we'll now allocate a
buffer from local memory and replace the original buffer with it.  This
is the first step in supporting buffers in remote memory domains. To
fully support it, we'll also need to pull/push the data before/after
executing a task.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I3c86bbb6dbe6a31cb2cae8ce7d73e272ddc2734c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15944
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
957076108f accel: remove nbytes from spdk_accel_task
All operations are using iovecs to describe their buffers and only
encrypt/decrypt additionally used nbytes to store the total size of a
src buffer.  We don't really need this value in the generic accel code,
so we can let modules calculate it, if necessary.  That way, we won't
waste cycles calculating it if a module doesn't use it and it makes the
code a bit easier, as we won't have to deal with the fact that nbytes is
only valid for certain operations.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I29252be34a9af9fd40f4c7fec9d0a0c1139c562d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16306
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
1866faffe2 accel: use iovecs for compress operations
Also, since this was the last operation using dst and nbytes, these
fields were removed from spdk_accel_task.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I0d6b090e101c016d1bdcbe7a3bee7d6f691f1c9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15943
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
a374f8ba19 accel: use iovecs for copy+crc32c operations
Also, since this was the last operation using src, remove this field
from spdk_accel_task.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I55fd98697ef4f92a13dd0563b4adf9ccb0af171b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15942
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2023-01-17 23:34:43 +00:00
Konrad Sztyber
3d1d5452e0 accel: use iovecs for crc32c operations
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ic9f1f002edf273e9cd2247f353b5d7de9d2dea05
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15941
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2023-01-16 15:35:15 +00:00
Konrad Sztyber
bc6a14636a accel: use iovecs for fill operations
Also, make it possible to remove copy operations following a fill
operation if they're using the same buffers.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I7da195ce80650a02c5db99d9400ee692f797b1f8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15940
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-16 15:35:15 +00:00
Konrad Sztyber
4d1ba5f294 accel: use iovecs for compare operations
Also, replace src2 with an iovec + iovcnt and rename it to s2 to
keep the naming consistent with the source buffer (s).

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I44787128377addd514818ec5aaec084b1a31f0c3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15939
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-16 15:35:15 +00:00
Konrad Sztyber
135396b0bc accel: use iovecs for dualcast operations
Also, replace dst2 with an iovec + iovcnt and rename it to d2 to
keep the naming consistent with the destination buffer (d).

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ib394c127eeb5890451535ff485f96f7edd2897a4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15938
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-16 15:35:15 +00:00
Konrad Sztyber
dee8e1f4c0 accel: use iovecs for copy operations
This patch is first in the series of patches aimed to make all accel
operations describe their buffers with iovecs.  The intention is to make
it easier to handle tasks in a generic way.

It doesn't mean that we change the API - all function signatures are
preserved.  If a function doesn't use iovecs, we use the aux_iovs array.
However, this does mean that each accel module that provides support for
a given operation will need to be adjusted to use iovecs.

Additionally, update the unit test checking copy elision to verify the
buffers of the copy operation that is left.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I9e6d8d1be3b8b9706cb4a6222dad30e8c373d8fb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15937
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-16 15:35:15 +00:00
Konrad Sztyber
58b12fc4b9 accel: support for buffers allocated from accel domain
Users can now specify buffers allocated through `spdk_accel_get_buf()`
when appending operations to a sequence.  When an operation in a
sequence is executed, we check it if it uses buffers from accel domain,
allocate data buffers and update all operations within a sequence that
were also using those buffers.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I430206158f6a4289e15f04ddb18f0d1a2137f0b4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15748
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-16 15:35:15 +00:00
Alexey Marchuk
2608d129d0 accel: Add crypto operation support
Add functions to submit encrypt/decrypt operations
Add RPCS to register and dump crypto keys
Software accel module uses isa-l_crypto AEX_XTS
functionality

Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: Iecf0e9913edf11ab85171d0fa467a2a62dfff984
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14858
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <qun.wan@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2023-01-11 09:16:59 +00:00
Konrad Sztyber
940be80363 accel: accel buffer allocation functions
The data buffers backed by these accel buffers aren't allocated
immediately, but only when they're necessary to execute a given
operation.  It allows users to append operations to a sequence, without
actually reserving large space for the data.  That way, if some of these
buffers aren't needed to execute a sequence, they won't be allocated.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ieeea8a011b40c7f2f33e9a6f03fe34264e9316f3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15746
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-09 12:37:37 +00:00
Konrad Sztyber
7b0f452b4f accel: add iobuf channel to accel channel
It will be used for allocating buffers from accel domain and
allocating bounce buffers to push/pull the data from memory domains for
modules that don't support memory domains.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Idbe4d2129d0aff87d9e517214e9f81e8470c5088
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15745
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-01-09 12:37:37 +00:00
Konrad Sztyber
d3ac42caa4 dma: add "virtual" accel memory domain
This domain is meant to represent data being transformed by accel
engine.  Users will be able to allocate buffers from that memory domain
and use them when appending operations to an accel sequence.

Since these buffers are only meant to be used as placeholders for actual
buffers, none of the push/pull/translate callbacks are implemented. To
access the data after it was transformed by accel, users should make
sure that the final command's destination buffer isn't allocated from
accel memory domain.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ia031c7b205e98792d0a93f01513101b86afa9faa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15744
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2023-01-09 12:37:37 +00:00
Konrad Sztyber
7b36fe5238 accel: add support for reversing a sequence
Reversing a sequence means that the order of its operations is reversed,
i.e. the first operation becomes last and vice versa.  It's especially
useful in read paths, as it makes it possible to build the sequence
during submission, then, once the data is read from storage, reverse the
sequence and execute it.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I93d617c1e6d251f8c59b94c50dc4300e51908096
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15636
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-09 12:37:37 +00:00
Konrad Sztyber
f778e8e53a accel: remove redundant copy operations
Operation sequence should always be treated as a whole, meaning that
users cannot rely on the contents of any intermediate buffers and should
only care about the buffer that's the destination of the whole
operation.  This allows us to remove some of those copy operations by
changing source / destination buffer of a preceding / following
operation.

If a sequence is using buffers from non-local memory domain, users can
append a copy operation to a sequence to specify a local destination
buffer.  If the module executing the operations is aware of memory
domains, this can avoid doing an extra spdk_memory_domain_pull_data().

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I93b94d46ee32700819e9e6f1c55350692db8a67a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15530
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-09 12:37:37 +00:00
Konrad Sztyber
59f55d23f2 accel: add support for appending a decompress operation
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I5f091a554e08f0e052ab9e7eb9a1789d381b885f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15635
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-09 12:37:37 +00:00
Konrad Sztyber
6293ac8759 accel: initial operation chaining support
This patch introduces the concept of chaining multiple accel operations
and executing them all at once in a single step.  This means that it
will be possible to schedule accel operations at different layers of the
stack (e.g. copy in NVMe-oF transport, crypto in bdev_crypto), but
execute them all in a single place.  Thanks to this, we can take
advantage of hardware accelerators that supports executing multiple
operations as a single operation (e.g. copy + crypto).

This operation group is called spdk_accel_sequence and operations can be
appended to that object via one of the spdk_accel_append_* functions.
New operations are always added at the end of a sequence.  Users can
specify a callback to be notified when a particular operation in a
sequence is completed, but they don't receive the status of whether it
was successful or not.  This is by design, as they shouldn't care about
the status of an individual operation and should rely on other means to
receive the status of the whole sequence.  It's also important to note
that any intermediate steps within a sequence may not produce observable
results.  For instance, appending a copy from A to B and then a copy
from B to C, it's indeterminate whether A's data will be in B after a
sequence is executed.  It is only guaranteed that A's data will be in C.

A sequence can also be reversed using spdk_accel_sequence_reverse(),
meaning that the first operation becomes last and vice versa.  It's
especially useful in read paths, as it makes it possible to build the
sequence during submission, then, once the data is read from storage,
reverse the sequence and execute it.

Finally, there are two ways to terminate a sequence: aborting or
executing.  It can be aborted via spdk_accel_sequence_abort() which will
execute individual operations' callbacks and free any allocated
resources.  To execute it, one must use spdk_accel_sequence_finish().

For now, each operation is executed one by one and is submitted to the
appropriate accel module.  Executing multiple operations as a single one
will be added in the future.

Also, currently, only fill and copy operations can be appended to a
sequence.  Support for more operations will be added in subsequent
patches.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id35d093e14feb59b996f780ef77e000e10bfcd20
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15529
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2023-01-09 12:37:37 +00:00
Michal Berger
3f912cf0e9 misc: Fix spelling mistakes
Found with misspell-fixer.

Signed-off-by: Michal Berger <michal.berger@intel.com>
Change-Id: If062df0189d92e4fb2da3f055fb981909780dc04
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15207
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-12-09 08:16:18 +00:00
paul luse
a6dbe3721e update Intel copyright notices
per Intel policy to include file commit date using git cmd
below.  The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc).  Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line.  Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot
2022-11-10 08:28:53 +00:00
Alexey Marchuk
c77b537786 accel: Save overridden options in json config file
Signed-off-by: Alexey Marchuk <alexeymar@nvidia.com>
Change-Id: Ida2c6f1c460c2b66d2d4159d225036377e488e62
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14856
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
2022-10-19 07:47:58 +00:00
paul luse
850cd90082 accel/idxd/iaa: Convert to use iovecs
In prep for upcoming iovec based compression/decompression patches.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I413493f764bead9e56266e488b74f8bca979e225
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14633
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-23 00:10:08 +00:00
paul luse
28886ac352 lib/accel: rename iovec elements with src prefix
In prep for adding both src and dst iovec support for compression.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I704b8d2bd459de03deb7f8ee45d76261910a3727
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13746
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2022-09-23 00:10:08 +00:00
Ben Walker
712e8cb7ef accel: Refer to plugins as 'modules' instead of 'engines'
This is consistent with the use of terms in other parts of SPDK and fits
with the code living under module/

Change-Id: If182f7cf2d160d57443a1b5f24e0065f191b59b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2022-09-21 08:17:48 +00:00
Ben Walker
34c48f1b3b accel: Do not refer to the "framework" as "engine"
The word engine was both used (interchangeably with module) to refer to
the things that plug into the framework and to the framework itself.
This patch eliminates all use of the word engine that meant the
framework. It leaves uses of the word that meant "module".

Change-Id: I6b9b50e2f045ac39f2a74d0152ee8d6269be4bd1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13918
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
2022-09-06 07:16:17 +00:00