If DPDK is using virtual addresses, we should use virtual addresses.
If DPDK is using physical addresses, we should use physical addresses.
This way there can never be a conflict and everything is consistent.
Change-Id: Ie4b0e885e9a52dd6cbc81000a87908102a9771cb
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475928
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
DPDK by default guesses that it should be using iova-mode=va
so that it can support running as an unprivileged user. However,
some systems (especially virtual machines) don't have an IOMMU capable
of handling the full virtual address space and DPDK doesn't
currently catch that. Add a check in SPDK and force iova-mode=pa
here.
Change-Id: Ib3a5691a584190feaab4b9064b5a500e361328f2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475149
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In DPDK, the ppc iommu support does not currently allow for
iova-mode=va, but DPDK doesn't detect ppc and so still attempts
to guess iova-mode=va in some modes. Force iova-mode=pa from
SPDK to fix this.
Change-Id: I6a1ee25ab74873826ac211c3e0dfdf54afc74502
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/475148
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: JinYu <jin.yu@intel.com>
DPDK intr thread is designed that it can't unregister the src
callback in this callback handler. So I think we can't detach
the PCI device in the hotremove callback as it needs to unregister
the VFIO notification callback which will be not successful
but it still can free the device. So at the next req notification
in the handler function, we meet the freed device.
Fix#994
Change-Id: Id4b45a2d0fe6b45b132355d59471bc80240fad70
Signed-off-by: Jin Yu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/473176
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Ideally we'd have a way to query if DPDK is already
initialized but we don't have that yet. We want that
for the case where we have an SPDK application that's part
of a framework that may (or may not) have already initialized
DPDK. If it's already been initialized, let's print an
error message that isn't quite as inflammatory.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ifc095245dcdef24cdeeaab2dbe791ca4e840870e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471422
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The header file already says it returns negative errnos,
but the env_dpdk implementation was just returning -1
on failure.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie2236f83094672548327dba945b33e3f28fee338
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471421
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The function allows the user to get string representation of the type of
a PCI device.
Change-Id: I02abcd9fc98ba912ca4d7936be22e9d5b4950ea2
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470648
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
The patch of this purpose is to exclude the CPU cores
occupied by the DPDK thread. To mitigate the corner
case, we only do it when the number of online CPU cores
is larger than then DPDK thread occupied cpu cores.
The purpose is uset to improve the performance and avoid the
contention between DPDK thread and user's own thread.
Change-Id: I1a4a28074df97c55ac531440aea41059a75543f6
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471000
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: JinYu <jin.yu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Create dpdk_lib_list_to_libs and dpdk_env_linker_args
functions to generate the library filename list and the
linker arguments respectively. Use these functions
internally as well.
These will be useful as part of the Seastar work, where
Seastar pkg-config includes a bunch of the DPDK libraries,
and SPDK needs to just add a few more.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaa6b49a8e1defacf63b3f6b414cd2e947670f8eb
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469751
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
SPDK_ERRLOG lists the function name, so remove old references that
assume it doesn't and reprint the function name.
Change-Id: I69da6ca0a25bf0eda07d8dad52bcfadf964ac715
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469487
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
spdk_pci_device_claim() could create a file on the
filesystem that couldn't be deleted programatically.
It could only be overwritten - e.g. by another spdk
instance - but this didn't really work if that
another instance had less privileges and hence no
access to the previous file.
This is exactly the case we're seeing on our CI when
running SPDK as non-root. In general it's a good idea
not to leave any leftover files, so now we'll delete
the pci claim file when the spdk process exits.
spdk_pci_device_claim() used to return a file descriptor
that could be simply closed to "un-claim" the device.
It'll now return only a return code. The fd will be
stored inside spdk_pci_device and will be closed either
when user calls the newly introduced spdk_pci_device_unclaim(),
or when the device is detached.
We'll still need to clean up those files somewhere in
our test scripts (probably ./setup.sh cleanup) to
clean up after crashed processes or so - but we don't
necessarily want to run such scripts inside the autotest
whenever a non-root spdk is about to be started.
Change-Id: I797e079417bb56491013cc5b92f0f0d14f451d18
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467107
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Support for these options was not introduced until DPDK commit
7f0bb634a1406b132ff15c9cd56a0a9f33e5f11d
Signed-off-by: Benjamin Saunders <bsaunders@google.com>
Change-Id: Id6db73dd48ac01aa1b05eca4c920c5753e8cc6f0
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467703
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In the past, memory in spdk could have been unregistered in
different chunks than it was registered, so to account
for that the vtophys code used to register each hugepage
(2MB chunk of memory) separately to the VFIO driver. This
really made the code generally simple.
Now that memory in spdk can only be unregistered in the same
chunks it was registered in, we no longer have to register
each hugepage to VFIO separately. We could register the
entire memory region with just a single VFIO ioctl instead,
so that's we'll do now.
This serves as an optimization as we obviously send less
ioctls now, but most importantly it prevents SPDK from
reaching a VFIO registrations limit that was introduced
in Linux 5.1. [1]
The default limit is 65535, which results in SPDK being able to
make only the first 128GB of memory DMA-able. This is most
problematic for vhost where we need to register the memory
of all the VMs.
Fixes#915
[1] 492855939bdb59c6f947b0b5b44af9ad82b7e38c
("vfio/type1: Limit DMA mappings per container")
Change-Id: Ida40306b2684e20daa2fd8d12e0df2eef5a4bff1
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/432442
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
We'll be now able to check contiguity for more than 2MB
regions.
Change-Id: I738ff451d534075c944972918d08e5e0cadea4f5
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466073
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Previously, aplications were always being linked against the
spdk_env_dpdk.a file even during the shared object build. Change this to
be consistent with our linking.
Also, the old behavior causes issues with resolving symbols in FreeBSD.
Change-Id: I96e2e6044c16e7579cff35ad46e3688ce6fa2b5a
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464733
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When building shared libs, we specifically don't set the --as-needed
flag so that we still link against libraries that have constructor
functions. LD reports on its man page that the default behavior is
equivalent to --no-as-needed.
For RHEL based distros like fedora and centos this works fine. While
the LD man page shipped with ubuntu distributions also states that
--no-as-needed is the default, they don't respect that behavior and do
--as-needed linking unless you force them with --no-as-needed.
Change-Id: I914ab849323de198af5c5e53fffb1f57fcaff5fe
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464621
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Scan-build complains that spdk_vtophys_notify() can segfault
on vaddr == 0. We know that and it's not a programming error.
In fact, SPDK with UIO pci driver can segfault upon registering
any non-mapped memory address. The user is just not supposed
to do that.
Assert vaddr != 0 to silence scan-build.
Change-Id: I9a14696361eca0aeea5ede9f9f2956fcbf59bdb5
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/462869
Reviewed-by: yidong0635 <dongx.yi@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
DPDK defines the minimum alignment as "suitable for any
kind of variable (in the same manner as malloc())", but
internally the alignment is always rounded up to the
cache line size, even if the requested alignment is 0.
We would like to start relying on this behavior in FTL,
where lba maps are allocated using DMA-able memory and
are constantly looked up or modified by different threads.
By having the lba maps unaligned, we risk having those
threads pollute each other's cache lines.
Rather than enforcing this memory alignment in FTL, we
do it in spdk_*malloc directly. In general it makes
sense to have DMA-able memory always cache-line-size
aligned for the same reason as above.
Change-Id: Ib6edda4a7bf3f4952eb1875a4e1753be96bed642
Signed-off-by: Mateusz Kozlowski <mateusz.kozlowski@intel.com>
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460329
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
By making dpdk device detach asynchronous we have
actually broken some cases where devices are re-attached
immediately after and fail since they were not detached
yet, so now we're making device detach synchronous again.
For that we'll simply wait inside spdk_pci_device_detach()
for the background dpdk thread to perform all necessary
actions before we return. We'll also print an error msg
if DPDK failed the detach (probably because of some
internal error).
Change-Id: I7657ac1b169169eae3325de2d28c2cc311e7d901
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460286
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: <jacek.kalwas@intel.com>
By making dpdk device detach asynchronous we have
actually broken some cases where devices are re-attached
immediately after and fail since they were not detached
yet.
We'll need to make detach synchronous again, and for that
we'll wait for the background dpdk thread to perform all
necessary actions before we return from spdk_pci_device_detach().
However, device detach could be triggered from the very
same dpdk background thread as well. Waiting there would
cause a deadlock, so now we'll schedule asynchronous
device detach to the dpdk thread only if we're not on
that thread already.
This patch itself serves also as an optimization.
Change-Id: I86b7ac1b669169eee3325de2d28c2cc313e7d901
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460285
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Latest DPDK moved some definitions around and we don't
compile with it right now. Adding the missing include
fixes it.
Change-Id: I9b0a915632996acfedbcf3d0f03feed986889a2d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/460905
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Add spdk_mempool_lookup to lookup the memory pool created by the
primary process. This will be utilized in SPDK multi process
application future.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I90505b6566dfc93ef5957ef4c73b1a6438c30742
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459739
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Added spdk_pci_get_first_device() and
spdk_pci_get_next_device() to iterate
over all devices on g_pci_devices list.
Change-Id: I65079fb3e274195707dee64bc1fb8b4b72d07352
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450924
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Put the locks inside cleanup_pci_devices().
This serves as cleanup.
Change-Id: I040b28006e5584d1f33af26b63cafedbafe04fdb
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458934
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
The global pci tailq is no longer modified on the dpdk
thread, so on the spdk thread we can access it safely
without any lock. The code is slightly more readable
then.
This shows that cleanup_pci_devices() is always wrapped
with lock/unlock. We'll put the locks inside this
function in the next patch.
Change-Id: Ia4d386b78a87078761df0a3b953bfc4ff44102f8
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458933
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
To safely access the global pci device list on an spdk
thread, we'll need not to modify this list on any other
thread. When device gets hotplugged on a dpdk thread,
it will be now inserted into a new global tailq that
can be accessed only under g_pci_mutex. Then any
subsequently called public pci function will add it to
the regular device tailq.
Change-Id: I9cb9d6b24fd731641fd764d0da71bedab38824c9
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458932
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
To safely access the global pci device list on an spdk
thread, we'll need not to modify this list on any other
thread. When device gets hotremoved on a dpdk thread,
it will now set a new per-device `removed` flag. Then
any subsequently called public pci function will remove
it from the list.
Change-Id: I0f16237617e0bea75b322ab402407780616424c3
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458931
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
For VMD driver we'll need to introduce some way of
iterating over all spdk pci device objects and we would
like to achieve that with simple spdk_pci_get_first_dev()/get_next_dev()
APIs. To make it thread safe though, we would have to
expose some public pci mutex to be locked around the
iteration and we don't want to do that, so we'll make
PCI APIs usable from only a single thread - this will
prevent any pci devices from being removed inbetween
subsequent get_first/get_next calls.
We currently have the following players accessing pci
device state:
1) public APIs, obviously (on any thread right now)
2) VFIO hotremove callback (dpdk interrupt thread)
3) rte_eal_alarm for detaching rte_pci_devices (dpdk
interrupt thread)
4) DPDK hotplug IPC (dpdk interrupt thread)
There is g_pci_mutex providing the thread safety, but
even today it doesn't protect #3 and #4, making the
entire pci layer prone to data corruption.
To make #3 and #4 safe, we would have to lock inside
device init/fini callbacks (spdk_pci_device_init/fini),
but those are called directly inside the public device
attach/detach functions which already lock.
So now, with the decision to drop thread safety from
public pci APIs, we narrow down the locks inside public
functions and introduce locks inside those lower-level
init/fini callbacks.
Change-Id: I5dcbc9cdcbab65ee76cd3c42890f596069ec9a8a
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458930
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
As the name suggests, this function iterates through all elements of the
mempool invoking a callback function on each one. It's particularly
useful when deinitializing mempool that requires freeing resources tied
to each element (e.g. allocated through spdk_mempool_create_ctor).
Change-Id: I3da1fee527a36bf99f0b0e2dd3d6f9297422ff25
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455971
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In prep for full QAT compression support later in this patch
series. dpdkbuild/Makefile slightly refactored for readability,
x86 crypto check removed as it pre-dated checks we now have in
configure.
Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Iaaaf51b9eb5e18840f47d2d4f431c5a6e8c420ee
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456408
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
DPDK rte_ring_enqueue_bulk() has free_space parameter to return
the amount of space in the ring after enqueue operation has finished.
This parameter can be used to wait when the ring is almost full and
wake up when there is enough space available in the ring.
Hence we add free_space to spdk_ring_enqueue() and spdk_ring_enqueue()
passes it to rte_ring_enqueue_bulk() simply.
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I9b9d6a5a097cf6dc4b97dfda7442f2c4b0aed4d3
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456734
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
This patch add support for VMD driver object.
New PCI device ID for VMD device was added.
Change-Id: I47bd8772a15ad370a14b7cc9460a177c91e6dd6a
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Signed-off-by: Orden Smith <orden.e.smith@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455545
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This will eliminate a lot of unnecessary verbose
messages, especially from cryptodev.
Change-Id: I000adfa524c86f6379ebab4ba2087a8d6fabfe5f
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454099
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Will need this for upcoming compress patches so remove
from its current conditional linking and link for all.
Change-Id: Iba0cf0f529a0765b6d54f7f88eb86e516c5b89ee
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453026
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
rte_vhost currently logs every received vhost message as
a separate log entry. Every VM (re)boot results in over 100
lines of output printed for every queue in every vhost
controller. In our vhost initiator test suite we've got
over 5k lines of those in total. These logs do not help
in any debugging, so we disable them now. We'll still print
vhost messages of log level >= NOTICE.
Change-Id: Ief3f4230056478472412b13efc4da438b92b1d18
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/371691
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
While detaching the device, DPDK may try to unregister
a VFIO interrupt callback which is currently "in use".
The unregister call may fail, but the error doesn't get
propagated to upper DPDK layers. Practically, detaching
the device may stop in the middle but still return 0 to
SPDK.
This effectively breaks hotremove as the device would
be neither usable or removable.
We work around it in SPDK by internally scheduling the
DPDK device detach on the DPDK interrupt thread. This
prevents any other interrupt callback to be "in use"
while the device is detached.
Since device detach in SPDK can be asynchronous now,
we add a few checks to prevent re-attaching devices
that are still being detached.
Change-Id: Ibb56a8017e34418db0304fe32774811427b056aa
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448928
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is an attempt to fix device hotremove with VFIO.
A soft device hotremove request through sysfs [1] would
currently just block until the SPDK process manually
releases that device - e.g. upon an RPC request.
VFIO won't get unbound from the device untill userspace
releases all its resources. VFIO can signal a pending
hotremove request by kicking any file descriptor provided
by the userspace - and DPDK does provide such descriptor -
but SPDK does not listen on it.
DPDK does offer handy API to listen and in this patch
we make use of it inside our env/pci layer. Within
a DPDK callback we set an internal per-device hotremove
flag, which upper-layer SPDK drivers can poll with a new
env API - spdk_pci_device_is_removed().
The VFIO hotremove event will be sent to primary
processes only, so that's where we listen.
We make use of this new API in the NVMe hotplug poller,
which will process it just like any other supported
hotremove event.
Fixes#595Fixes#690
[1] # echo 1 > /sys/bus/pci/devices/<bdf>/remove
Change-Id: I03d88271c2089c740e232056d9340e5a640d442c
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448927
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
It's mostly needed for the next patch, but even
now it provides some value by printing errors if
there any leaked (still attached) PCI devices
at shutdown.
Change-Id: I8459a6049b3c6612d9f1d99444bf3acfd474a839
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449082
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Some customized SPDK application needs DPDK to run legacy memory mode,
but it is unusual. So this patch uses env_context, the opaque option
variable in the struct spdk_env_opts, and if the application sets
"--legacy-mem" to it, spdk_env_opts_init() skips adding
"--match-allocations" for DPDK 19.02 or later and skips adding
"--legacy-mem" for DPDK 18.05 or before.
Change-Id: I6f40c726c66c29f0aabfeeaecd00611954dc774f
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448263
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This flag is needed to enssure that we can access all of the elements in
the ring. Otherwise we end up being able to access n-1 elements.
Change-Id: I7a9216e69c0599c662e96ddbf6ee79383b6d20dd
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448489
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The function now has to be called before application
exit. At the moment it only frees the dynamically
allocated DPDK command line option strings - something
that was previously done from an atexit() callback -
but there's more to free there.
Note: the function descriptions were partially copied
from equivalent DPDK functions.
Change-Id: I5f4a6607fdfadff9325917259f58fcbc2cedba1a
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447676
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
DPDK 17.11 is the oldest version still supported by DPDK,
so drop support for DPDKs older than that in SPDK. This
lets us remove a huge amount of ifdefs.
Change-Id: I500987648e388cd5418a25845b6cccf4b55a4e5b
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447674
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Historically, all memory returned from spdk_*malloc()
used to be physically contiguous, hence it could be
addressed by offsetting just a single physical address.
Since DPDK dynamic memory management came along, the
above is no longer true. Memory returned from spdk_*malloc()
doesn't have to be physically contiguous anymore. The
phys_addr returned from spdk_*malloc() only applies to
the beginning of the allocated buffer and user can't
possibly know how big that "beginning" is.
The phys_addr parameter in spdk_*malloc() is useless on
its own in most cases and only suggests that the returned
buffer is physically contiguous, which is wrong.
This information can be returned from spdk_vtophys(),
which is the only safe way to retrieve physical addresses.
That's why phys_addr param in spdk_*malloc() is now
deprecated.
Change-Id: I934292f7db28b869b05caca4cb5c68c436e228d4
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448168
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This practically reverts commit 2fe7aa5e4 [1].
The extra rte_bus was supposed to allow running SPDK
as a non-priviledged user by enabling RTE_IOVA_VA mode.
DPDK uses RTE_IOVA_PA by default - which means there are
physical addresses used as memseg IOVAs and hence the
root access is required to retrieve those physical
addresses. This patch was supposed to be paired with
a different DPDK patch of mine, but DPDK rejected that
one. Instead, in DPDK 18.11+ the user can force iova mode
by specifying --iova-mode=<mode> command line option,
where <mode> is either pa or va.
Either way, apparently there are cases where physical
address contiguity is required even without UIO (#707)
so let's revert this patch and consistently stick with
RTE_IOVA_PA.
SPDK requires some more effort to support running as
a non-priviledged user anyway.
Fixes#707
[1] env_dpdk/vtophys: register a fake rte_bus supporting RTE_IOVA_VA
Change-Id: I90bf097fd6a7f129444229dc7cf07a462d4f2d09
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448121
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
It's disabled by default, so no functionality is changed yet.
The intention is to use the upstream rte_vhost from DPDK,
which - starting from DPDK 19.05 - is finally capable of
running with storage device backends.
SPDK still requires a lot of changes in order to support
that upstream version, but the most fundamental change is
dropping vhost-nvme support. It'll remain usable only with
the internal rte_vhost copy and with the upstream rte_vhost
it simply won't be compiled. This allows us at least to
compile with that upstream rte_vhost, where we can pursue
adding the full integration.
Change-Id: Ic8bc5497c4d77bfef77c57f3d5a1f8681ffb6d1f
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446082
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This will help keep the definition of that function fresh as things
change around it over time.
Change-Id: Id30864df132459a0ff889a725aa70abe072f3087
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446972
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Since the fuzz tester will be submitting random commands with random
memory addresses and such to the NVMe drives, we want to be especially
sure that we are using the IOMMU while running this test to prevent
memory corruption in the event that an errant command triggers a bad
DMA.
This function exposes to the application whether or not we are using the
IOMMU.
Change-Id: Ie4d26c706967a520967bfc81f72f7b581b792437
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446568
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
We specify the --match-allocations rte init parameter
now, which gives us guarantees that memory will be
freed in the same units it was allocated.
Note that if user initialized DPDK separately from
SPDK, we aren't sure if --match-allocations was
specified, so will still mark the segments to not
be freed.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I599747d4b917e91adfabf64c904cd7891a77b3cf
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446459
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This function indicates whether DPDK was initialized
external to the SPDK env_dpdk library.
This can be used in cases where we need to implement
different behavior when DPDK is initialized outside
of SPDK - in that case certain flags that SPDK would
prefer may not have been specified. This will
be used in the next patch.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I58d285bd4d9cda96b108624d65dedbec32164cfe
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446458
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This feature was added to DPDK by Jim to avoid the failures that can
come from splitting a buffer over memory regions in RDMA.
Change-Id: I13b646e22a4e2a4ccf915b0274061d31d02c03f7
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/446166
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
spdk_env_opts->env_context may now contain a DPDK-specific
string that will be appended directly into rte_eal_init().
It can be used to e.g. override the default EAL loglevel,
which was hardcoded to RTE_LOG_NOTICE so far.
This is primarily meant to be used during development.
As a test for this feature, the vtophys test app will now
set the highest possible EAL loglevel which will give us
a ton of additional debug logs.
Note: the opts->env_context field is implementation-specific
and hence the vtophys app needs to check if it's run with
our env_dpdk. As SPDK_CONFIG_ENV is a raw text not even
surrounded with quotation marks, the vtophys app needs to
do a bit of #define magic to make it a string.
Change-Id: I0b2196770e5b59a6c33d0170337c34f9f8b8466e
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/445111
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
When we were trying to push a newly allocated string
into the arg array and the array realloc() failed,
the string we were about to insert was leaked.
Change-Id: I31ccd5a09956d5407b2938792ecc9b482b2419d1
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/445149
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This helps ensure it gets inlined in the spdk_vtophys
code path, now that spdk_vtophys is defined in the same
compilation module.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0d0d9bba4295f0d9a7c0657834aa5d39f3b682d8
Reviewed-on: https://review.gerrithub.io/c/445354
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
CPU profiling on workloads with intensive vtophys
operations (i.e. very small CB-DMA transfers) exposed
overhead introduce by spdk_vtophys having to call
spdk_mem_map_translate in a different compilation
unit. Let's just move the vtophys.c contents into
memory.c so that spdk_vtophys can inline
spdk_mem_map_translate and avoid this extra overhead.
This of course breaks the memory and vtophys unit
tests, so some additional changes are needed there
to keep everything linking correctly.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I295ed5f441d3eec7abdbc9d881c49d2174ec9f48
Reviewed-on: https://review.gerrithub.io/c/444975
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Currently, SPDK does not support vfio in no-IOMMU mode. However, it
seems quite easy to extend the vtophys code to add support for this.
vfio in no-IOMMU mode does not support DMA remapping. This implies that
physical DMA addresses are used instead of IOVAs.
This patch checks whether the vfio no-IOMMU mode is enabled using
function rte_vfio_noiommu_is_enabled() from the DPDK RTE vfio interface.
In this case, physical addresses are used for the DMA mappings. This is
the same code path for the DMA translations as when the uio is used as a
kernel driver.
Change-Id: I6fb3c849a345c6f2f2b4141dddb8c17be2581495
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
Reviewed-on: https://review.gerrithub.io/c/441061
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Includes the required DPDK dependencies for SPDK block Reduce aka
Compression.
Change-Id: Ic1ea3cbeb9373a7700f6f0c2a3194d65d6a34a41
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/429523
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add rte_pause to waiting while loop
This commit also adds spdk_pause as interface for rte_pause
Change-Id: I56e1023731e2e78febaa4f45808d6f07656d290f
Signed-off-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Reviewed-on: https://review.gerrithub.io/c/436494
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Bump the log level for EAL to RTE_LOG_NOTICE.
Reading from rte_log.h:
```
RTE_LOG_NOTICE 6U /**< Normal but significant condition. */
RTE_LOG_INFO 7U /**< Informational. */
RTE_LOG_DEBUG 8U /**< Debug-level messages. */
```
We're doing this primarily for the NVMe hotplug poller,
which calls spdk_pci_enumerate() and constantly bloats
the output with logs describing which device is currently
iterated over. We don't want to see those.
Change-Id: I1a90e514fdf467bc95da910f786f1818757cfdcf
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/441789
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This allows us to remove the requirement to install intel-ipsec-mb to
system directories.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I579655a98b515cf148b7cd17823a9bb541ea6ad7
Reviewed-on: https://review.gerrithub.io/c/440785
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We use those values in various places in SPDK,
so let's define them in a single place now.
Change-Id: Iad9a5745d69166a6e6032370d4e5a0e604914e45
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439369
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This follows the same trend as the mem_map APIs.
Currently, most of the spdk_vtophys() callers manually
detect physically noncontiguous buffers to split them
into multiple physically contiguous chunks. This patch
is a first step towards encapsulating most of that logic
in a single place - in spdk_vtophys() itself.
This patch doesn't change any functionality on its own,
it only extends the API.
Change-Id: I16faa9dea270c370f2a814cd399f59055b5ccc3d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438449
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
spdk_mem_map_translate() will no longer update its
translation size parameter to a value that's bigger
than requested. This will be handy once we introduce
a similar translation length parameter to spdk_vtophys().
Change-Id: Ia662cd3f1340c57a3341182fa0e8137163084779
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438447
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In spdk_mem_map_translate() we used to set the translation
length to 0 if the provided memory region wasn't registered.
This doesn't really have any use case and is now removed,
which means that the translation length parameter will only
be updated for those memory regions that were successfully
translated.
This serves as a minor optimization and code cleanup.
Change-Id: I4c953f17e3f2181266bdcc71cf7e30c7244541f2
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438446
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
spdk_mem_map_translate() inside the vtophys unregister
callback could potentially read unitialized variable.
This isn't critical since vtophys mem map doesn't yet
implement the callback for checking mem contiguity and
the spdk_mem_map_translate() was exiting early - before
the unitialized variable was actually processed. It was
still technically dereferenced though, which is a bug.
Change-Id: I2af52e6f41bee35528c3d771aafd9c768c9d2fc7
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438445
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The enumerate callback doesn't currently iterate through
any hotplugged devices, as it uses an outdated device list
underneath. What updates that list is a bus rescan, which
happens implicitly on DPDK init or a specific device attach.
This wasn't crucial until we refactored NVMe bdev hotplug
poller to use enumerate instead of attach, which broke the
hotplug entirely. Unluckily, the hotplug tests were broken
as well and didn't detect this in time.
We fix the above by rescanning the pci bus before iterating
through its devices inside spdk_pci_enumerate().
Change-Id: I9643514ff07883eff0f3004b6991ca43ce0b2804
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438243
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Based on requirement of initializing SPDK env
from a DPDK application, relative to spdk_env_init,
add spdk_env_dpdk_post_init for calling after rte_env_init.
More details, visit
https://github.com/spdk/spdk/issues/529
Change-Id: I6fda1593e0296ef93b705e31cc76bcd0d248673a
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/437225
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Addresses issues with building both compressDev and cryptoDev at the
same time that we will run into shortly. We now build both API all
the time but those things that have ext deps are still built only
when their respective configure options are set.
Change-Id: Ia4b1d4f18826a8d78c2f09881fb268a8aff61f56
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/437989
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
This should have always been the case with spdk_mem_map_translate. For
some memory maps (like RDMA) this doesn't matter, but for others like
our virtual to physical map, this is critical for retrieving valid
translations.
This behavior change will only affect maps that have a registered
contiguous memory callback.
Change-Id: I67517667f01d974702d7daa7c81238281aae0cf6
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/436562
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
As a part of cleanup they're replaced by a device-agnostic
attach API, which is easier for us to manage.
Change-Id: I2ec68f20ba209f02ee5c2de4b6fe5330a4bc0853
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/436480
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
As a part of cleanup they're replaced by a device-agnostic
attach API, which is easier for us to manage.
Change-Id: Ia92bd8f4525712bd27ade16ead67435c5e0fbe7a
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/436479
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
As a part of cleanup they're replaced by a device-agnostic
attach API, which is easier for us to manage.
Change-Id: I7558590e41e5c580a130a6aba7ae4f7dcff58da8
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/436478
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This follows the overall model introduced together
with PCI device hooks. Having an additional set of
attach/enumerate/hook functions for each device type
doesn't scale well. We can simplify this by moving
the driver-agnostic attach and enumerate functions to
the public headers. It'll be used directly by the
upcoming VMD driver.
Change-Id: Ie2039389b6ea530d74d568dc7ebe8b214f547057
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/435804
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Devices behind a VMD aren't visible directly on the PCI
bus. In order to support them, we'll need an additional
VMD driver that's going to enumerate the devices behind
it and hook those into the SPDK PCI layer.
We want those devices to be accessible with the same APIs
that are used to access physical PCI devices.
The physical devices are still created and managed by
DPDK, but additional devices can be now hooked externally.
The hook API slightly departs from how env layer worked
so far. Instead of keeping the generic hook functions
internal-only and adding per-driver (NVMe, I/OAT, Virtio)
public functions, this patch makes the generic hook API
public from the start. It accepts the device driver as
a parameter, which needs to be exposed now. That's why
spdk_pci_nvme_get_driver() is introduced. It's only the
NVMe driver that's exposed so far, but other drivers and
their attach APIs should eventually follow the same path.
The previous model really didn't scale well and there's
no need to stretch it further.
Change-Id: Iade018a43b1e23527bd2914be42b403551e73bb6
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/435802
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
In order to populate our PCI device list with devices
located behind the VMD, we'll need to fill out those
device structures from within a special VMD driver. That
driver will base on PCI configuration and BAR accesses,
but definitely not on DPDK. We want to put the VMD driver
outside of the env lib, so we provide it with a direct
access to the device struct.
Change-Id: Iabddf361a805e69d7e857c2d07ceaed36aca261d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/435800
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In order to populate our PCI device list with devices
located behind the VMD, we'll need to fill out those
device structures from within a special VMD driver. That
driver will base on PCI configuration and BAR accesses,
but definitely not on DPDK. We want to put the VMD driver
outside of the env lib, so we're about to provide it with
a direct access to the device struct. Before we do that,
let's group all the env-internal fields into an extra
struct "internal".
The spdk_pci_device struct does actually depend on DPDK
now as it contains an `rte_pci_device *dev_handle` field,
but we can easily break that dependency. The field is only
used as an arguement to DPDK functions, so we can change
its type to void* and let the implicit type conversion do
the magic. After all, the VMD driver will potentially use
it to store its (non-DPDK) data as well.
Change-Id: I425d6dfa7af13e022f5377ceaff39efbd4a01b3d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/435799
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
DPDK 18.11+ multi-process hotplug isn't robust.
Multiple secondary processes starting at the same
time might cause the internal IPC to misbehave.
Just retry hotplugging/hotremoving the device
in such case.
Change-Id: I1f830c2c0dbe1d63eca9a116101b3d202172b2ca
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434539
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
With all the error checks and segfault preventions in place,
we can finally enable hotplug in a multi-process scenario
for DPDK 18.11+.
If a device is attached in the primary process, it will send
an attach IPC request to the secondary process which needs
to succeed. Until now it would get rejected, and the attach
would fail in all the processes.
The device in secondary process will be now probed by DPDK
and will be put into the process local SPDK list of devices
to be locally attached. Either SPDK will attach it sometime
later on any attach/enumerate request, or DPDK will remove
it automatically once the same device in the primary process
gets removed.
We also allow the surprise attach in primary processes, as
it's technically possible for the pci devices (NVMe) to
be attached exclusively from the secondary process. The
fact that the NVMe stack doesn't support it is another story.
Currently the NVMe stack will handle the failure by itself
just fine.
Change-Id: Ia24a8b4610cc7c659f59a2fdda9d8a78e58af873
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434416
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
DPDK 18.11+ does its best to ensure all devices are
equally attached or detached in all processes within
a shared memory group. For SPDK it means that if
a device is hotplugged in the primary, then DPDK will
automatically send an IPC hotplug request to all other
processes. Those other processes may not have the same
SPDK PCI driver registered and may fail to attach the
device. DPDK will send back the failure status and the
primary process will also fail to hotplug its device.
To prevent that, we need to pre-register the pci
drivers on env init.
We register the drivers just after the EAL init
because we don't want the matching devices to be picked
up by the initial bus probe in DPDK. That's for 2 reasons:
1) we don't want to attach *all* available devices
2) devices attached from non-SPDK context (that is,
outside of the spdk attach or enumerate functions)
will still fail to attach - the entire attaching
process will only take significant amount of time
and will bloat the log with useless status messages
Change-Id: I7b4c3a2e355f98ea755649f789137f5a727bc935
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434415
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Although the struct is used as an enumeration context,
it really is a pci driver. The subsuequent patch introduces
a few functions around the pci driver, so rename the struct
to make it align nicely with those functions.
Change-Id: I919c30e55d9f42d795ecd8e20e5d29f3918c17a5
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434414
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Upon detaching a device in a secondary process, DPDK 18.11
will try to detach it from the primary process as well.
SPDK doesn't support such hot-detach and will reject it
in the primary process. That will cause the secondary
process to also reject its detach. The device in the
secondary process will be still there in DPDK, but for
SPDK it will remain inaccessible - neither attach, nor
enumerate will work on it.
To fix it, we make our attach and enumerate functions
always check the process local list of devices probed
by DPDK, but not attached in SPDK.
Looking at the patch from a different perspective, it
simply introduces error handling for the DPDK detach
function. If a device failed to detach, we'll now maintain
it locally in SPDK to make it attach-able again.
Change-Id: I8c509a571bea7a9fb413c9c2bfd64c62ad91074b
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434413
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
It's handy to store the SPDK structs within the device
structure. The subsequent patch will make us use
spdk_pci_addr much more frequently, so it makes sense
to keep it around rather than build it up from rte_pci_addr
everytime.
The upcoming VMD driver will also benefit from this patch
by being able to fill the spdk_pci_device struct with any
custom PCI details.
Change-Id: I236a19e28beba9a593b29f23b79b1b0b92ef1fa7
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434418
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In DPDK 18.11, a device can be potentially detached not only
upon an SPDK request, but also directly from within the DPDK
itself. In a multi-process scenario, when one process detaches
the PCI device, an IPC message - detach request - will be sent
to every other process in the same shared memory group. As we
don't propagate the removal notification to upper layers, the
still-referenced rte_pci_device object will just disappear at
one moment.
SPDK is still not ready for supporting the above case and will
try to avoid it, but just in case some detach request slips
through, then this patch provides the sanity checks preventing
SPDK from crashing.
Change-Id: I3e35d8efb33085163b9acd8a565e86a4221df844
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434412
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Very minor cleanup before we start refactoring the code.
Change-Id: I00d768ec0c84f2a37c54b7575de695281c5ebb22
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434411
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: Jim Harris <james.r.harris@intel.com>
DPDK already prints at least one error message, so
there's no need to print a yet another one.
Change-Id: I1c7bdfe5ca2095b93ec282bf193a717627d5fa27
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434410
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Prepare for storing additional per-device data.
The struct doesn't store any interesting data yet,
but already has a TAILQ_ENTRY that allows us to
put it into a global pci device list. Right now
we use the list only to find the SPDK device once
the corresponding DPDK device gets removed, but
more usages will be implemented soon.
Change-Id: If3abc1da60446e0a647d8d4c642f111ebfbcdb9e
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/434409
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Now that even DPDK 16.11 (LTS) reaches its end of life in
November 2018, we can surely drop support for DPDK
versions older than that.
The PCI code will go through a major refactor soon, so this
patch cleans it up first.
Since this is the very first SPDK patch that drops support
for older DPDK versions, it also introduces an #error
directive that'll directly fail the build if the used DPDK
lib is too old.
Change-Id: I9bae30c98826c75cc91cda498e47e46979a08ed1
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/433865
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We assumed spdk_mem_map_translate() translates only 2MB-aligned
addresses, but that's not true. Both vtophys and NVMf can use it
with any user-provided address and that breaks our contiguous memory
length calculations. Right now each buffer appeared to have the
first n * 2MB of memory always contiguous.
This is a bugfix for NVMf which does check the mapping length
internally. It will also become handy when adding the similar
functionality to spdk_vtophys().
Change-Id: I3bc8e0b2b8d203cb90320a79264effb7ea7037a7
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/433076
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
This keeps us from having to deal with ALLOC and FREE events
for mismatching regions - which necessitated splitting new
regions into individual pages. This caused all kinds of
problems with NVMe-oF - for example, buffers that spanned
memory regions, or bumping up against MR limits on RDMA
NICs.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I18dcdae148436b55d4481bb9fb8799f4832c7de1
Reviewed-on: https://review.gerrithub.io/434895
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Despite the scary commit title, this patch just unifies
per-driver mutexes into a single pci mutex.
On each hotplug we modify some DPDK global resources,
which per-driver locks aren't sufficient for. If
multiple threads try to attach devices at the same time,
then we'll likely have a data race. DPDK hotplug APIs
don't provide any kind of thread safety on their own.
Change-Id: I89cca9fea04ecf576ec5854c662bae1d3712b3fb
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/433864
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We need to do it only for DPDK 16.11, which leaks the
mappings otherwise. DPDK was fixed in version 17.02 with
the following commit:
e84ad157 (pci: unmap resources if probe fails)
Unmapping the resources twice doesn't actually cause
us any trouble, but prints an ambiguous error message.
Change-Id: I8b62e86d5fff8fe924dbf9ae2e37cff29298d412
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/433863
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This was dead code. Registering the same device more than
once has never been possible within a single process.
Change-Id: I04fa2a62cd9f3d7f99ff799ea4504c49a2232da4
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/433866
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Previously we used VFIO if only the vfio-pci kernel module
was loaded, which is different from what our setup.sh script
did. On a fairly usual system configuration, setup.sh could
have bound devices to UIO, but SPDK would still try to map
memory to an (empty) DPDK VFIO container just because its fd
was available. That would fail obviously.
setup.sh checks for IOMMU presence in order to use vfio-pci
and SPDK should probably do the same. We could check the
kernel driver of each attached PCI device, but there's no
chance right now of supporting both UIO and VFIO devices
at the same time with IOMMU passthrough. That's not
a reasonable configuration anyway. To keep things simple,
we just add a single check on vtophys initialization.
Fixes#462
Change-Id: Ica653f117743be322291a1b7e37ed00e34ef5035
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/432518
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <wuzhouhui@kingsoft.com>
BSD implementation for config access in DPDK seems to
return 0 on success while Linux implementation returns 0
only on failure. The env wrapper was always treating 0 as
an error and caused some of our PCI initialization code
to fail prematurely.
At one point DPDK harmonized this BSD behavior with Linux,
but only for config reads.
Fixes#484
Change-Id: I4ea850ea50f5e667fad28e8125209b21c377a2a3
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/432401
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Allow specifying a custom hugetlbfs directory.
This can be useful e.g. when trying to use hugepages
with fixed size, different size limit, or different
access permissions.
Change-Id: I418cbab99ed183383300b3c3d9945095a03478db
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/432105
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
DPDK 18.11 sets the default base-virtaddr to an address
that falls into an area reserved by ASAN. DPDK will try
to remap its memory over and over with the closest
base-virtaddr hint and for ASAN case this would take
a huge amount of time.
This was already raised on DPDK mailing list [1] and
might be eventually fixed or worked around in upstream,
but for now let's just override the default base-virtaddr
to a value that ASAN is known not to occupy.
[1] http://patches.dpdk.org/patch/46130/#88395
Change-Id: Ieada30e82355e8ead458e53795ab98cd12692c1c
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/431257
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The previous functions were deprecated and now removed.
Change-Id: I076125aaf80b97c627ca45b860700fdf6d87e925
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/430557
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
In early development of crypto we built the lirbary using
dpdkbuild/Makefile but later due to a DPDK change we needed
to move to a newer version of the ipsec library that is now
built and installed in pkgdep.sh btu we never removed it
from the makefile.
This patch removes the extra building of the ipsec module.
Change-Id: I367618440074d952ff13c6e2098fc3a572cc785e
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/429519
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
With all SPDK patches in place, we can now enable dynamic
memory management for all DPDK versions >= 18.05.1
Hugepages can be still reserved with [-s|--mem-size <size>]
option at application startup, but once we use them all up,
instead of failing user allocations with -ENOMEM, we'll try
to dynamically reserve even more. This allows starting DPDK
with `--mem-size 0` and using only as many hugepages as it
was actually necessary.
Change-Id: I9e6f58ea50af2234f96a53e7a32d9e14d2df65ff
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/426828
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is an NVMe-specific issue and I/OA or VirtIO devices don't
need it. Additionally, the delay is now asynchronous, meaning
that potentially multiple NVMe controllers can wait all at once.
The drawback of this change is that we're needlessly waiting
even when using uio_pci_generic. However, since the delay does
not block anymore, its impact is significantly minimized.
Change-Id: I5d16a7fd7cb66c785acb687f14690e95f6188b9e
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/429414
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
All PCI device management is done only by the primary process,
so there's no need to delay device initialization in secondary
processes. If device is being initialized in a secondary
process, then it must have been already initialized by the
primary.
Change-Id: I087da77f981018dabf3feed59c76b294a16ca88d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/429413
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
While here, change spdk_lib_list_to_files to
spdk_lib_list_to_static_libs to differentiate it from
the new spdk_lib_list_to_shared_libs.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6e5913addfbdd556fae2451d4e2b2c43feaf33ab
Reviewed-on: https://review.gerrithub.io/429286
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Lance Hartmann <lance.hartmann@oracle.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Minor cleanup.
Change-Id: I9554163ae22836b50b954ec27ed27bcb848cb193
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/428883
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
The ENV_LINKER_ARGS was employing both the linker --[start|end]-group
and --[whole|no-whole]-archive options around the DPDK_LIBs. With
the use of whole/no-whole, the start/end bracketing is unnecessary.
Change-Id: I97a2ac22df8c6b48ba674b9b292f5eea01823901
Signed-off-by: Lance Hartmann <lance.hartmann@oracle.com>
Reviewed-on: https://review.gerrithub.io/428737
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
The problem with registering the entire hotplugged memory
region is that it won't necessarily be unregistered in one
go. Registering each hugepage separately solves that
problem.
This puts a limitation on the number of pages that can
be allocated when using RDMA. We'll hopefully lift this
limitation sometime in future - probably levereging
ibv_rereg_mr, but for now we'll have to resort to either:
a) using 1GB hugepages
b) preallocating memory (with [-s|--mem-size <size>] app
param) as it will be registered as just one region no
matter what size it is. This memory won't be returned
to the system until the SPDK app exits.
Change-Id: I6de997fb4901b772730ba6fe995dcc0640b85749
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/428716
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
All mem_maps will still receive separate unregister
notification for each registered region, but the
public memory unregister API is more flexible now.
This follows the VFIO_TYPE1v2_IOMMU interface, which
allows the same.
Change-Id: Ifc008afdc6bff39d9b3b4c892c379ade10c3098e
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/428715
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Added sanity checks to prevent unregistering a memory
range that wasn't registered as a one, complete region.
Change-Id: I819b57560b2e48b0802113ffff9f72949d7a148a
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/425556
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Removed the reference count from the registrations map.
Although technically supported, registering a single memory
region more than once had a lot of unhandled cases and could
easily lead to a segfault.
RDMA maps require all memory to be unregistered in the same
chunks the memory was registered, which is often impossible
to achieve if a region was registered more than once:
1. register region 0x0 - 0x3 -> it gets mapped to
a single ibv_mr
2. register region 0x1 - 0x2 -> nothing happens, this region
is already registered
3. unregister region 0x0 - 0x3 -> 0x0-0x1 gets unregistered as
one region. 0x2-0x3 gets
unregistered as another
(leading to segfault in the
the current RDMA implementation)
The problem is that the last two regions share the same ibv_mr,
which SPDK tries to free twice. The second free causes a segfault.
vtophys map handles this case by registering each 2MB chunk
separately, but this solution cannot be applied for RDMA, as
NICs put a limitation (~2048) on the number of regions registered.
Another option is to keep a refcount of each ibv_mr allocated,
and free it only when the entire region was unregistered from the
SPDK mem map. This is however very tricky and RDMAmojo mentions
that freeing a memory buffer before unregistering its ibv_mr
may lead to a segfault.
Change-Id: I545c56e24ffa55bda211dea22aeb8a55d9631fe5
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/426085
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Prevents us from unregistering two or more separately
registered regions with a single notification.
This fixes an ibv_mr leak in RDMA. When multiple registrations
were unregistered with a single notification, only the first
ibv_mr one would be freed and the remaining memory would
possibly still remain DMA-able.
As of now, unregistering multiple complete regions with
a single unregister call is not possible. It will be
implemented later, after the rest of the code is cleaned up.
Change-Id: I7d61867fa61fd7a4a8a644ff45cab17125d63e1b
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/425555
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Newly registered mem maps weren't notified at vaddr 0
as we used value "0" to denote an unitialized, unset
variable. Since we *do* register vaddr 0 in our memory
unit tests let's switch unitialized vaddr value to
something different - like UINT64_MAX.
Change-Id: I9c902165e76155e068642abb9a656f3ae8ca1105
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/428713
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In spdk_mem_map_alloc() we only do the memory walk when
notify_cb is provided, but spdk_mem_map_free() does the
memory walk undonditionally. Not anymore.
Change-Id: Ic8dfdc5cb2c99dc58e62ab0523cf5a18ba8691cc
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/428722
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
If during mem_map creation we managed to register only a part
of the total registered memory and then failed in the middle,
we will now undo our registrations and gracefully return
an error code leading to a spdk_mem_map_alloc() failure().
Change-Id: Id549f35109bd381318adc676392d9ee26d035359
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/423538
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Didn't enable this in initial patch, probably could have, but we
need it for performance testing now and will need it to start getting
the hardware into CI.
Change-Id: I688cc94713146380933e5cddd7ed5b2d168c1fb2
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/427274
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
eal_get_runtime_dir() that we used so far is not
available in DPDK shared libraries, so we remove
its usages in SPDK together with the functionality
of removing files left over by DPDK. Luckily for
us, this only affects DPDK 18.05, as for DPDK 18.08+
the --no-shconf option that we use prevents all
those files from being created whatsoever.
Change-Id: I078fb7686d2445a6acb067b0c3762a9c99f9a429
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/426218
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Lance Hartmann <lance.hartmann@oracle.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Dynamic memory management was broken in DPDK 18.05
and got fixed in 18.08/18.05.1, but SPDK still needs
a couple of patches to ensure we support it. While
those patches slowly make their way through the review
process, let's stick with legacy mem mgmt for all DPDK
versions.
Change-Id: I6a0bc7b46b28dd75bef6847dde1ef57dc60a829e
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/426817
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This function will now check for whether or not a memory region is
contiguous accross 2MB map entries and return the total length of that
contiguous buffer up to the size specified by the user.
Also includes unittests
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: I2ce582427d451be5a317808d0825c770e12e9a69
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/425329
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: I90da6d4d31c669a3bf046f7721923dd743c5ef21
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/425328
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The spdk_ring creation does not support "multi-producer/multi-consumer" type
and does not honor all of the types during enqueue/dequeue operations.
- Add SPDK_RING_TYPE_MC and update spdk_ring_create() to support it.
- Update spdk_ring_enqueue() to call rte_ring_enqueue_bulk() instead of
rte_ring_mp_enqueue_bulk().
- Update spdk_ring_dequeue() to call rte_ring_dequeue_burst() instead of
rte_ring_mp_dequeue_burst().
Change-Id: I15219513f9c45a8ec8a0af19cdc35428ba728454
Signed-off-by: John Barnard <john.barnard@broadcom.com>
Reviewed-on: https://review.gerrithub.io/426143
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We recently increased the top level map from 128TB
(47 bits) to 256TB (48 bits) but missed fixing one
comment that still referred to the old number of bits.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic993bd136cd09789ea2a65b480151805d231c3d5
Reviewed-on: https://review.gerrithub.io/426002
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Initial support for softare AESNI_MB DPDK driver only.
Have tested (both aesni and QAT seprately and concurrrently) on underlying NVMe devices
with bdevio and a bdevperf script that runs IOs from 512B to 128K each with Q depths from
1 to 512 in powers of 2 for 30 seconds each run.
QAT can be included in the code (but not makefile) and marked as experimental
until we are ready to test in CI. It works well on 2 systems but is a big PITA to get
the hardware setup and configured for use with DPDK (IMHO).
Change-Id: If518c3df8e74e00efa18afdf194824c5e69778fc
Signed-off-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/403107
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This struct will hold the unique operations for the mem_map.
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: Ifdd82497f238d99345033f2615c718802a591438
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/425327
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The function now takes a pointer as it's last argument, and copies the
size of the memory region for which the translation is validinto that
pointer.
For now, that will always be 2MB. However that behavior can change in
the future.
This series of changes is aimed at enabling spdk_mem_map_translate to
report back to the user the length of the valid mem_map up to the
function that requested the translation.
This will be useful when retrieving memory regions associated with I/O
buffers in NVMe-oF. For large I/O it will be possible that the buffer is
split over multiple MRs and the I/O will have to be split into multiple
SGLs.
Change-Id: I8686c166ec956507f5ae55cf602341281482cb89
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/424888
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The underlying DPDK function we use reads an array
at the provided index without checking for any out
of bounds access. The array is RTE_MAX_LCORE elements
long, so always manually check against that to keep
our APIs safe.
This fixes potential crashes with lcore == SPDK_ENV_LCORE_ID_ANY
Change-Id: I3081b888275fbecba8ab95feb20d2074341e2fc7
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/425042
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This is an artifact from before SPDK had a configure
script or a DPDK submodule. Make configure the
only supported way for specifying the location of the
DPDK installation to use with SPDK.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5c197c46220928bb18b97c8807755967d76ea42c
Reviewed-on: https://review.gerrithub.io/424893
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
the function pointer .get_iommu_class was not defined until v17.11 of
dpdk so this function causes packaged versions of dpdk <17.11 to fail to
compile with SPDK. Adding a couple preporocessor directives to avoid
this problem.
Change-Id: I70cf44877ddd712d42d117e6fa5f82494675d603
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/424609
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
To use virtual address IOVAs, DPDK has to ensure there
are absolutely no predispositions to require physical
addresses neither now, nor in future. All available buses,
including PCI, report whether they require physical
addresses. For PCI, the registered drivers may report
RTE_PCI_DRV_IOVA_AS_VA to mark a driver as RTE_IOVA_VA
compliant. However, this model assumes that all (PCI)
drivers are registered on DPDK init, which is not the case
in SPDK. With no devices attached, most buses will report
RTE_IOVA_DC (don't care) and EAL will default to
RTE_IOVA_PA.
SPDK needs a hook to report whether it supports RTE_IOVA_VA,
and the fake rte_bus provides it.
Change-Id: Iba2d904200d1b70140d81943a57b98fd290783f9
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/423491
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This allows us to avoid creating a shared configuration file in the dpdk
rte directory.
Change-Id: Id57d9111669ee8fd51e82c0b05e8c2c07b6dcd7e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/421652
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Now that DPDK can reserve additional memory on the fly,
the initial requested memory only serves as preallocation
and it makes sense to request no preallocated memory.
Right now, passing `--mem-size 0` counterintuitively
tries to reserve all hugepages on the system.
Change-Id: I281900efc7bd0bbb7eed30161ba0b88e16195170
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/422241
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
DPDK 18.08 includes the fixes we proposed
and we're now ready to make the switch.
Change-Id: Ie5abf25480d2449cb2c8e23de10c8090fc7b14fb
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/422240
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We used to support it by default in our DPDK forks,
but starting with DPDK 18.08, a new PCI driver flag
RTE_PCI_DRV_WC_ACTIVATE is required.
We enable now it for NVMe and Virtio, but not for I/OAT,
as our I/OAT driver currently assumes strong memory
ordering, which prefetchable resources do not provide.
Change-Id: I1a13356e28535981153b3d3e52bfe9d66b6172af
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/422239
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Starting with DPDK 18.08, rte_kvargs is a dependency
of rte_eal.
Change-Id: I0cde78f632fc313cec745d41ee519fb8b37de81b
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/422238
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
We were passing a map structure to unmap ioctl,
where we should obviously pass an unmap structure
instead.
Due to how map/unmap structures are, we were
unmapping much more memory than necessary on pci
hotremove.
Change-Id: Iabbea4b277228dc3de300849f751202f42bded2b
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/421693
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
DPDK 18.05 introduces a new "runtime" directory,
which contains hugepage metadata to be accessed
by secondary processes. There may be multiple
files per each memseg and we cannot easily predict
their count, so this patch deletes all regular
files in that directory.
$ ls -l /var/run/dpdk/spdk_pidX
total 16784
-rw------- 1 root root 4096 Jul 27 15:10 fbarray_memseg-1048576k-1-0
-rw------- 1 root root 188416 Jul 27 15:10 fbarray_memzone
-rw-r--r-- 1 root root 16974144 Jul 27 15:10 hugepage_data
-rw-r--r-- 1 root root 12432 Jul 27 15:10 hugepage_info
srwxr-xr-x 1 root root 0 Jul 27 15:10 mp_socket
The DPDK config file is not located in this directory
yet to preserve some of the backward compability.
Eventually, it will moved there as well.
Change-Id: I61cf1a47b306b51b0817c9d870841508f1e5e604
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/420659
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Currently passing pci whitelist and blacklist information to DPDK is done
by using single dash option -w and -b, respectively.
However this didn't work as reported in the Github issue #362.
Hence use two dashes option --pci-whitelist= and --pci-blacklist for
pci whitelist and blacklist, respectively.
In the Github issue #362, more issues are reported. These will be done
in subsequent patches.
Change-Id: I4e36edfce90e34577e34605d5b19658554fb2496
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/420211
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
SPDK holds off with using the new dynamic memory allocation
mode just yet.
Change-Id: I75541135a020c691d981d2b16e7c9509b1cc57b1
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/419567
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
DPDK has an option "huge-unlink" which can be used to
remove mmapped files after EAL initialization, SPDK
doesn't enable the option by default, ethier DPDK,
while here, export a new parameter which can let
user to decide enable it or not.
Fix issues #349 and #350.
Change-Id: Ic516b9f48f7b1c7c51712cc7bb7475ed904ff24b
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/419156
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Add a check to prevent spdk_mem_map_set_translation() or
spdk_mem_map_clear_translation() calls that start within the valid
address range but specify a size that would access parts of the mem map
outside of the valid region.
spdk_mem_map_translate() is safe without any extra checks since it only
accesses the first entry regardless of size, and the MASK_256TB check
catches out-of-range accesses to that entry.
Change-Id: Ie1437e57b5158363bb98a6b42a26fb41a089bbad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/418106
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
The arrays for both the map_256tb and map_1gb structures were twice
as large as necessary; fix the sizes and add unit tests for the boundary
conditions to verify that the fix works.
Change-Id: I66bce463f234f54e69cf2a697db9f806d398ca1e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/418105
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia413a913857dad00ae091b8cea02a647b8996a5c
Reviewed-on: https://review.gerrithub.io/418108
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This is an spdk_memzone_reserve variant with additional
alignment parameter. Now that memzones must be used for
physically contiguous memory, it will become extremely useful.
Change-Id: Ie48d682217e0e2f5c859a1603bb8a81fd2a7d7df
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416978
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Future DPDK versions may drop physical memory contiguity
guarantee for common memzones. DPDK 18.05 introduces
an RTE_MEMZONE_IOVA_CONTIG (0x00100000) flag, which is
documented as follows:
> RTE_MEMZONE_IOVA_CONTIG - Ensure reserved memzone is IOVA-contiguous.
> This option should be used when allocating
> memory intended for hardware rings etc.
To preserve backward compatibility, SPDK introduces an opposite
flag, SPDK_MEMZONE_NO_IOVA_CONTIG.
Change-Id: I9ea79b096fdb094051f13c9a802740b0e4ccc98e
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416977
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This const makes the array passed in immutable, but that
isn't helpful or interesting since it just contains
invalid pointer addresses. It may also make sense in the
future to NULL out the addresses in the array in a debug
build. So drop the const.
Change-Id: I921551c7cb1dbf6c765fb301c31906b8b93b7f16
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/417362
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Same as we do with SPDK_ENV_SOCKET_ID_ANY, memzone
flags should be translated from SPDK defines to their
DPDK equivalents instead of passing those 1:1.
Note that we don't use any memzone flags so far.
Change-Id: I4100dd7ac8825eecd7f323bc21b3b6b8347bfa8d
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416976
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
ASAN has some memory address regions that it will not
allow applications to use for mappings (including mmap).
So when specifying --base-virtaddr to DPDK for
shared/mutli-process mappings, pick an address that
ASAN will allow the application to map.
Ref: https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm
We will still disable ASLR while using the test stub for now.
Maybe we can eliminate this too in the future, but for now just
modify the autotest_common.sh comment since we've figured out
how to keep ASAN from messing with the mmap hint.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie24fd35bd22aa3ceab6271e8936775b157e5c330
Reviewed-on: https://review.gerrithub.io/416420
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
We used to check RTE_LIBRTE_VHOST_NUMA to decide
whether or not libnuma should be linked, but EAL
may need it as well. With our DPDK 18.02 fork we
build fine just because RTE_LIBRTE_VHOST_NUMA is
always set
Change-Id: I2bdf03b328d3ae074437565c8b66e2b6adcbfe89
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/415481
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
