Spdk/lib/env_dpdk/env_internal.h

56 lines
1.5 KiB
C
Raw Normal View History

/* SPDX-License-Identifier: BSD-3-Clause
* Copyright (c) Intel Corporation.
* All rights reserved.
*/
#ifndef SPDK_ENV_INTERNAL_H
#define SPDK_ENV_INTERNAL_H
#include "spdk/stdinc.h"
#include "spdk/env.h"
#include <rte_config.h>
#include <rte_version.h>
#include <rte_eal.h>
#if RTE_VERSION < RTE_VERSION_NUM(19, 11, 0, 0)
#error RTE_VERSION is too old! Minimum 19.11 is required.
#endif
/* x86-64 and ARM userspace virtual addresses use only the low 48 bits [0..47],
* which is enough to cover 256 TB.
*/
#define SHIFT_256TB 48 /* (1 << 48) == 256 TB */
#define MASK_256TB ((1ULL << SHIFT_256TB) - 1)
#define SHIFT_1GB 30 /* (1 << 30) == 1 GB */
#define MASK_1GB ((1ULL << SHIFT_1GB) - 1)
int pci_env_init(void);
void pci_env_reinit(void);
void pci_env_fini(void);
int mem_map_init(bool legacy_mem);
int vtophys_init(void);
int vtophys_iommu_map_dma_bar(uint64_t vaddr, uint64_t iova, uint64_t size);
int vtophys_iommu_unmap_dma_bar(uint64_t vaddr);
struct rte_pci_device;
vtophys: remap vfio dma memory when necessary VFIO requires at least one IOMMU group to be added to the VFIO container to be able to perform any IOMMU operations on that container. [1] Without any groups added, VFIO_IOMMU_MAP_DMA would always respond with errno 22 (Invalid argument). Also, if the last IOMMU group is removed from the container (device hotremove), all the IOMMU mappings are lost. In both cases we need to remap vfio memory as soon as the first IOMMU group is attached. The attach is done inside DPDK during device attach and we can't hook into it directly. Instead, this patch hooks into our PCI init/fini callbacks. There's now a PCI device ref counter in our vfio manager and a history of all registered memory pages. When the refcount is increased from 0 to 1, the vtophys will remap all vfio dma memory. [1] https://www.kernel.org/doc/Documentation/vfio.txt "On its own, the container provides little functionality, with all but a couple version and extension query interfaces locked away. The user needs to add a group into the container for the next level of functionality. [...] With a group (or groups) attached to a container, the remaining ioctls become available, enabling access to the VFIO IOMMU interfaces." Change-Id: I744e07043dbe7ffd433fc95d604dad39647675f4 Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com> Reviewed-on: https://review.gerrithub.io/390655 Tested-by: SPDK Automated Test System <sys_sgsw@intel.com> Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-12-06 21:52:56 +00:00
/**
* Report a DMA-capable PCI device to the vtophys translation code.
* Increases the refcount of active DMA-capable devices managed by SPDK.
vtophys: remap vfio dma memory when necessary VFIO requires at least one IOMMU group to be added to the VFIO container to be able to perform any IOMMU operations on that container. [1] Without any groups added, VFIO_IOMMU_MAP_DMA would always respond with errno 22 (Invalid argument). Also, if the last IOMMU group is removed from the container (device hotremove), all the IOMMU mappings are lost. In both cases we need to remap vfio memory as soon as the first IOMMU group is attached. The attach is done inside DPDK during device attach and we can't hook into it directly. Instead, this patch hooks into our PCI init/fini callbacks. There's now a PCI device ref counter in our vfio manager and a history of all registered memory pages. When the refcount is increased from 0 to 1, the vtophys will remap all vfio dma memory. [1] https://www.kernel.org/doc/Documentation/vfio.txt "On its own, the container provides little functionality, with all but a couple version and extension query interfaces locked away. The user needs to add a group into the container for the next level of functionality. [...] With a group (or groups) attached to a container, the remaining ioctls become available, enabling access to the VFIO IOMMU interfaces." Change-Id: I744e07043dbe7ffd433fc95d604dad39647675f4 Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com> Reviewed-on: https://review.gerrithub.io/390655 Tested-by: SPDK Automated Test System <sys_sgsw@intel.com> Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-12-06 21:52:56 +00:00
* This must be called after a `rte_pci_device` is created.
*/
void vtophys_pci_device_added(struct rte_pci_device *pci_device);
vtophys: remap vfio dma memory when necessary VFIO requires at least one IOMMU group to be added to the VFIO container to be able to perform any IOMMU operations on that container. [1] Without any groups added, VFIO_IOMMU_MAP_DMA would always respond with errno 22 (Invalid argument). Also, if the last IOMMU group is removed from the container (device hotremove), all the IOMMU mappings are lost. In both cases we need to remap vfio memory as soon as the first IOMMU group is attached. The attach is done inside DPDK during device attach and we can't hook into it directly. Instead, this patch hooks into our PCI init/fini callbacks. There's now a PCI device ref counter in our vfio manager and a history of all registered memory pages. When the refcount is increased from 0 to 1, the vtophys will remap all vfio dma memory. [1] https://www.kernel.org/doc/Documentation/vfio.txt "On its own, the container provides little functionality, with all but a couple version and extension query interfaces locked away. The user needs to add a group into the container for the next level of functionality. [...] With a group (or groups) attached to a container, the remaining ioctls become available, enabling access to the VFIO IOMMU interfaces." Change-Id: I744e07043dbe7ffd433fc95d604dad39647675f4 Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com> Reviewed-on: https://review.gerrithub.io/390655 Tested-by: SPDK Automated Test System <sys_sgsw@intel.com> Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-12-06 21:52:56 +00:00
/**
* Report the removal of a DMA-capable PCI device to the vtophys translation code.
* Decreases the refcount of active DMA-capable devices managed by SPDK.
vtophys: remap vfio dma memory when necessary VFIO requires at least one IOMMU group to be added to the VFIO container to be able to perform any IOMMU operations on that container. [1] Without any groups added, VFIO_IOMMU_MAP_DMA would always respond with errno 22 (Invalid argument). Also, if the last IOMMU group is removed from the container (device hotremove), all the IOMMU mappings are lost. In both cases we need to remap vfio memory as soon as the first IOMMU group is attached. The attach is done inside DPDK during device attach and we can't hook into it directly. Instead, this patch hooks into our PCI init/fini callbacks. There's now a PCI device ref counter in our vfio manager and a history of all registered memory pages. When the refcount is increased from 0 to 1, the vtophys will remap all vfio dma memory. [1] https://www.kernel.org/doc/Documentation/vfio.txt "On its own, the container provides little functionality, with all but a couple version and extension query interfaces locked away. The user needs to add a group into the container for the next level of functionality. [...] With a group (or groups) attached to a container, the remaining ioctls become available, enabling access to the VFIO IOMMU interfaces." Change-Id: I744e07043dbe7ffd433fc95d604dad39647675f4 Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com> Reviewed-on: https://review.gerrithub.io/390655 Tested-by: SPDK Automated Test System <sys_sgsw@intel.com> Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-12-06 21:52:56 +00:00
* This must be called before a `rte_pci_device` is destroyed.
*/
void vtophys_pci_device_removed(struct rte_pci_device *pci_device);
vtophys: remap vfio dma memory when necessary VFIO requires at least one IOMMU group to be added to the VFIO container to be able to perform any IOMMU operations on that container. [1] Without any groups added, VFIO_IOMMU_MAP_DMA would always respond with errno 22 (Invalid argument). Also, if the last IOMMU group is removed from the container (device hotremove), all the IOMMU mappings are lost. In both cases we need to remap vfio memory as soon as the first IOMMU group is attached. The attach is done inside DPDK during device attach and we can't hook into it directly. Instead, this patch hooks into our PCI init/fini callbacks. There's now a PCI device ref counter in our vfio manager and a history of all registered memory pages. When the refcount is increased from 0 to 1, the vtophys will remap all vfio dma memory. [1] https://www.kernel.org/doc/Documentation/vfio.txt "On its own, the container provides little functionality, with all but a couple version and extension query interfaces locked away. The user needs to add a group into the container for the next level of functionality. [...] With a group (or groups) attached to a container, the remaining ioctls become available, enabling access to the VFIO IOMMU interfaces." Change-Id: I744e07043dbe7ffd433fc95d604dad39647675f4 Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com> Reviewed-on: https://review.gerrithub.io/390655 Tested-by: SPDK Automated Test System <sys_sgsw@intel.com> Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-12-06 21:52:56 +00:00
#endif