ivampiresp/Spdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
549be9ad81
Spdk/docker/docker-compose.yaml

83 lines
1.9 KiB
YAML
Raw Normal View History

docker: Add docker-compose for building basic SPDK containers This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
2021-08-03 07:16:49 +00:00
# SPDX-License-Identifier: Apache-2.0
update Intel copyright notices per Intel policy to include file commit date using git cmd below. The policy does not apply to non-Intel (C) notices. git log --follow -C90% --format=%ad --date default <file> | tail -1 and then pull just the 4 digit year from the result. Intel copyrights were not added to files where Intel either had no contribution ot the contribution lacked substance (ie license header updates, formatting changes, etc). Contribution date used "--follow -C95%" to get the most accurate date. Note that several files in this patch didn't end the license/(c) block with a blank comment line so these were added as the vast majority of files do have this last blank line. Simply there for consistency. Signed-off-by: paul luse <paul.e.luse@intel.com> Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192 Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Ben Walker <benjamin.walker@intel.com> Community-CI: Mellanox Build Bot
2022-11-01 20:26:26 +00:00
# Copyright (C) 2021 Intel Corporation
#
docker: Add docker-compose for building basic SPDK containers This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
2021-08-03 07:16:49 +00:00
version: "3.8"
services:
build_base:
image: spdk
build:
context: build_base
container_name: build_base
storage-target:
image: spdk-app
build:
context: spdk-app
container_name: storage-target
depends_on:
- build_base
networks:
spdk:
ipv4_address: 192.168.42.2
volumes:
- /dev/hugepages:/dev/hugepages
- ./spdk-app/storage-target.conf:/config
environment:
- SPDK_ARGS=-m 0x2
privileged: true
proxy-container:
image: spdk-app
build:
context: spdk-app
container_name: proxy-container
depends_on:
- storage-target
networks:
spdk:
ipv4_address: 192.168.42.3
volumes:
- /dev/hugepages:/dev/hugepages
- ./spdk-app/proxy-container.conf:/config
docker: Add virtio traffic generator Expand the traffic generator container into two: - traffic-generator-nvme, which uses NVMe-oF to connect to proxy-container - traffic-generator-virtio, which uses Virtio to connect to proxy-container Added second device Malloc device in storage-target, and second subsystem shared between storage-target and proxy-container. The proxy-container and traffic-generator-virtio share named volume in order to pass the vhost socket file. Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I889dc19f523255f10b22e15f5e5f437b33ae796d Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9667 Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-09-27 14:17:10 +00:00
- vhost-user:/vhost-user
docker: Add docker-compose for building basic SPDK containers This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
2021-08-03 07:16:49 +00:00
environment:
docker: Add virtio traffic generator Expand the traffic generator container into two: - traffic-generator-nvme, which uses NVMe-oF to connect to proxy-container - traffic-generator-virtio, which uses Virtio to connect to proxy-container Added second device Malloc device in storage-target, and second subsystem shared between storage-target and proxy-container. The proxy-container and traffic-generator-virtio share named volume in order to pass the vhost socket file. Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I889dc19f523255f10b22e15f5e5f437b33ae796d Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9667 Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-09-27 14:17:10 +00:00
- SPDK_ARGS=-m 0x4 -S /vhost-user
docker: Add docker-compose for building basic SPDK containers This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
2021-08-03 07:16:49 +00:00
privileged: true
docker: Add virtio traffic generator Expand the traffic generator container into two: - traffic-generator-nvme, which uses NVMe-oF to connect to proxy-container - traffic-generator-virtio, which uses Virtio to connect to proxy-container Added second device Malloc device in storage-target, and second subsystem shared between storage-target and proxy-container. The proxy-container and traffic-generator-virtio share named volume in order to pass the vhost socket file. Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I889dc19f523255f10b22e15f5e5f437b33ae796d Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9667 Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-09-27 14:17:10 +00:00
traffic-generator-virtio:
docker: Add docker-compose for building basic SPDK containers This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
2021-08-03 07:16:49 +00:00
image: traffic-generator
build:
context: traffic-generator
docker: Add virtio traffic generator Expand the traffic generator container into two: - traffic-generator-nvme, which uses NVMe-oF to connect to proxy-container - traffic-generator-virtio, which uses Virtio to connect to proxy-container Added second device Malloc device in storage-target, and second subsystem shared between storage-target and proxy-container. The proxy-container and traffic-generator-virtio share named volume in order to pass the vhost socket file. Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I889dc19f523255f10b22e15f5e5f437b33ae796d Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9667 Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-09-27 14:17:10 +00:00
container_name: traffic-generator-virtio
docker: Add docker-compose for building basic SPDK containers This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
2021-08-03 07:16:49 +00:00
depends_on:
- proxy-container
networks:
spdk:
volumes:
- /dev/hugepages:/dev/hugepages
docker: Add virtio traffic generator Expand the traffic generator container into two: - traffic-generator-nvme, which uses NVMe-oF to connect to proxy-container - traffic-generator-virtio, which uses Virtio to connect to proxy-container Added second device Malloc device in storage-target, and second subsystem shared between storage-target and proxy-container. The proxy-container and traffic-generator-virtio share named volume in order to pass the vhost socket file. Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I889dc19f523255f10b22e15f5e5f437b33ae796d Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9667 Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-09-27 14:17:10 +00:00
- ./traffic-generator/conf-virtio:/config
- vhost-user:/vhost-user
- ./traffic-generator/fio-virtio.conf:/fio.conf
privileged: true
traffic-generator-nvme:
image: traffic-generator
build:
context: traffic-generator
container_name: traffic-generator-nvme
depends_on:
- proxy-container
networks:
spdk:
volumes:
- /dev/hugepages:/dev/hugepages
- ./traffic-generator/conf-nvme:/config
- ./traffic-generator/fio-nvme.conf:/fio.conf
docker: Add docker-compose for building basic SPDK containers This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
2021-08-03 07:16:49 +00:00
privileged: true
networks:
spdk:
name: "spdk"
ipam:
config:
- subnet: 192.168.42.0/29
gateway: 192.168.42.1
docker: Add virtio traffic generator Expand the traffic generator container into two: - traffic-generator-nvme, which uses NVMe-oF to connect to proxy-container - traffic-generator-virtio, which uses Virtio to connect to proxy-container Added second device Malloc device in storage-target, and second subsystem shared between storage-target and proxy-container. The proxy-container and traffic-generator-virtio share named volume in order to pass the vhost socket file. Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I889dc19f523255f10b22e15f5e5f437b33ae796d Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9667 Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-09-27 14:17:10 +00:00
volumes:
vhost-user:
Reference in New Issue Copy Permalink