doc: update vhost doc

Remove experimental attribute from multiqueue section, add description
of QEMU num_queues parameter and expand example.

Change-Id: I13d67ece0734aa8c57eeb07eabc34442ca532d47
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-on: https://review.gerrithub.io/386501
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Paweł Niedźwiecki <pawelx.niedzwiecki@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

doc/vhost.md

@@ -1,6 +1,6 @@
 # vhost {#vhost}
 
-# vhost Getting Started Guide {#vhost_getting_started}
+# Getting Started Guide {#vhost_getting_started}
 
 The Storage Performance Development Kit vhost application is named `vhost`.
 This application extends SPDK to present virtio storage controllers to QEMU-based
@@ -108,12 +108,12 @@ pick one core from `ReactorMask` masked by `Cpumask`. `Cpumask` must be a subset
 
 Userspace vhost-scsi adds the following command line option for QEMU:
 ~~~
--device vhost-user-scsi-pci,id=scsi0,chardev=char0
+-device vhost-user-scsi-pci,id=scsi0,chardev=char0[,num_queues=N]
 ~~~
 
 Userspace vhost-blk adds the following command line option for QEMU:
 ~~~
--device vhost-user-blk-pci,logical_block_size=4096,size=512M,chardev=char0
+-device vhost-user-blk-pci,logical_block_size=4096,size=512M,chardev=char0[,num_queues=N]
 ~~~
 
 In order to start qemu with vhost you need to specify following options:
@@ -140,29 +140,9 @@ A full list of command line arguments to vhost can be obtained by:
 app/vhost/vhost -h
 ~~~
 
-
-## Example
-
-Assume that qemu and spdk are in respectively `qemu` and `spdk` directories.
-~~~
-./qemu/build/x86_64-softmmu/qemu-system-x86_64 \
-        -m 1024 \
-        -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on \
-        -numa node,memdev=mem \
-        -drive file=$PROJECTS/os.qcow2,if=none,id=disk \
-        -device ide-hd,drive=disk,bootindex=0 \
-        -chardev socket,id=char0,path=./spdk/vhost.0 \
-        -device vhost-user-scsi-pci,id=scsi0,chardev=char0 \
-        -chardev socket,id=char1,path=./spdk/vhost.1 \
-        -device vhost-user-blk-pci,logical_block_size=4096,size=512M,chardev=char1 \
-        --enable-kvm
-~~~
-
-# Experimental Features {#vhost_experimental}
-
 ## Multi-Queue Block Layer (blk-mq)
 
-It is possible to use the Linux kernel block multi-queue feature with vhost.
+For best performance use the Linux kernel block multi-queue feature with vhost.
 To enable it on Linux, it is required to modify kernel options inside the
 virtual machine.
 
@@ -174,7 +154,88 @@ Instructions below for Ubuntu OS:
 4. Reboot virtual machine
 
 To achieve better performance, make sure to increase number of cores
-assigned to the VM.
+assigned to the VM and add `num_queues` parameter to the QEMU `device`. It should be enough
+to set `num_queues=4` to saturate physical device. Adding too many queues might lead to SPDK
+vhost performance degradation if many vhost devices are used because each device will require
+additional `num_queues` to be polled.
+
+# Example {#vhost_example}
+
+Run SPDK vhost with two controllers: Virtio SCSI and Virtio block.
+
+Virtio SCSI controller with two LUNs:
+
+- SCSI target 1, LUN 0 backed by Malloc0 bdev
+- SCSI target 5, LUN 0 backed by Malloc1 bdev
+
+Virtio block device backed by Malloc2 bdev
+
+For better performance use 4 VCPU (`-smp 4`) and 4 queues for each controller
+(`num_queues=4`). Assume that QEMU and SPDK are in respectively `qemu` and `spdk` directories.
+~~~
+host: $ cd spdk
+host: $ cat vhost.conf
+[Rpc]
+  Enable Yes
+  Listen 127.0.0.1:5261
+
+[Malloc]
+  NumberOfLuns 3
+  LunSizeInMb 128
+  BlockSize 512
+
+[VhostScsi0]
+  Name vhost_scsi0_socket
+  Dev 1 Malloc0
+  Dev 5 Malloc1
+
+[VhostBlk0]
+  Name vhost_blk0_socket
+  Dev Malloc2
+
+host: $ sudo ./app/vhost/vhost -c vhost.conf -s 1024 -m 1 &
+[ DPDK EAL parameters: vhost -c 1 -m 1024 --file-prefix=spdk_pid191213 ]
+EAL: Detected 48 lcore(s)
+EAL: Probing VFIO support...
+EAL: VFIO support initialized
+<< REMOVED CONSOLE LOG >>
+VHOST_CONFIG: bind to vhost_scsi0_socket
+vhost.c: 592:spdk_vhost_dev_construct: *NOTICE*: Controller vhost_scsi0_socket: new controller added
+vhost_scsi.c: 840:spdk_vhost_scsi_dev_add_dev: *NOTICE*: Controller vhost_scsi0_socket: defined device 'Dev 1' using lun 'Malloc0'
+vhost_scsi.c: 840:spdk_vhost_scsi_dev_add_dev: *NOTICE*: Controller vhost_scsi0_socket: defined device 'Dev 5' using lun 'Malloc1'
+VHOST_CONFIG: vhost-user server: socket created, fd: 65
+VHOST_CONFIG: bind to vhost_blk0_socket
+vhost.c: 592:spdk_vhost_dev_construct: *NOTICE*: Controller vhost_blk0_socket: new controller added
+vhost_blk.c: 720:spdk_vhost_blk_construct: *NOTICE*: Controller vhost_blk0_socket: using bdev 'Malloc2'
+
+host: $ cd ..
+host: $ sudo ./qemu/build/x86_64-softmmu/qemu-system-x86_64 --enable-kvm -m 1024 \
+  -cpu host -smp 4 -nographic \
+  -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem \
+  -drive file=guest_os_image.qcow2,if=none,id=disk \
+  -device ide-hd,drive=disk,bootindex=0 \
+  -chardev socket,id=spdk_vhost_scsi0,path=./spdk/vhost_scsi0_socket \
+  -device vhost-user-scsi-pci,id=scsi0,chardev=spdk_vhost_scsi0,num_queues=4 \
+  -chardev socket,id=spdk_vhost_blk0,path=./spdk/vhost_blk0_socket \
+  -device vhost-user-blk-pci,logical_block_size=512,size=128M,chardev=spdk_vhost_blk0,num_queues=4
+
+<< LOGIN TO GUEST OS >>
+guest: ~$ lsblk --output "NAME,KNAME,MODEL,HCTL,SIZE,VENDOR,SUBSYSTEMS"
+NAME   KNAME MODEL            HCTL        SIZE VENDOR   SUBSYSTEMS
+fd0    fd0                                  4K          block:platform
+sda    sda   QEMU HARDDISK    1:0:0:0      80G ATA      block:scsi:pci
+  sda1 sda1                                80G          block:scsi:pci
+sdb    sdb   Malloc disk      2:0:1:0     128M INTEL    block:scsi:virtio:pci
+sdc    sdc   Malloc disk      2:0:5:0     128M INTEL    block:scsi:virtio:pci
+vda    vda                                128M 0x1af4   block:virtio:pci
+
+guest: $ sudo poweroff
+host: $ fg
+<< CTRL + C >>
+vhost.c:1006:session_shutdown: *NOTICE*: Exiting
+~~~
+
+We can see that `sdb` and `sdc` are SPDK vhost-scsi LUNs, and `vda` is SPDK vhost-blk disk.
 
 # Known bugs and limitations {#vhost_bugs}