huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-11 12:24:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update docs after rebase.

Browse Source
This commit is contained in:
Nicolas Patry 2024-02-09 11:44:42 +00:00
parent 4b06f318cb
commit 7143130ba4
1 changed files with 2 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/source/basic_tutorials/launcher.md
View File

@ -198,23 +198,22 @@ Options:
[default: 20] [default: 20]
``` ```
## MAX_BATCH_SIZE ## MAX_BATCH_SIZE
```shell ```shell
--max-batch-size <MAX_BATCH_SIZE> --max-batch-size <MAX_BATCH_SIZE>
Enforce a maximum number of requests per batch Specific flag for hardware targets that do not support unpadded inference Enforce a maximum number of requests per batch Specific flag for hardware targets that do not support unpadded inference
[env: MAX_BATCH_SIZE=] [env: MAX_BATCH_SIZE=]
```
```
## ENABLE_CUDA_GRAPHS ## ENABLE_CUDA_GRAPHS
```shell ```shell
--enable-cuda-graphs --enable-cuda-graphs
Enable experimental support for cuda graphs Enable experimental support for cuda graphs
[env: ENABLE_CUDA_GRAPHS=] [env: ENABLE_CUDA_GRAPHS=]
```
```
## HOSTNAME ## HOSTNAME
```shell ```shell
--hostname <HOSTNAME> --hostname <HOSTNAME>