huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-22 15:32:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update the doc.

Browse Source
This commit is contained in:
Nicolas Patry 2024-04-12 12:09:23 +00:00
parent b75bd5b720
commit f66c9f340b
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/basic_tutorials/launcher.md
View File

@ -168,7 +168,7 @@ Options:
## MAX_BATCH_PREFILL_TOKENS ## MAX_BATCH_PREFILL_TOKENS
```shell ```shell
--max-batch-prefill-tokens <MAX_BATCH_PREFILL_TOKENS> --max-batch-prefill-tokens <MAX_BATCH_PREFILL_TOKENS>
Limits the number of tokens for the prefill operation. Since this operation take the most memory and is compute bound, it is interesting to limit the number of requests that can be sent. Default to `max_input_length + 50` to give a bit of room Limits the number of tokens for the prefill operation. Since this operation take the most memory and is compute bound, it is interesting to limit the number of requests that can be sent. Default to `max_input_tokens + 50` to give a bit of room
[env: MAX_BATCH_PREFILL_TOKENS=] [env: MAX_BATCH_PREFILL_TOKENS=]