huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-24 16:32:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update docstring in launcher/src/main.rs instead

Browse Source
This commit is contained in:
Alvaro Bartolome 2025-03-07 13:25:24 +01:00
parent 7a40844734
commit a1b3887846
No known key found for this signature in database
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
launcher/src/main.rs
View File

@ -702,8 +702,8 @@ struct Args {
/// Overall this number should be the largest possible amount that fits the
/// remaining memory (after the model is loaded). Since the actual memory overhead
/// depends on other parameters like if you're using quantization, flash attention
/// or the model implementation, text-generation-inference cannot infer this number
/// automatically.
/// or the model implementation, text-generation-inference infers this number automatically
/// if not provided ensuring that the value is as large as possible.
#[clap(long, env)]
max_batch_total_tokens: Option<u32>,