diff --git a/docs/source/reference/launcher.md b/docs/source/reference/launcher.md index 159b22e7..2d9b58d9 100644 --- a/docs/source/reference/launcher.md +++ b/docs/source/reference/launcher.md @@ -198,7 +198,7 @@ Options: For `max_batch_total_tokens=1000`, you could fit `10` queries of `total_tokens=100` or a single query of `1000` tokens. - Overall this number should be the largest possible amount that fits the remaining memory (after the model is loaded). Since the actual memory overhead depends on other parameters like if you're using quantization, flash attention or the model implementation, text-generation-inference cannot infer this number automatically. + Overall this number should be the largest possible amount that fits the remaining memory (after the model is loaded). Since the actual memory overhead depends on other parameters like if you're using quantization, flash attention or the model implementation, text-generation-inference infers this number automatically if not provided ensuring that the value is as large as possible. [env: MAX_BATCH_TOTAL_TOKENS=] diff --git a/launcher/src/main.rs b/launcher/src/main.rs index cd4b2231..321d7c69 100644 --- a/launcher/src/main.rs +++ b/launcher/src/main.rs @@ -702,8 +702,8 @@ struct Args { /// Overall this number should be the largest possible amount that fits the /// remaining memory (after the model is loaded). Since the actual memory overhead /// depends on other parameters like if you're using quantization, flash attention - /// or the model implementation, text-generation-inference cannot infer this number - /// automatically. + /// or the model implementation, text-generation-inference infers this number automatically + /// if not provided ensuring that the value is as large as possible. #[clap(long, env)] max_batch_total_tokens: Option,