huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-11 20:34:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Changing the waiting_served_ratio default (stack more aggressively by

Browse Source 
default).
This commit is contained in:
Nicolas Patry 2024-04-26 19:16:39 +02:00
parent a8fd4236eb
commit 80c23bdd38
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
launcher/src/main.rs
View File

@ -251,7 +251,7 @@ struct Args {
/// ///
/// This setting is only applied if there is room in the batch /// This setting is only applied if there is room in the batch
/// as defined by `max_batch_total_tokens`. /// as defined by `max_batch_total_tokens`.
#[clap(default_value = "1.2", long, env)] #[clap(default_value = "0.3", long, env)]
waiting_served_ratio: f32, waiting_served_ratio: f32,
/// Limits the number of tokens for the prefill operation. /// Limits the number of tokens for the prefill operation.