huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-09 19:34:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

really fix pruning of cancelled requests from queue

Browse Source
This commit is contained in:
Nick Hill 2023-05-16 14:26:46 -07:00
parent 51355ea722
commit 9b1868b2c2
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
router/src/queue.rs
View File

@ -456,7 +456,7 @@ impl<B: BatchType> State<B> {
batch_stats = next_stats;
chosen_indices.push(queue_index - 1);
chosen_indices.push(checked_up_to_index - 1);
total_count += 1;
if total_count >= config.size_limit {
break