huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-25 20:12:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix.

Browse Source
This commit is contained in:
Nicolas Patry 2024-10-23 17:26:18 +08:00
parent 5c3efbc763
commit 82a6cb82e1
No known key found for this signature in database
GPG Key ID: 788A1EA699458B2F
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
server/text_generation_server/models/flash_causal_lm.py
View File

@ -1418,9 +1418,9 @@ class FlashCausalLM(Model):
)
max_total_tokens = available_blocks
else:
max_total_tokens = len(batch.input_ids)
max_total_tokens = sum(len(input_ids) for input_ids in batch.input_ids)
max_input_tokens = (
batch.num_blocks - 1
max_total_tokens - 1
if max_input_tokens is None
else max_input_tokens
)