huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-07-25 09:20:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Attempt #2

Browse Source
This commit is contained in:
Nicolas Patry 2024-10-23 15:23:39 +08:00
parent 0a01dde986
commit 5c3efbc763
No known key found for this signature in database
GPG Key ID: 788A1EA699458B2F
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
server/text_generation_server/models/flash_causal_lm.py
View File

@ -1418,7 +1418,7 @@ class FlashCausalLM(Model):
)
max_total_tokens = available_blocks
else:
max_total_tokens = batch.num_blocks
max_total_tokens = len(batch.input_ids)
max_input_tokens = (
batch.num_blocks - 1
if max_input_tokens is None