huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-07-26 01:40:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Attempt #2

Browse Source
This commit is contained in:
Nicolas Patry 2024-10-23 15:23:39 +08:00
parent 0a01dde986
commit 5c3efbc763
No known key found for this signature in database
GPG Key ID: 788A1EA699458B2F
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
server/text_generation_server/models/flash_causal_lm.py
View File

@ -1418,7 +1418,7 @@ class FlashCausalLM(Model):
) )
max_total_tokens = available_blocks max_total_tokens = available_blocks
else: else:
max_total_tokens = batch.num_blocks max_total_tokens = len(batch.input_ids)
max_input_tokens = ( max_input_tokens = (
batch.num_blocks - 1 batch.num_blocks - 1
if max_input_tokens is None if max_input_tokens is None