huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-22 15:32:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add qwen2 fp8 support (#210)

Browse Source 
Signed-off-by: changwang <changwang@habana.ai>
Co-authored-by: changwang <changwang@habana.ai>
This commit is contained in:
Wang, Chang 2024-08-26 17:02:58 +08:00 committed by GitHub
parent e33db1877c
commit 55d60a103c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
server/text_generation_server/models/causal_lm.py
View File

@ -691,9 +691,9 @@ class CausalLM(Model):
"return_dict": True,
}
if model.config.model_type in ["llama", "mistral", "starcoder2"]:
if model.config.model_type in ["llama", "mistral", "starcoder2", "qwen2"]:
if model.config.model_type in ["llama", "mistral"]:
if model.config.model_type in ["llama", "mistral", "qwen2"]:
kwargs["attn_softmax_bf16"] = True
kwargs["trim_logits"] = True