huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-11 04:14:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Enable KQV offload by default

Browse Source 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
This commit is contained in:
Adrien Gallouët 2025-02-06 18:33:30 +00:00
parent 809e288b5a
commit 3b1b049b32
No known key found for this signature in database
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
backends/llamacpp/src/main.rs
View File

@ -60,7 +60,7 @@ struct Args {
use_mlock: bool,
/// Enable offloading of KQV operations to the GPU.
#[clap(default_value = "false", long, env)]
#[clap(default_value = "true", long, env)]
offload_kqv: bool,
/// Enable flash attention for faster inference. (EXPERIMENTAL)