huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-12 12:54:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix in regression in ipex flashattention

Browse Source 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
This commit is contained in:
Wang, Yi A 2024-08-06 22:00:36 -07:00
parent 30e70f2ceb
commit 6abcab843d
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
server/text_generation_server/layers/attention/ipex.py
View File

@ -21,7 +21,7 @@ def attention(
out = torch.empty_like(q)
# We do not need to check window_size_left (not supported) here, so it is already checked ahead of time at model load.
return ipex.llm.functional.varlen_attention(
ipex.llm.functional.varlen_attention(
q,
k,
v,
@ -38,6 +38,8 @@ def attention(
None,
)
return out
def reshape_and_cache(
key: torch.Tensor,