huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-10 20:04:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Merve Noyan 2023-08-21 00:23:10 +03:00 committed by GitHub
parent c4422e5678
commit 08bf10ca17
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/source/conceptual/flash_attention.md Normal file
View File

@ -0,0 +1,7 @@
# Flash Attention
Scaling transformer architecture is heavily bottlenecked by self-attention mechanism, which has quadratic time and memory complexity. Recent developments in accelerator hardware are mainly focused on enhancing compute capacities, and not memory and transferring data between hardware. This results in attention operation to have a bottleneck in memory, also called as _memory-bound_. Flash Attention is an attention algorithm used to overcome this problem and scale transformer based models more efficiently, enabling faster training and inference.
In standard attention implementation, cost of loading and writing key, query, values from High Bandwidth Memory (HBM) is high. It loads key, query, value from HBM to GPU, performs each singular step of attention mechanism and writes it back to HBM repeatedly. Instead, Flash Attention loads Q, K, V once, fuses the operations in attention mechanism and writes it back.
It is implemented for models with custom kernels, you can check out full list of models that support Flash Attention [here](https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/models).