From 181dcb6219f8d7b773d6ed5db61a037b50eee684 Mon Sep 17 00:00:00 2001 From: Merve Noyan Date: Mon, 21 Aug 2023 11:14:05 +0300 Subject: [PATCH] Nits --- docs/source/conceptual/flash_attention.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/conceptual/flash_attention.md b/docs/source/conceptual/flash_attention.md index 5214d96d..47511cff 100644 --- a/docs/source/conceptual/flash_attention.md +++ b/docs/source/conceptual/flash_attention.md @@ -1,7 +1,7 @@ # Flash Attention -Scaling transformer architecture is heavily bottlenecked by self-attention mechanism, which has quadratic time and memory complexity. Recent developments in accelerator hardware are mainly focused on enhancing compute capacities, and not memory and transferring data between hardware. This results in attention operation to have a bottleneck in memory, also called as _memory-bound_. Flash Attention is an attention algorithm used to overcome this problem and scale transformer based models more efficiently, enabling faster training and inference. -In standard attention implementation, cost of loading and writing key, query, values from High Bandwidth Memory (HBM) is high. It loads key, query, value from HBM to GPU, performs each singular step of attention mechanism and writes it back to HBM repeatedly. Instead, Flash Attention loads Q, K, V once, fuses the operations in attention mechanism and writes it back. -It is implemented for models with custom kernels, you can check out full list of models that support Flash Attention [here](https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/models). +Scaling transformer architecture is heavily bottlenecked by the self-attention mechanism, which has quadratic time and memory complexity. Recent developments in accelerator hardware are mainly focused on enhancing compute capacities and not memory and transferring data between hardware. This results in attention operation having a bottleneck in memory, also called _memory-bound_. Flash Attention is an attention algorithm used to overcome this problem and scale transformer-based models more efficiently, enabling faster training and inference. +In standard attention implementation, the cost of loading and writing keys, queries, and values from High Bandwidth Memory (HBM) is high. It loads key, query, value from HBM to GPU, performs a single step of the attention mechanism and writes it back to HBM, and repeats this for every singular step of the attention. Instead, Flash Attention loads keys, queries, and values once, fuses the operations of the attention mechanism and writes them back. +It is implemented for models with custom kernels, you can check out the full list of models that support Flash Attention [here](https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/models).