mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-21 14:52:20 +00:00
85dfc39222
Add support for Phi-3-medium The main difference between the medium and mini models is that medium uses grouped query attention with a packed QKV matrix. This change adds support for GQA with packed matrixes to `Weights.get_weights_col_packed` and uses it for Phi-3. This also allows us to remove the custom implementation of GQA from dbrx attention loading. |
||
|---|---|---|
| .. | ||
| attention | ||
| awq | ||
| gptq | ||
| __init__.py | ||
| bnb.py | ||
| conv.py | ||
| eetq.py | ||
| exl2.py | ||
| fp8.py | ||
| layernorm.py | ||
| linear.py | ||
| marlin.py | ||
| medusa.py | ||
| mlp.py | ||
| rotary.py | ||
| speculative.py | ||
| tensor_parallel.py | ||