huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-11-18 15:05:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
feat/backend_feature
text-generation-inference/server/text_generation_server/layers
History
Daniël de Kok 093a27c528
Add support for GPTQ Marlin (#2052) 
Add support for GPTQ Marlin kernels

GPTQ Marlin extends the Marlin kernels to support common GPTQ
configurations:

- bits: 4 or 8
- groupsize: -1, 32, 64, or 128
- desc_act: true/false

Using the GPTQ Marlin kernels requires repacking the parameters in the
Marlin quantizer format.

The kernels were contributed by Neural Magic to VLLM. We vendor them
here for convenience.
2024-06-14 09:45:42 +02:00
..
attention ROCm and sliding windows fixes (#2033) 2024-06-10 15:09:50 +08:00
awq Refactor layers. (#1866) 2024-05-13 12:44:30 +02:00
gptq Fix GPTQWeight import (#2020) 2024-06-05 14:49:15 +02:00
__init__.py MLPSpeculator. (#1865) 2024-05-14 12:33:18 +02:00
bnb.py [Bug Fix] Update torch import reference in bnb quantization (#1902) 2024-05-15 21:08:32 +02:00
conv.py Refactor layers. (#1866) 2024-05-13 12:44:30 +02:00
eetq.py Refactor layers. (#1866) 2024-05-13 12:44:30 +02:00
exl2.py Add support for exl2 quantization 2024-05-30 11:28:05 +02:00
fp8.py Refactor layers. (#1866) 2024-05-13 12:44:30 +02:00
layernorm.py MI300 compatibility (#1764) 2024-05-17 15:30:47 +02:00
linear.py Add support for GPTQ Marlin (#2052) 2024-06-14 09:45:42 +02:00
marlin.py Add support for GPTQ Marlin (#2052) 2024-06-14 09:45:42 +02:00
medusa.py fix: use path inside of speculator config (#1935) 2024-05-22 20:46:29 +02:00
mlp.py MLPSpeculator. (#1865) 2024-05-14 12:33:18 +02:00
rotary.py fix(layers): fix SuRotaryEmbedding (#2060) 2024-06-12 18:24:47 +02:00
speculative.py MLPSpeculator. (#1865) 2024-05-14 12:33:18 +02:00
tensor_parallel.py Add Phi-3 medium support (#2039) 2024-06-10 09:22:29 +02:00