huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-21 23:12:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
851 Commits 228 Branches 59 Tags 47 MiB
cb150eb295
Commit Graph

3 Commits

Author SHA1 Message Date
Daniël de Kok
cb150eb295
Add support for FP8 on compute capability >=8.0, <8.9 (#2213) 
Use FP8 GPTQ-Marlin kernels to enable FP8 support on CUDA GPUs
with compute capability >=8.0 and <8.9.

Co-authored-by: Florian Zimmermeister <flozi00.fz@gmail.com>
 2024-07-11 16:03:26 +02:00
Daniël de Kok
f1f98e369f
Add support for Marlin 2:4 sparsity (#2102) 
This change adds support for 2:4 sparsity when using Marlin
quantization. The 2:4 kernel is used when:

* The quantizer is `marlin`;
* the quantizer checkpoint format is `marlin_24`.

Fixes #2098.
 2024-06-25 21:09:42 +02:00
Daniël de Kok
093a27c528
Add support for GPTQ Marlin (#2052) 
Add support for GPTQ Marlin kernels

GPTQ Marlin extends the Marlin kernels to support common GPTQ
configurations:

- bits: 4 or 8
- groupsize: -1, 32, 64, or 128
- desc_act: true/false

Using the GPTQ Marlin kernels requires repacking the parameters in the
Marlin quantizer format.

The kernels were contributed by Neural Magic to VLLM. We vendor them
here for convenience.
 2024-06-14 09:45:42 +02:00