huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-24 08:22:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
993 Commits 229 Branches 59 Tags 23 MiB
118ee57f82
Commit Graph

3 Commits

Author SHA1 Message Date
Daniël de Kok
85c3c5d64f Add support for FP8 on compute capability >=8.0, <8.9 (#2213) 
Use FP8 GPTQ-Marlin kernels to enable FP8 support on CUDA GPUs
with compute capability >=8.0 and <8.9.

Co-authored-by: Florian Zimmermeister <flozi00.fz@gmail.com>
 2024-09-25 05:27:40 +00:00
Daniël de Kok
4700ea413f Add support for Marlin 2:4 sparsity (#2102) 
This change adds support for 2:4 sparsity when using Marlin
quantization. The 2:4 kernel is used when:

* The quantizer is `marlin`;
* the quantizer checkpoint format is `marlin_24`.

Fixes #2098.
 2024-09-24 03:55:04 +00:00
Daniël de Kok
f1f28404e7 Add support for GPTQ Marlin (#2052) 
Add support for GPTQ Marlin kernels

GPTQ Marlin extends the Marlin kernels to support common GPTQ
configurations:

- bits: 4 or 8
- groupsize: -1, 32, 64, or 128
- desc_act: true/false

Using the GPTQ Marlin kernels requires repacking the parameters in the
Marlin quantizer format.

The kernels were contributed by Neural Magic to VLLM. We vendor them
here for convenience.
 2024-09-24 03:43:30 +00:00