mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-24 16:32:12 +00:00
2ce8019480
GPTQ-Marlin is currently the best-performing kernel for GPTQ models. So let's use it by default if the kernels are installed, the GPU supports it, and the kernels support the configuration. For models generated by `text-generation-server quantize`, use `sym=False`. This subcommand symmetric quantization since the beginning and incorrectly reporting the model to be symmetric will use GPTQ-Marlin (which does not support asymmetric quantization). |
||
|---|---|---|
| .. | ||
| attention | ||
| awq | ||
| gptq | ||
| __init__.py | ||
| bnb.py | ||
| conv.py | ||
| eetq.py | ||
| exl2.py | ||
| fp8.py | ||
| layernorm.py | ||
| linear.py | ||
| lora.py | ||
| marlin.py | ||
| medusa.py | ||
| mlp.py | ||
| rotary.py | ||
| speculative.py | ||
| tensor_parallel.py | ||