Daniël de Kok 2dd680b799 Improve the handling of quantized weights (#2250) ... * Improve the handling of quantized weights Handling of quantized weights was split between two mechanisms: - For quantized checkpoints, we used the new weight loader infrastructure. - For quantization while loading (EETQ, FP8, bitsandbytes) we instead relied on conditional in `get_linear`. Weight loaders support context managers to selectively load particular layers with different weight loaders, which is useful for models like Idefics2 AWQ, which uses a quantized text model, but unquantized vision and connector models. However, the context manager would be overrided by `get_linear`, which string-checks `quantizer`. Also, the context manager would not work with EETQ, FP8, and bitsandbytes. This change migrates all quantizers to the weight loader infrastructure. This has several benefits: - We can use context managers with all quantizers. - All the implementation details move down to the quantizer layers, `get_linear` does not need to know how to handle quantizer linear layers. - All quantizer weights are strongly typed, we don't pass around raw tensors. - We don't have to pass around the `quantizer` string everywhere. * Exclude non-MLP layers when using FP8 quantization with Llama		2024-09-25 05:27:40 +00:00
..
__init__.py	Improve the handling of quantized weights (#2250)	2024-09-25 05:27:40 +00:00
custom_autotune.py	Refactor layers. (#1866)	2024-07-17 05:36:58 +00:00
exllama.py	Fix GPTQWeight import (#2020)	2024-09-24 03:34:15 +00:00
exllamav2.py	Do not initialize scratch space when there are no ExLlamaV2 layers (#2015)	2024-09-24 03:32:55 +00:00
quant_linear.py	Aligin the source code with main branch 2.0.4	2024-09-24 03:06:55 +00:00
quantize.py	Use symmetric quantization in the quantize subcommand (#2120)	2024-09-25 05:27:40 +00:00