mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-21 23:12:07 +00:00
ba291dad9f
* Improve the handling of quantized weights Handling of quantized weights was split between two mechanisms: - For quantized checkpoints, we used the new weight loader infrastructure. - For quantization while loading (EETQ, FP8, bitsandbytes) we instead relied on conditional in `get_linear`. Weight loaders support context managers to selectively load particular layers with different weight loaders, which is useful for models like Idefics2 AWQ, which uses a quantized text model, but unquantized vision and connector models. However, the context manager would be overrided by `get_linear`, which string-checks `quantizer`. Also, the context manager would not work with EETQ, FP8, and bitsandbytes. This change migrates all quantizers to the weight loader infrastructure. This has several benefits: - We can use context managers with all quantizers. - All the implementation details move down to the quantizer layers, `get_linear` does not need to know how to handle quantizer linear layers. - All quantizer weights are strongly typed, we don't pass around raw tensors. - We don't have to pass around the `quantizer` string everywhere. * Exclude non-MLP layers when using FP8 quantization with Llama |
||
|---|---|---|
| .. | ||
| merges | ||
| __init__.py | ||
| adapter.py | ||
| chunks.py | ||
| convert.py | ||
| dist.py | ||
| hub.py | ||
| import_utils.py | ||
| log.py | ||
| logits_process.py | ||
| peft.py | ||
| quantization.py | ||
| segments.py | ||
| sgmv.py | ||
| speculate.py | ||
| tokens.py | ||
| watermark.py | ||
| weights.py | ||