mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-05-09 19:32:06 +00:00
ba291dad9f
* Improve the handling of quantized weights Handling of quantized weights was split between two mechanisms: - For quantized checkpoints, we used the new weight loader infrastructure. - For quantization while loading (EETQ, FP8, bitsandbytes) we instead relied on conditional in `get_linear`. Weight loaders support context managers to selectively load particular layers with different weight loaders, which is useful for models like Idefics2 AWQ, which uses a quantized text model, but unquantized vision and connector models. However, the context manager would be overrided by `get_linear`, which string-checks `quantizer`. Also, the context manager would not work with EETQ, FP8, and bitsandbytes. This change migrates all quantizers to the weight loader infrastructure. This has several benefits: - We can use context managers with all quantizers. - All the implementation details move down to the quantizer layers, `get_linear` does not need to know how to handle quantizer linear layers. - All quantizer weights are strongly typed, we don't pass around raw tensors. - We don't have to pass around the `quantizer` string everywhere. * Exclude non-MLP layers when using FP8 quantization with Llama |
||
|---|---|---|
| .. | ||
| test_bloom_560m | ||
| test_bloom_560m_sharded | ||
| test_chat_llama | ||
| test_completion_prompts | ||
| test_flash_awq | ||
| test_flash_awq_sharded | ||
| test_flash_falcon | ||
| test_flash_gemma | ||
| test_flash_gemma_gptq | ||
| test_flash_gpt2 | ||
| test_flash_grammar_llama | ||
| test_flash_llama | ||
| test_flash_llama_exl2 | ||
| test_flash_llama_gptq | ||
| test_flash_llama_marlin | ||
| test_flash_llama_marlin_24 | ||
| test_flash_medusa | ||
| test_flash_mistral | ||
| test_flash_neox | ||
| test_flash_neox_sharded | ||
| test_flash_pali_gemma | ||
| test_flash_phi | ||
| test_flash_qwen2 | ||
| test_flash_santacoder | ||
| test_flash_starcoder | ||
| test_flash_starcoder2 | ||
| test_flash_starcoder_gptq | ||
| test_grammar_llama | ||
| test_grammar_response_format_llama | ||
| test_idefics | ||
| test_idefics2 | ||
| test_llava_next | ||
| test_lora_mistral | ||
| test_mamba | ||
| test_mpt | ||
| test_mt0_base | ||
| test_neox | ||
| test_neox_sharded | ||
| test_server_gptq_quantized | ||
| test_t5_sharded | ||
| test_tools_llama | ||