huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-21 23:12:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
2dd680b799
text-generation-inference/server/text_generation_server/models/custom_modeling
History
Daniël de Kok 2dd680b799 Improve the handling of quantized weights (#2250) 
* Improve the handling of quantized weights

Handling of quantized weights was split between two mechanisms:

- For quantized checkpoints, we used the new weight loader
  infrastructure.
- For quantization while loading (EETQ, FP8, bitsandbytes) we
  instead relied on conditional in `get_linear`.

Weight loaders support context managers to selectively load
particular layers with different weight loaders, which is useful
for models like Idefics2 AWQ, which uses a quantized text model,
but unquantized vision and connector models. However, the context
manager would be overrided by `get_linear`, which string-checks
`quantizer`. Also, the context manager would not work with
EETQ, FP8, and bitsandbytes.

This change migrates all quantizers to the weight loader infrastructure.
This has several benefits:

- We can use context managers with all quantizers.
- All the implementation details move down to the quantizer layers,
  `get_linear` does not need to know how to handle quantizer linear
  layers.
- All quantizer weights are strongly typed, we don't pass around
  raw tensors.
- We don't have to pass around the `quantizer` string everywhere.

* Exclude non-MLP layers when using FP8 quantization with Llama
2024-09-25 05:27:40 +00:00
..
__init__.py feat(server): flash santacoder (#153) 2023-04-03 19:06:42 +02:00
bloom_modeling.py Consistently take prefix in model constructors (#2191) 2024-09-25 05:21:34 +00:00
clip.py Consistently take prefix in model constructors (#2191) 2024-09-25 05:21:34 +00:00
flash_cohere_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_dbrx_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_gemma2_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_gemma_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_gpt2_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_llama_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_mistral_modeling.py Consistently take prefix in model constructors (#2191) 2024-09-25 05:21:34 +00:00
flash_mixtral_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_neox_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_pali_gemma_modeling.py Enable multiple LoRa adapters (#2010) 2024-09-24 03:55:04 +00:00
flash_phi_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_qwen2_modeling.py Consistently take prefix in model constructors (#2191) 2024-09-25 05:21:34 +00:00
flash_rw_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_santacoder_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
flash_starcoder2_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
idefics2.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
idefics_config.py chore: add pre-commit (#1569) 2024-04-24 15:32:02 +03:00
idefics_image_processing.py chore: formatting 2024-04-18 16:26:00 +03:00
idefics_modeling.py reenable xpu for tgi (#1939) 2024-07-17 05:36:58 +00:00
idefics_perceiver.py Refactor layers. (#1866) 2024-07-17 05:36:58 +00:00
idefics_processing.py chore: add pre-commit (#1569) 2024-04-24 15:32:02 +03:00
idefics_vision.py Refactor layers. (#1866) 2024-07-17 05:36:58 +00:00
llava_next.py Refactor dead code - Removing all flash_xxx.py files. (#2166) 2024-09-25 05:20:28 +00:00
mamba_modeling.py Refactor layers. (#1866) 2024-07-17 05:36:58 +00:00
mpt_modeling.py Improve the handling of quantized weights (#2250) 2024-09-25 05:27:40 +00:00
neox_modeling.py Consistently take prefix in model constructors (#2191) 2024-09-25 05:21:34 +00:00
opt_modeling.py fix dbrx & opt model prefix bug (#2201) 2024-09-25 05:21:34 +00:00
phi_modeling.py Consistently take prefix in model constructors (#2191) 2024-09-25 05:21:34 +00:00
siglip.py Removing some unused code. (#1915) 2024-07-17 05:36:58 +00:00
t5_modeling.py Refactor layers. (#1866) 2024-07-17 05:36:58 +00:00
vlm.py Pali gemma modeling (#1895) 2024-07-17 05:36:58 +00:00