mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-23 07:52:06 +00:00
9a12941bef
Let's start discussing implementation. - Need to expose the quantization scripts (either included here or add doc on how to use https://github.com/qwopqwop200/GPTQ-for-LLaMa) - Make sure GPTQ works for multiple models (priority to Falcon). Currently it means that every place we use `get_{tensor|sharded}` to check for quantization. My idea is to reintegrate as much as possible into `utils/layer.py` by expanding `load_multi` to be a bit more generic. This might require some thinking, but ultimately the `qweight,qzeros,scales,g_idx` should be in a single place, and independant of bias presence. |
||
|---|---|---|
| .. | ||
| gptq | ||
| __init__.py | ||
| convert.py | ||
| dist.py | ||
| hub.py | ||
| layers.py | ||
| logits_process.py | ||
| tokens.py | ||
| watermark.py | ||
| weights.py | ||