Ubuntu 9a12941bef [WIP] Inference support for GPTQ (llama at least) ... Let's start discussing implementation. - Need to expose the quantization scripts (either included here or add doc on how to use https://github.com/qwopqwop200/GPTQ-for-LLaMa) - Make sure GPTQ works for multiple models (priority to Falcon). Currently it means that every place we use `get_{tensor|sharded}` to check for quantization. My idea is to reintegrate as much as possible into `utils/layer.py` by expanding `load_multi` to be a bit more generic. This might require some thinking, but ultimately the `qweight,qzeros,scales,g_idx` should be in a single place, and independant of bias presence.		2023-06-14 09:42:55 +02:00
..
custom_autotune.py	[WIP] Inference support for GPTQ (llama at least)	2023-06-14 09:42:55 +02:00
quant_linear.py	[WIP] Inference support for GPTQ (llama at least)	2023-06-14 09:42:55 +02:00