6 Commits

Author	SHA1	Message	Date
Ubuntu	ae308f88ec	Some fixes.	2023-06-14 09:42:55 +02:00
Ubuntu	a0a194c391	Functionning quantization script.	2023-06-14 09:42:55 +02:00
Ubuntu	5a72715344	Adding quantization scripts.	2023-06-14 09:42:55 +02:00
Nicolas Patry	da8ebf16fe	Typo.	2023-06-14 09:42:55 +02:00
Ubuntu	92f85c964d	Removing dead code.	2023-06-14 09:42:55 +02:00
Ubuntu	9a12941bef	[WIP] Inference support for GPTQ (llama at least) ... Let's start discussing implementation. - Need to expose the quantization scripts (either included here or add doc on how to use https://github.com/qwopqwop200/GPTQ-for-LLaMa) - Make sure GPTQ works for multiple models (priority to Falcon). Currently it means that every place we use `get_{tensor|sharded}` to check for quantization. My idea is to reintegrate as much as possible into `utils/layer.py` by expanding `load_multi` to be a bit more generic. This might require some thinking, but ultimately the `qweight,qzeros,scales,g_idx` should be in a single place, and independant of bias presence.	2023-06-14 09:42:55 +02:00