text-generation-inference/server/text_generation_server/layers/marlin/__init__.py

from text_generation_server.layers.marlin.fp8 import GPTQMarlinFP8Linear
from text_generation_server.layers.marlin.gptq import (
    GPTQMarlinWeightsLoader,
    can_use_gptq_marlin,
    repack_gptq_for_marlin,
)
from text_generation_server.layers.marlin.marlin import MarlinWeightsLoader

__all__ = [
    "GPTQMarlinFP8Linear",
    "GPTQMarlinWeightsLoader",
    "MarlinWeightsLoader",
    "can_use_gptq_marlin",
    "repack_gptq_for_marlin",
]
Split up `layers.marlin` into several files (#2292) The marlin.py file was getting large, split it up. 2024-07-24 14:33:26 +00:00			`from text_generation_server.layers.marlin.fp8 import GPTQMarlinFP8Linear`
			`from text_generation_server.layers.marlin.gptq import (`
Handle GPTQ-Marlin loading in `GPTQMarlinWeightLoader` (#2300) The `GPTWeightLoader` was structured like this in pseudocode: if marlin: Set up tensors in a way that GPTQ-Marlin expects else: Set up tensors in a way that ExLlama/GPTQ/AWQ expect However, the GPT-Marlin implementation details should really be in the `marlin` module. So move the former part out to a separate `GPTQMarlinWeightsLoader`. 2024-07-31 11:08:41 +00:00			`GPTQMarlinWeightsLoader,`
Split up `layers.marlin` into several files (#2292) The marlin.py file was getting large, split it up. 2024-07-24 14:33:26 +00:00			`can_use_gptq_marlin,`
			`repack_gptq_for_marlin,`
			`)`
			`from text_generation_server.layers.marlin.marlin import MarlinWeightsLoader`

			`__all__ = [`
			`"GPTQMarlinFP8Linear",`
Handle GPTQ-Marlin loading in `GPTQMarlinWeightLoader` (#2300) The `GPTWeightLoader` was structured like this in pseudocode: if marlin: Set up tensors in a way that GPTQ-Marlin expects else: Set up tensors in a way that ExLlama/GPTQ/AWQ expect However, the GPT-Marlin implementation details should really be in the `marlin` module. So move the former part out to a separate `GPTQMarlinWeightsLoader`. 2024-07-31 11:08:41 +00:00			`"GPTQMarlinWeightsLoader",`
Split up `layers.marlin` into several files (#2292) The marlin.py file was getting large, split it up. 2024-07-24 14:33:26 +00:00			`"MarlinWeightsLoader",`
			`"can_use_gptq_marlin",`
			`"repack_gptq_for_marlin",`
			`]`