Daniël de Kok
e618ce3ada
Fix: make moe_kernels imports conditional
...
`moe-kernels` is an optional dependency, so make sure that we run
without installing this package.
Fixes #2621 .
2024-10-08 11:05:28 +00:00
Daniël de Kok
64142489b6
Add support for fused MoE Marlin for AWQ ( #2616 )
...
* Add support for fused MoE Marlin for AWQ
This uses the updated MoE Marlin kernels from vLLM.
* Add integration test for AWQ MoE
2024-10-08 11:56:41 +02:00
Daniël de Kok
1c84a30fe6
MoE Marlin: support desc_act for groupsize != -1 ( #2590 )
...
This change uses the updated Marlin MoE kernel from vLLM to support
MoE with activation sorting and groups.
2024-09-30 19:40:25 +02:00
Daniël de Kok
90a1d04a2f
Add support for GPTQ-quantized MoE models using MoE Marlin ( #2557 )
...
This change add support for MoE models that use GPTQ quantization.
Currently only models with the following properties are supported:
- No `desc_act` with tensor parallelism, unless `group_size=-1`.
- No asymmetric quantization.
- No AWQ.
2024-09-30 11:14:32 +02:00
