Nicolas Patry
cb747b33da
Add deepseekv3 ( #2968 )
...
* Add fp8 support moe models
add deepseekv3
format codfe'
update dockerfile
update doc
* Small modifications.
* Moe kernels 0.8.1
* Upgrade to 0.8.1
* Fixing moe import.
* Black.
* Apply suggestions from code review
Co-authored-by: Mohit Sharma <mohit21sharma.ms@gmail.com>
* Fixing Mixtral + Nits.
* Put link to ref.
* Fix other call locations.
* Scoring func `softmax` is the only one that works.
---------
Co-authored-by: Mohit Sharma <mohit21sharma.ms@gmail.com>
2025-01-30 16:40:25 +01:00
Daniël de Kok
64142489b6
Add support for fused MoE Marlin for AWQ ( #2616 )
...
* Add support for fused MoE Marlin for AWQ
This uses the updated MoE Marlin kernels from vLLM.
* Add integration test for AWQ MoE
2024-10-08 11:56:41 +02:00
Daniël de Kok
1c84a30fe6
MoE Marlin: support desc_act for groupsize != -1 ( #2590 )
...
This change uses the updated Marlin MoE kernel from vLLM to support
MoE with activation sorting and groups.
2024-09-30 19:40:25 +02:00
Daniël de Kok
90a1d04a2f
Add support for GPTQ-quantized MoE models using MoE Marlin ( #2557 )
...
This change add support for MoE models that use GPTQ quantization.
Currently only models with the following properties are supported:
- No `desc_act` with tensor parallelism, unless `group_size=-1`.
- No asymmetric quantization.
- No AWQ.
2024-09-30 11:14:32 +02:00
