5 Commits

Author	SHA1	Message	Date
Daniël de Kok	775e5f4c64	MoE Marlin: support desc_act for groupsize != -1 (#2590) ... This change uses the updated Marlin MoE kernel from vLLM to support MoE with activation sorting and groups.	2024-10-25 09:12:03 +00:00
Daniël de Kok	288bcb0027	Add support for GPTQ-quantized MoE models using MoE Marlin (#2557) ... This change add support for MoE models that use GPTQ quantization. Currently only models with the following properties are supported: - No `desc_act` with tensor parallelism, unless `group_size=-1`. - No asymmetric quantization. - No AWQ.	2024-10-25 09:07:52 +00:00
Mohit Sharma	ff905aeff3	Update ROCM libs and improvements (#2579) ... * style * update torch * ix issues * fix clone * revert mkl * added custom PA * style * fix style * style * hide env vart * fix mixtral model * add skinny kernel and merge fixes * fixed style * fix issue for sliding window models * addressed review comments * fix import * improved error messag * updated default value * remove import * fix imports after rebase * float16 dep * improve dockerfile * cleaned dockerfile	2024-10-25 09:01:04 +00:00
Daniël de Kok	d4f995e718	Add DenseMoELayer and wire it up in Mixtral/Deepseek V2 (#2537) ... This replaces the custom layers in both models.	2024-10-25 09:01:04 +00:00
Daniël de Kok	29a93b78ba	Move to moe-kernels package and switch to common MoE layer (#2511) ... * Move to moe-kernels package and switch to common MoE layer This change introduces the new `moe-kernels` package: - Add `moe-kernels` as a dependency. - Introduce a `SparseMoELayer` module that can be used by MoE models. - Port over Mixtral and Deepseek. * Make `cargo check` pass * Update runner	2024-09-25 06:18:05 +00:00