mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-24 08:22:07 +00:00
84ab88d843
* launcher: ensure correct detection of Gemma 3 head size * Support flashinfer for Gemma3 prefill Gemma3 uses bidirectional attention for images. Flashinfer supports custom masks. Hook up the mask with flashinfer, so that we do not have to use the slower SDPA implementation for prefills with images. * Update Gemma3 test outputs * Fixed unused import |
||
|---|---|---|
| .. | ||
| custom_kernels | ||
| exllama_kernels | ||
| exllamav2_kernels | ||
| tests | ||
| text_generation_server | ||
| .gitignore | ||
| bounds-from-nix.py | ||
| kernels.lock | ||
| Makefile | ||
| Makefile-awq | ||
| Makefile-eetq | ||
| Makefile-exllamav2 | ||
| Makefile-flash-att | ||
| Makefile-flash-att-v2 | ||
| Makefile-flashinfer | ||
| Makefile-lorax-punica | ||
| Makefile-selective-scan | ||
| Makefile-vllm | ||
| pyproject.toml | ||
| README.md | ||
| req.txt | ||
| requirements_cuda.txt | ||
| requirements_gen.txt | ||
| requirements_intel.txt | ||
| requirements_rocm.txt | ||
| uv.lock | ||
Text Generation Inference Python gRPC Server
A Python gRPC server for Text Generation Inference
Install
make install
Run
make run-dev