mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-21 14:52:20 +00:00
2358c2bb54
* Add basic FP8 KV cache support This change adds rudimentary FP8 KV cache support. The support is enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so uses this type for the KV cache. However support is still limited: * Only the `fp8_e5m2` type is supported. * The KV cache layout is the same as `float16`/`bfloat16` (HND). * The FP8 KV cache is only supported for FlashInfer. * Loading of scales is not yet supported. * Fix Cargo.toml |
||
|---|---|---|
| .. | ||
| basic_tutorials | ||
| conceptual | ||
| reference | ||
| _toctree.yml | ||
| architecture.md | ||
| index.md | ||
| installation_amd.md | ||
| installation_gaudi.md | ||
| installation_inferentia.md | ||
| installation_intel.md | ||
| installation_nvidia.md | ||
| installation.md | ||
| quicktour.md | ||
| supported_models.md | ||
| usage_statistics.md | ||