text-generation-inference/docs/source
Daniël de Kok 2358c2bb54
Add basic FP8 KV cache support ()
* Add basic FP8 KV cache support

This change adds rudimentary FP8 KV cache support. The support is
enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so
uses this type for the KV cache. However support is still limited:

* Only the `fp8_e5m2` type is supported.
* The KV cache layout is the same as `float16`/`bfloat16` (HND).
* The FP8 KV cache is only supported for FlashInfer.
* Loading of scales is not yet supported.

* Fix Cargo.toml
2024-10-04 17:51:48 +02:00
..
basic_tutorials Revert "Unroll notify error into generate response" () 2024-10-03 17:56:40 -04:00
conceptual CI (2592): Allow LoRA adapter revision in server launcher () 2024-10-02 10:51:04 -04:00
reference Add basic FP8 KV cache support () 2024-10-04 17:51:48 +02:00
_toctree.yml Add links to Adyen blogpost () 2024-09-06 17:00:54 +02:00
architecture.md Update architecture.md () 2024-09-30 08:56:20 +02:00
index.md fix typos in docs and add small clarifications () 2024-04-22 12:15:48 -04:00
installation_amd.md New release 2.3.1 () 2024-10-03 14:43:49 +02:00
installation_gaudi.md MI300 compatibility () 2024-05-17 15:30:47 +02:00
installation_inferentia.md MI300 compatibility () 2024-05-17 15:30:47 +02:00
installation_intel.md New release 2.3.1 () 2024-10-03 14:43:49 +02:00
installation_nvidia.md New release 2.3.1 () 2024-10-03 14:43:49 +02:00
installation.md MI300 compatibility () 2024-05-17 15:30:47 +02:00
quicktour.md New release 2.3.1 () 2024-10-03 14:43:49 +02:00
supported_models.md Mllama flash version () 2024-10-02 11:22:13 +02:00
usage_statistics.md refactor usage stats () 2024-07-31 16:29:07 +02:00