huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-21 23:12:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,074 Commits 228 Branches 59 Tags 47 MiB
adjust-where-request-max-tokens-is-defaulted
Commit Graph

3 Commits

Author SHA1 Message Date
Nicolas Patry
8b295aa498
Upgrade minor rust version (Fixes rust build compilation cache) (#2617) 
* Upgrade minor rust version (Fixes rust build compilation cache)

* Black
 2024-10-08 09:42:50 +02:00
Florian Zimmermeister
0da4df4b96
Fix FP8 KV-cache condition (#2611) 
Update kv_cache.py
 2024-10-07 09:34:19 +02:00
Daniël de Kok
2358c2bb54
Add basic FP8 KV cache support (#2603) 
* Add basic FP8 KV cache support

This change adds rudimentary FP8 KV cache support. The support is
enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so
uses this type for the KV cache. However support is still limited:

* Only the `fp8_e5m2` type is supported.
* The KV cache layout is the same as `float16`/`bfloat16` (HND).
* The FP8 KV cache is only supported for FlashInfer.
* Loading of scales is not yet supported.

* Fix Cargo.toml
 2024-10-04 17:51:48 +02:00