text-generation-inference

huggingface/text-generation-inference

Fork 0

mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-05-01 23:12:09 +00:00

Commit Graph

Author	SHA1	Message	Date
Nicolas Patry	8b295aa498	Upgrade minor rust version (Fixes rust build compilation cache) (#2617 ) * Upgrade minor rust version (Fixes rust build compilation cache) * Black	2024-10-08 09:42:50 +02:00
Florian Zimmermeister	0da4df4b96	Fix FP8 KV-cache condition (#2611 ) Update kv_cache.py	2024-10-07 09:34:19 +02:00
Daniël de Kok	2358c2bb54	Add basic FP8 KV cache support (#2603 ) * Add basic FP8 KV cache support This change adds rudimentary FP8 KV cache support. The support is enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so uses this type for the KV cache. However support is still limited: * Only the `fp8_e5m2` type is supported. * The KV cache layout is the same as `float16`/`bfloat16` (HND). * The FP8 KV cache is only supported for FlashInfer. * Loading of scales is not yet supported. * Fix Cargo.toml	2024-10-04 17:51:48 +02:00

Author

SHA1

Message

Date

Nicolas Patry

8b295aa498

Upgrade minor rust version (Fixes rust build compilation cache) (#2617 )

* Upgrade minor rust version (Fixes rust build compilation cache)

* Black

2024-10-08 09:42:50 +02:00

Florian Zimmermeister

0da4df4b96

Fix FP8 KV-cache condition (#2611 )

Update kv_cache.py

2024-10-07 09:34:19 +02:00

Daniël de Kok

2358c2bb54

Add basic FP8 KV cache support (#2603 )

* Add basic FP8 KV cache support

This change adds rudimentary FP8 KV cache support. The support is
enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so
uses this type for the KV cache. However support is still limited:

* Only the `fp8_e5m2` type is supported.
* The KV cache layout is the same as `float16`/`bfloat16` (HND).
* The FP8 KV cache is only supported for FlashInfer.
* Loading of scales is not yet supported.

* Fix Cargo.toml

2024-10-04 17:51:48 +02:00

3 Commits