huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-10 11:54:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Added RoPE and quantization

Browse Source
This commit is contained in:
Merve Noyan 2023-08-03 15:56:13 +03:00 committed by GitHub
parent 5bfc5a8f60
commit e20a5aeac5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 13 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
docs/source/basic_tutorials/preparing_model.md
View File

@ -1,2 +1,15 @@
# Preparing the Model
Text Generation Inference improves the model in several aspects.
## Quantization
TGI supports `bits-and-bytes` and `GPT-Q` quantization. To speed up inference with quantization, simply set `quantize` flag to `bitsandbytes` or `gptq` depending on the quantization technique you wish to use.
## RoPE Scaling
RoPE scaling can be used to increase the sequence length of the model during the inference time without necessarily fine-tuning it. To enable RoPE scaling, simply set `ROPE_SCALING` and `ROPE_FACTOR` variables. `ROPE_SCALING` can take the values `linear` or `dynamic`. If your model is not fine-tuned to a longer sequence length, use `dynamic`. `ROPE_FACTOR` is the ratio between the intended max sequence length and the model's original max sequence length.
## Safetensors Conversion