huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-10 20:04:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Added trust-remote-code

Browse Source
This commit is contained in:
Merve Noyan 2023-08-23 16:27:55 +03:00 committed by GitHub
parent cf0453182e
commit e16ecaf0c9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 6 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
docs/source/basic_tutorials/non_core_models.md
View File

@ -2,20 +2,14 @@
TGI supports various LLM architectures (see full list [here](./supported_models)). If you wish to serve a model that is not one of the supported models, TGI will fallback to transformers implementation of that model. This means you will be unable to use some of the features introduced by TGI, such as tensor-parallel sharding or flash attention. However, you can still get many benefits of TGI, such as continuous batching or streaming outputs.
They can be loaded by:
```python
from transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM
AutoModelForCausalLM.from_pretrained(<model>, device_map="auto")``
#or
AutoModelForSeq2SeqLM.from_pretrained(<model>, device_map="auto")
```
You can serve these models using Docker like below 👇
```bash
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:latest --model-id gpt2
```
If the model you wish to serve is not a transformers model, but weights and implementation is included in the repository, you can still serve the model by passing `--trust-remote-code` flag to `docker run` command like below 👇
```bash
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:latest --model-id <CUSTOM_MODEL_ID> --trust-remote-code
```