From cf0453182e0a02855181393f2a23169dd92d9330 Mon Sep 17 00:00:00 2001 From: Merve Noyan Date: Tue, 22 Aug 2023 23:45:56 +0300 Subject: [PATCH] Restructure --- docs/source/basic_tutorials/non_core_models.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/basic_tutorials/non_core_models.md b/docs/source/basic_tutorials/non_core_models.md index e10d6718..5ceb79bc 100644 --- a/docs/source/basic_tutorials/non_core_models.md +++ b/docs/source/basic_tutorials/non_core_models.md @@ -1,6 +1,8 @@ # Non-core Model Serving -TGI supports various LLM architectures (see full list [here](./supported_models)). If you wish to serve a model that is not one of the supported models, TGI will fallback to transformers implementation of that model. They can be loaded by: +TGI supports various LLM architectures (see full list [here](./supported_models)). If you wish to serve a model that is not one of the supported models, TGI will fallback to transformers implementation of that model. This means you will be unable to use some of the features introduced by TGI, such as tensor-parallel sharding or flash attention. However, you can still get many benefits of TGI, such as continuous batching or streaming outputs. + +They can be loaded by: ```python from transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM @@ -12,8 +14,6 @@ AutoModelForCausalLM.from_pretrained(, device_map="auto")`` AutoModelForSeq2SeqLM.from_pretrained(, device_map="auto") ``` -This means you will be unable to use some of the features introduced by TGI, such as tensor-parallel sharding or flash attention. However, you can still get many benefits of TGI, such as continuous batching or streaming outputs. - You can serve these models using Docker like below 👇 ```bash