huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-11 04:14:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: prefer message api naming

Browse Source
This commit is contained in:
drbh 2024-01-23 17:20:28 +00:00
parent 8f4dc804c5
commit e4fa84ba26
1 changed files with 6 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
docs/source/openai_compatible.md
View File

@ -1,10 +1,12 @@
# Open AI Compatible # Messages API
Text Generation Inference (TGI) is compatible with OpenAI's API. This means you can use OpenAI's client libraries to interact with TGI. Here are some examples of how to do this. _Messages API is compatible to OpenAI Chat Completion API_
Text Generation Inference (TGI) now supports the Message API which is fully compatible with the OpenAI Chat Completion API. This means you can use OpenAI's client libraries to interact with TGI's Messages API. Below are some examples of how to utilize this compatibility.
## Making a Request ## Making a Request
You can make a request to TGI using `curl`. Here's an example: You can make a request to TGI's Messages API using `curl`. Here's an example:
```bash ```bash
curl localhost:3000/v1/chat/completions \ curl localhost:3000/v1/chat/completions \
@ -87,7 +89,7 @@ TGI can be deployed on various cloud providers for scalable and robust text gene
Amazon SageMaker allows two routes: `/invocations` and `/ping` (or `/health`) for health checks. By default, we map `/generate` to `/invocations`. However, SageMaker does not allow requests to any other routes. Amazon SageMaker allows two routes: `/invocations` and `/ping` (or `/health`) for health checks. By default, we map `/generate` to `/invocations`. However, SageMaker does not allow requests to any other routes.
To provide the new feature of chat completion API, we have introduced an environment variable `OAI_ENABLED`. If `OAI_ENABLED=true`, the `chat_completions` method is used when `/invocations` is called, otherwise it defaults to `generate`. This allows users to opt in for the OAI format. To provide the new feature of Messages API, we have introduced an environment variable `OAI_ENABLED`. If `OAI_ENABLED=true`, the `chat_completions` method is used when `/invocations` is called, otherwise it defaults to `generate`. This allows users to opt in for the OAI format.
Here's an example of running the router with `OAI_ENABLED` set to `true`: Here's an example of running the router with `OAI_ENABLED` set to `true`: