huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-12 04:44:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Up.

Browse Source
This commit is contained in:
Vaibhav Srivastav 2024-08-13 19:58:21 +02:00
parent 4abd7d3971
commit 7007394766
2 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/basic_tutorials/consuming_tgi.md
View File

@ -125,7 +125,7 @@ Assume you are serving your model on port 8080, we will query through [Inference
import gradio as gr import gradio as gr
from huggingface_hub import InferenceClient from huggingface_hub import InferenceClient
client = InferenceClient(model="http://127.0.0.1:8080") client = InferenceClient(base_url="http://127.0.0.1:8080")
def inference(message, history): def inference(message, history):
partial_message = "" partial_message = ""

4
docs/source/conceptual/streaming.md
View File

@ -48,7 +48,7 @@ To stream tokens with `InferenceClient`, simply pass `stream=True` and iterate o
```python ```python
from huggingface_hub import InferenceClient from huggingface_hub import InferenceClient
client = InferenceClient("http://127.0.0.1:8080") client = InferenceClient(base_url="http://127.0.0.1:8080")
output = client.chat.completions.create( output = client.chat.completions.create(
messages=[ messages=[
{"role": "system", "content": "You are a helpful assistant."}, {"role": "system", "content": "You are a helpful assistant."},
@ -78,7 +78,7 @@ The `huggingface_hub` library also comes with an `AsyncInferenceClient` in case
```python ```python
from huggingface_hub import AsyncInferenceClient from huggingface_hub import AsyncInferenceClient
client = AsyncInferenceClient("http://127.0.0.1:8080") client = AsyncInferenceClient(base_url="http://127.0.0.1:8080")
async def main(): async def main():
stream = await client.chat.completions.create( stream = await client.chat.completions.create(
messages=[{"role": "user", "content": "Say this is a test"}], messages=[{"role": "user", "content": "Say this is a test"}],