mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-21 23:12:07 +00:00
b2b5df0e94
This PR adds support for AMD Instinct MI210 & MI250 GPUs, with paged attention and FAv2 support. Remaining items to discuss, on top of possible others: * Should we have a `ghcr.io/huggingface/text-generation-inference:1.1.0+rocm` hosted image, or is it too early? * Should we set up a CI on MI210/MI250? I don't have access to the runners of TGI though. * Are we comfortable with those changes being directly in TGI, or do we need a fork? --------- Co-authored-by: Felix Marty <felix@hf.co> Co-authored-by: OlivierDehaene <olivier@huggingface.co> Co-authored-by: Your Name <you@example.com>
94 lines
3.1 KiB
Markdown
94 lines
3.1 KiB
Markdown
# Quick Tour
|
|
|
|
The easiest way of getting started is using the official Docker container. Install Docker following [their installation instructions](https://docs.docker.com/get-docker/).
|
|
|
|
Let's say you want to deploy [Falcon-7B Instruct](https://huggingface.co/tiiuae/falcon-7b-instruct) model with TGI. Here is an example on how to do that:
|
|
|
|
```bash
|
|
model=tiiuae/falcon-7b-instruct
|
|
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
|
|
|
|
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.1.1 --model-id $model
|
|
```
|
|
|
|
<Tip warning={true}>
|
|
|
|
To use GPUs, you need to install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) . We also recommend using NVIDIA drivers with CUDA version 11.8 or higher.
|
|
|
|
To use TGI on RoCm-enabled AMD GPUs (only MI210 and MI250 are tested), please use the image `ghcr.io/huggingface/text-generation-inference:1.1.1+rocm` instead. For details about the usage on RoCm, please refer to the [Supported Hardware section](./supported_models#supported-hardware) and [AMD documentation](https://rocm.docs.amd.com/en/latest/deploy/docker.html).
|
|
|
|
</Tip>
|
|
|
|
Once TGI is running, you can use the `generate` endpoint by doing requests. To learn more about how to query the endpoints, check the [Consuming TGI](./basic_tutorials/consuming_tgi) section, where we show examples with utility libraries and UIs. Below you can see a simple snippet to query the endpoint.
|
|
|
|
|
|
<inferencesnippet>
|
|
<python>
|
|
|
|
```python
|
|
import requests
|
|
|
|
headers = {
|
|
"Content-Type": "application/json",
|
|
}
|
|
|
|
data = {
|
|
'inputs': 'What is Deep Learning?',
|
|
'parameters': {
|
|
'max_new_tokens': 20,
|
|
},
|
|
}
|
|
|
|
response = requests.post('http://127.0.0.1:8080/generate', headers=headers, json=data)
|
|
print(response.json())
|
|
# {'generated_text': '\n\nDeep Learning is a subset of Machine Learning that is concerned with the development of algorithms that can'}
|
|
```
|
|
</python>
|
|
<js>
|
|
|
|
```js
|
|
async function query() {
|
|
const response = await fetch(
|
|
'http://127.0.0.1:8080/generate',
|
|
{
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json'},
|
|
body: JSON.stringify({
|
|
'inputs': 'What is Deep Learning?',
|
|
'parameters': {
|
|
'max_new_tokens': 20
|
|
}
|
|
})
|
|
}
|
|
);
|
|
}
|
|
|
|
query().then((response) => {
|
|
console.log(JSON.stringify(response));
|
|
});
|
|
/// {"generated_text":"\n\nDeep Learning is a subset of Machine Learning that is concerned with the development of algorithms that can"}
|
|
```
|
|
|
|
</js>
|
|
<curl>
|
|
|
|
```curl
|
|
curl 127.0.0.1:8080/generate \
|
|
-X POST \
|
|
-d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}' \
|
|
-H 'Content-Type: application/json'
|
|
```
|
|
|
|
</curl>
|
|
</inferencesnippet>
|
|
|
|
<Tip>
|
|
|
|
To see all possible deploy flags and options, you can use the `--help` flag. It's possible to configure the number of shards, quantization, generation parameters, and more.
|
|
|
|
```bash
|
|
docker run ghcr.io/huggingface/text-generation-inference:1.1.1 --help
|
|
```
|
|
|
|
</Tip>
|