huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-21 23:12:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove References to torch compile mode in readme (#236)

Browse Source
This commit is contained in:
Thanaji Rao Thakkalapelli 2024-10-17 14:07:51 -07:00 committed by GitHub
parent 8ae5d4c7d6
commit 21c13ff3a6
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
README.md
View File

@ -58,19 +58,7 @@ To use [🤗 text-generation-inference](https://github.com/huggingface/text-gene
--max-total-tokens 2048
```
ii. On 1 Gaudi card using PyTorch eager mode with torch compile:
```bash
model=meta-llama/Llama-2-7b-hf
hf_token=YOUR_ACCESS_TOKEN
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run -p 8080:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all \
-e PT_HPU_LAZY_MODE=0 -e OMPI_MCA_btl_vader_single_copy_mechanism=none \
-e HF_TOKEN=$hf_token --cap-add=sys_nice --ipc=host \
ghcr.io/huggingface/tgi-gaudi:2.0.5 --model-id $model --max-input-tokens 1024 --max-total-tokens 2048
```
iii. On 8 Gaudi cards:
ii. On 8 Gaudi cards:
```bash
model=meta-llama/Llama-2-70b-hf
hf_token=YOUR_ACCESS_TOKEN