mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-20 14:22:08 +00:00
232e8d5227
Adds support for AMD Instinct MI300 in TGI.
Most changes are:
* Support PyTorch TunableOp to pick the GEMM/GEMV kernels for decoding
https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/cuda/tunable.
TunableOp is disabled by default, and can be enabled with
`PYTORCH_TUNABLEOP_ENABLED=1`.
* Update ROCm dockerfile to PyTorch 2.3 (actually patched with changes
from https://github.com/pytorch/pytorch/pull/124362)
* Support SILU & Linear custom kernels contributed by AMD
* Update vLLM paged attention to https://github.com/fxmarty/rocm-vllm/,
branching out of a much more recent commit
3489ce7936
* Support FA2 Triton kernel as recommended by AMD. Can be used by
specifying `ROCM_USE_FLASH_ATTN_V2_TRITON=1`.
* Update dockerfile to ROCm 6.1
By default, TunableOp tuning results are saved in `/data` (e.g.
`/data/tunableop_meta-llama-Llama-2-70b-chat-hf_tp1_rank0.csv`) in order
to avoid to have to rerun the tuning at each `docker run`.
Example:
```
Validator,PT_VERSION,2.3.0
Validator,ROCM_VERSION,6.1.0.0-82-5fabb4c
Validator,HIPBLASLT_VERSION,0.7.0-1549b021
Validator,GCN_ARCH_NAME,gfx942:sramecc+:xnack-
Validator,ROCBLAS_VERSION,4.1.0-cefa4a9b-dirty
GemmTunableOp_Half_TN,tn_8192_7_28672,Gemm_Rocblas_45475,0.132098
GemmTunableOp_Half_TN,tn_10240_4_8192,Gemm_Rocblas_45546,0.0484431
GemmTunableOp_Half_TN,tn_32000_6_8192,Default,0.149546
GemmTunableOp_Half_TN,tn_32000_3_8192,Gemm_Rocblas_45520,0.147119
GemmTunableOp_Half_TN,tn_8192_3_28672,Gemm_Rocblas_45475,0.132645
GemmTunableOp_Half_TN,tn_10240_3_8192,Gemm_Rocblas_45546,0.0482971
GemmTunableOp_Half_TN,tn_57344_5_8192,Gemm_Rocblas_45520,0.255694
GemmTunableOp_Half_TN,tn_10240_7_8192,Gemm_Rocblas_45517,0.0482522
GemmTunableOp_Half_TN,tn_8192_3_8192,Gemm_Rocblas_45546,0.0444671
GemmTunableOp_Half_TN,tn_8192_5_8192,Gemm_Rocblas_45546,0.0445834
GemmTunableOp_Half_TN,tn_57344_7_8192,Gemm_Rocblas_45520,0.25622
GemmTunableOp_Half_TN,tn_8192_2_28672,Gemm_Rocblas_45475,0.132122
GemmTunableOp_Half_TN,tn_8192_4_8192,Gemm_Rocblas_45517,0.0453191
GemmTunableOp_Half_TN,tn_10240_5_8192,Gemm_Rocblas_45517,0.0482514
GemmTunableOp_Half_TN,tn_8192_5_28672,Gemm_Rocblas_45542,0.133914
GemmTunableOp_Half_TN,tn_8192_2_8192,Gemm_Rocblas_45517,0.0446516
GemmTunableOp_Half_TN,tn_8192_1_28672,Gemm_Hipblaslt_TN_10814,0.131953
GemmTunableOp_Half_TN,tn_10240_2_8192,Gemm_Rocblas_45546,0.0481043
GemmTunableOp_Half_TN,tn_32000_4_8192,Gemm_Rocblas_45520,0.147497
GemmTunableOp_Half_TN,tn_8192_6_28672,Gemm_Rocblas_45529,0.134895
GemmTunableOp_Half_TN,tn_57344_2_8192,Gemm_Rocblas_45520,0.254716
GemmTunableOp_Half_TN,tn_57344_4_8192,Gemm_Rocblas_45520,0.255731
GemmTunableOp_Half_TN,tn_10240_6_8192,Gemm_Rocblas_45517,0.0484816
GemmTunableOp_Half_TN,tn_57344_3_8192,Gemm_Rocblas_45520,0.254701
GemmTunableOp_Half_TN,tn_8192_4_28672,Gemm_Rocblas_45475,0.132159
GemmTunableOp_Half_TN,tn_32000_2_8192,Default,0.147524
GemmTunableOp_Half_TN,tn_32000_5_8192,Default,0.147074
GemmTunableOp_Half_TN,tn_8192_6_8192,Gemm_Rocblas_45546,0.0454045
GemmTunableOp_Half_TN,tn_57344_6_8192,Gemm_Rocblas_45520,0.255582
GemmTunableOp_Half_TN,tn_32000_7_8192,Default,0.146705
GemmTunableOp_Half_TN,tn_8192_7_8192,Gemm_Rocblas_45546,0.0445489
```
---------
Co-authored-by: Mohit Sharma <mohit21sharma.ms@gmail.com>
84 lines
2.3 KiB
Markdown
84 lines
2.3 KiB
Markdown
# Installation from source
|
|
|
|
<Tip warning={true}>
|
|
|
|
Installing TGI from source is not the recommended usage. We strongly recommend to use TGI through Docker, check the [Quick Tour](./quicktour), [Installation for Nvidia GPUs](./installation_nvidia) and [Installation for AMD GPUs](./installation_amd) to learn how to use TGI with Docker.
|
|
|
|
</Tip>
|
|
|
|
## Install CLI
|
|
|
|
You can use TGI command-line interface (CLI) to download weights, serve and quantize models, or get information on serving parameters.
|
|
|
|
To install the CLI, you need to first clone the TGI repository and then run `make`.
|
|
|
|
```bash
|
|
git clone https://github.com/huggingface/text-generation-inference.git && cd text-generation-inference
|
|
make install
|
|
```
|
|
|
|
If you would like to serve models with custom kernels, run
|
|
|
|
```bash
|
|
BUILD_EXTENSIONS=True make install
|
|
```
|
|
|
|
## Local Installation from Source
|
|
|
|
Before you start, you will need to setup your environment, and install Text Generation Inference. Text Generation Inference is tested on **Python 3.9+**.
|
|
|
|
Text Generation Inference is available on pypi, conda and GitHub.
|
|
|
|
To install and launch locally, first [install Rust](https://rustup.rs/) and create a Python virtual environment with at least
|
|
Python 3.9, e.g. using conda:
|
|
|
|
```bash
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
|
|
|
conda create -n text-generation-inference python=3.9
|
|
conda activate text-generation-inference
|
|
```
|
|
|
|
You may also need to install Protoc.
|
|
|
|
On Linux:
|
|
|
|
```bash
|
|
PROTOC_ZIP=protoc-21.12-linux-x86_64.zip
|
|
curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v21.12/$PROTOC_ZIP
|
|
sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc
|
|
sudo unzip -o $PROTOC_ZIP -d /usr/local 'include/*'
|
|
rm -f $PROTOC_ZIP
|
|
```
|
|
|
|
On MacOS, using Homebrew:
|
|
|
|
```bash
|
|
brew install protobuf
|
|
```
|
|
|
|
Then run to install Text Generation Inference:
|
|
|
|
```bash
|
|
git clone https://github.com/huggingface/text-generation-inference.git && cd text-generation-inference
|
|
BUILD_EXTENSIONS=True make install
|
|
```
|
|
|
|
<Tip warning={true}>
|
|
|
|
On some machines, you may also need the OpenSSL libraries and gcc. On Linux machines, run:
|
|
|
|
```bash
|
|
sudo apt-get install libssl-dev gcc -y
|
|
```
|
|
|
|
</Tip>
|
|
|
|
Once installation is done, simply run:
|
|
|
|
```bash
|
|
make run-falcon-7b-instruct
|
|
```
|
|
|
|
This will serve Falcon 7B Instruct model from the port 8080, which we can query.
|