huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-09-09 19:34:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

Clarify installation steps

Browse Source
This commit is contained in:
lewtun 2023-02-03 12:24:20 +01:00 committed by OlivierDehaene
parent 20c3c5940c
commit bc8596390c
1 changed files with 21 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
README.md
View File

@ -93,7 +93,7 @@ curl 127.0.0.1:8080/generate_stream \
-H 'Content-Type: application/json' -H 'Content-Type: application/json'
``` ```
To use GPUs, you will need to install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). **Note:** To use GPUs, you need to install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).
### API documentation ### API documentation
@ -102,14 +102,31 @@ The Swagger UI is also available at: [https://huggingface.github.io/text-generat
### Local install ### Local install
You can also opt to install `text-generation-inference` locally. You will need to have cargo and Python installed on your You can also opt to install `text-generation-inference` locally.
machine
First [install Rust](https://rustup.rs/) and create a Python virtual environment with at least
Python 3.9, e.g. using `conda`:
```shell
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
conda create -n text-generation-inference python=3.9
conda activate text-generation-inference
```
Then run:
```shell ```shell
BUILD_EXTENSIONS=True make install # Install repository and HF/transformer fork with CUDA kernels BUILD_EXTENSIONS=True make install # Install repository and HF/transformer fork with CUDA kernels
make run-bloom-560m make run-bloom-560m
``` ```
**Note:** on some machines, you may also need the OpenSSL libraries. On Linux machines, run:
```shell
sudo apt-get install libssl-dev
```
### CUDA Kernels ### CUDA Kernels
The custom CUDA kernels are only tested on NVIDIA A100s. If you have any installation or runtime issues, you can remove The custom CUDA kernels are only tested on NVIDIA A100s. If you have any installation or runtime issues, you can remove