huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-21 23:12:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
2d4aa25b9c
text-generation-inference/backends/llamacpp
History
Adrien Gallouët 2d4aa25b9c
Make --model-gguf optional 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2025-03-05 11:08:17 +00:00
..
src Make --model-gguf optional 2025-03-05 11:08:17 +00:00
build.rs [Backend] Add Llamacpp backend (#2975) 2025-02-14 13:40:57 +01:00
Cargo.toml Make --model-gguf optional 2025-03-05 11:08:17 +00:00
make-gguf.sh Make --model-gguf optional 2025-03-05 11:08:17 +00:00
README.md [Backend] Add Llamacpp backend (#2975) 2025-02-14 13:40:57 +01:00
requirements.txt Make --model-gguf optional 2025-03-05 11:08:17 +00:00

README.md

Llamacpp backend

If all your dependencies are installed at the system level, running cargo build should be sufficient. However, if you want to experiment with different versions of llama.cpp, some additional setup is required.

Install llama.cpp

LLAMACPP_PREFIX=$(pwd)/llama.cpp.out

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build \
    -DCMAKE_INSTALL_PREFIX="$LLAMACPP_PREFIX" \
    -DLLAMA_BUILD_COMMON=OFF \
    -DLLAMA_BUILD_TESTS=OFF \
    -DLLAMA_BUILD_EXAMPLES=OFF \
    -DLLAMA_BUILD_SERVER=OFF
cmake --build build --config Release -j
cmake --install build

Build TGI

PKG_CONFIG_PATH="$LLAMACPP_PREFIX/lib/pkgconfig" cargo build