huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-04-24 00:12:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
5bd2ab6583
text-generation-inference/server
History
Nicolas Patry 5bd2ab6583
feat(server): Support for env value for GPTQ_BITS and GPTQ_GROUPSIZE. (#580) 
# What does this PR do?

Some models are already converted, and do not have those values in the
file, this enables users to use them with less friction.

Went for pure env based because adding flags would end up (imo) very
tedious to maintain. There's a lot of sanitation to do: those flags
would be errors if not used in conjuction with `--quantize gptq`.
Then the flags need to exist in the launcher and the server passing them
all throughout all function calls.

This PR is intended as an easy escape hatch, not the defacto method to
use gptq in TGI.

Fixes #500
2023-07-12 10:00:02 +02:00
..
custom_kernels feat(server): Rework model loading (#344) 2023-06-08 14:51:52 +02:00
tests fix(server): harden the weights choice to save on disk. (#561) 2023-07-07 14:50:12 +02:00
text_generation_server feat(server): Support for env value for GPTQ_BITS and GPTQ_GROUPSIZE. (#580) 2023-07-12 10:00:02 +02:00
.gitignore feat(clients): Python client (#103) 2023-03-07 18:52:22 +01:00
Makefile fix: Update server/Makefile to include Makefile-vllm (#520) 2023-07-04 09:39:25 +02:00
Makefile-flash-att feat(server): use latest flash attention commit (#543) 2023-07-04 20:23:55 +02:00
Makefile-vllm feat(server): add paged attention to flash models (#516) 2023-06-30 19:09:59 +02:00
poetry.lock feat(server): use latest flash attention commit (#543) 2023-07-04 20:23:55 +02:00
pyproject.toml v0.9.1 (#558) 2023-07-06 16:05:42 +02:00
README.md feat(router): refactor API and add openAPI schemas (#53) 2023-02-03 12:43:37 +01:00
requirements.txt feat(server): use latest flash attention commit (#543) 2023-07-04 20:23:55 +02:00

README.md

Text Generation Inference Python gRPC Server

A Python gRPC server for Text Generation Inference

Install

make install

Run

make run-dev