mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-19 13:52:07 +00:00
289aa48554
* Move JSON grammar -> regex grammar conversion to the router
This change moves the JSON grammar -> regex grammar conversion to the
router by adding a dependency on the `outlines-core` Rust crate. In
contrast to the Python implementation, the conversions are not LRU-cached
since they seem to be fast enough:
simple schema time: [5.8293 µs 5.8307 µs 5.8320 µs]
change: [-13.166% -12.884% -12.641%] (p = 0.00 < 0.05)
Performance has improved.
complex schema time: [14.875 µs 14.881 µs 14.887 µs]
change: [-2.1637% -1.9914% -1.7852%] (p = 0.00 < 0.05)
Performance has improved.
Using the schemas from:
https://github.com/dottxt-ai/outlines-core/blob/main/benchmarks/bench_json_schema.py
|
||
|---|---|---|
| .. | ||
| custom_kernels | ||
| exllama_kernels | ||
| exllamav2_kernels | ||
| tests | ||
| text_generation_server | ||
| .gitignore | ||
| Makefile | ||
| Makefile-awq | ||
| Makefile-eetq | ||
| Makefile-exllamav2 | ||
| Makefile-flash-att | ||
| Makefile-flash-att-v2 | ||
| Makefile-flashinfer | ||
| Makefile-lorax-punica | ||
| Makefile-selective-scan | ||
| Makefile-vllm | ||
| poetry.lock | ||
| pyproject.toml | ||
| README.md | ||
| requirements_cuda.txt | ||
| requirements_intel.txt | ||
| requirements_rocm.txt | ||
Text Generation Inference Python gRPC Server
A Python gRPC server for Text Generation Inference
Install
make install
Run
make run-dev