mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-21 14:52:20 +00:00
289aa48554
* Move JSON grammar -> regex grammar conversion to the router
This change moves the JSON grammar -> regex grammar conversion to the
router by adding a dependency on the `outlines-core` Rust crate. In
contrast to the Python implementation, the conversions are not LRU-cached
since they seem to be fast enough:
simple schema time: [5.8293 µs 5.8307 µs 5.8320 µs]
change: [-13.166% -12.884% -12.641%] (p = 0.00 < 0.05)
Performance has improved.
complex schema time: [14.875 µs 14.881 µs 14.887 µs]
change: [-2.1637% -1.9914% -1.7852%] (p = 0.00 < 0.05)
Performance has improved.
Using the schemas from:
https://github.com/dottxt-ai/outlines-core/blob/main/benchmarks/bench_json_schema.py
|
||
|---|---|---|
| .. | ||
| test_flash_llama_grammar_tools_auto.json | ||
| test_flash_llama_grammar_tools_choice.json | ||
| test_flash_llama_grammar_tools_insufficient_information_stream.json | ||
| test_flash_llama_grammar_tools_insufficient_information.json | ||
| test_flash_llama_grammar_tools_sea_creatures_stream_function_object.json | ||
| test_flash_llama_grammar_tools_sea_creatures_stream_none.json | ||
| test_flash_llama_grammar_tools_sea_creatures_stream_required.json | ||
| test_flash_llama_grammar_tools_sea_creatures_stream.json | ||
| test_flash_llama_grammar_tools_stream.json | ||
| test_flash_llama_grammar_tools.json | ||