mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-21 23:12:07 +00:00
55acb86f42
This WIP PR starts to add grammar support via outlines, currently this PR supports very simple regex grammars and does not optimize for precompiling or caching grammar fsm's. todo: - [X] add simple outlines guidance to `NextTokenChooser` - [X] update protos for grammar - [X] update generation params API - [X] constrain simple grammar - [ ] support parsing more complex grammar into fsm - [ ] support all outline support grammar types - [ ] explore optimizations to avoid recompiling grammars guided request ```bash curl -s 'http://localhost:3000/generate' \ --header 'Content-Type: application/json' \ --data-raw '{ "inputs": "make an email for david: \n", "parameters": { "max_new_tokens": 6, "grammar": "[\\w-]+@([\\w-]+\\.)+[\\w-]+" } }' | jq ``` response ```json { "generated_text": "david@example.com" } ``` unguided request ```bash curl -s 'http://localhost:3000/generate' \ --header 'Content-Type: application/json' \ --data '{ "inputs": "make an email for david: \n", "parameters": { "max_new_tokens": 6 } }' | jq ``` response ```json { "generated_text": " email = 'david" } ``` |
||
|---|---|---|
| .. | ||
| custom_modeling | ||
| __init__.py | ||
| bloom.py | ||
| cache_manager.py | ||
| causal_lm.py | ||
| flash_causal_lm.py | ||
| flash_llama.py | ||
| flash_mistral.py | ||
| flash_mixtral.py | ||
| flash_neox.py | ||
| flash_phi.py | ||
| flash_rw.py | ||
| flash_santacoder.py | ||
| galactica.py | ||
| globals.py | ||
| gpt_neox.py | ||
| idefics_causal_lm.py | ||
| idefics.py | ||
| mamba.py | ||
| model.py | ||
| mpt.py | ||
| opt.py | ||
| phi.py | ||
| rw.py | ||
| santacoder.py | ||
| seq2seq_lm.py | ||
| t5.py | ||
| types.py | ||