drbh
|
55acb86f42
|
Outlines guided generation (#1539)
This WIP PR starts to add grammar support via outlines, currently this
PR supports very simple regex grammars and does not optimize for
precompiling or caching grammar fsm's.
todo:
- [X] add simple outlines guidance to `NextTokenChooser`
- [X] update protos for grammar
- [X] update generation params API
- [X] constrain simple grammar
- [ ] support parsing more complex grammar into fsm
- [ ] support all outline support grammar types
- [ ] explore optimizations to avoid recompiling grammars
guided request
```bash
curl -s 'http://localhost:3000/generate' \
--header 'Content-Type: application/json' \
--data-raw '{
"inputs": "make an email for david: \n",
"parameters": {
"max_new_tokens": 6,
"grammar": "[\\w-]+@([\\w-]+\\.)+[\\w-]+"
}
}' | jq
```
response
```json
{
"generated_text": "david@example.com"
}
```
unguided request
```bash
curl -s 'http://localhost:3000/generate' \
--header 'Content-Type: application/json' \
--data '{
"inputs": "make an email for david: \n",
"parameters": {
"max_new_tokens": 6
}
}' | jq
```
response
```json
{
"generated_text": " email = 'david"
}
```
|
2024-04-24 14:57:37 +03:00 |
|
OlivierDehaene
|
0c207f71ed
|
feat: experimental support for cuda graphs (#1428)
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
|
2024-04-24 13:15:45 +03:00 |
|
OlivierDehaene
|
c974437ba7
|
fix: fix gpt-q params loading
|
2024-04-19 12:12:50 +03:00 |
|
OlivierDehaene
|
f9b58ac7a1
|
feat: add quant to mixtral (#1337)
|
2024-04-18 16:32:50 +03:00 |
|
OlivierDehaene
|
79f268f95a
|
chore: formatting
|
2024-04-18 16:26:00 +03:00 |
|
OlivierDehaene
|
9aef902982
|
feat: mixtral (#1328)
|
2024-04-18 12:39:52 +00:00 |
|
Nicolas Patry
|
a7f52f3812
|
Speculative (#1308)
|
2024-04-18 12:39:39 +00:00 |
|
OlivierDehaene
|
3b56d7669b
|
feat: add mistral model (#1071)
|
2023-09-28 09:55:47 +02:00 |
|