OlivierDehaene
|
7fbfbb0dc5
|
feat: Add token streaming using ServerSideEvents support (#36)
Add token streaming using ServerSideEvents (SSE).
The signature of the SSE events is:
```rust
struct Details {
finish_reason: String,
generated_tokens: u32,
seed: Option<u64>,
}
struct StreamResponse {
token: Token,
generated_text: Option<String>,
details: Option<Details>,
}
struct ErrorResponse {
error: String,
}
```
|
2023-01-31 11:49:43 +01:00 |
|
OlivierDehaene
|
cd298bc5e5
|
feat: Support sampling seeding (#37)
Co-authored-by: Yannic Kilcher <yk@users.noreply.github.com>
|
2023-01-30 15:36:16 +01:00 |
|
OlivierDehaene
|
1539d3cbbe
|
feat(router): Remove second lock from batcher hot path (#27)
@njhill
|
2023-01-26 16:29:13 +01:00 |
|
Nick Hill
|
60472f9d2b
|
feat(router): Add const parameters to validation logic (#15)
I noticed some opportunity to collapse some of the logic, in case you
are interested.
|
2023-01-03 10:41:22 +01:00 |
|
OlivierDehaene
|
32a253063d
|
feat: Return logprobs (#8)
|
2022-12-15 17:03:56 +01:00 |
|
OlivierDehaene
|
718096f695
|
feat: Support stop sequences (#7)
|
2022-12-12 18:25:22 +01:00 |
|
OlivierDehaene
|
427d7cc444
|
feat(server): Support AutoModelForSeq2SeqLM
|
2022-11-04 18:03:04 +01:00 |
|
OlivierDehaene
|
3cf6368c77
|
feat(server): Support all AutoModelForCausalLM on a best effort basis
|
2022-10-28 19:24:00 +02:00 |
|
OlivierDehaene
|
c837893370
|
feat(router): Add max_waiting_tokens
|
2022-10-21 16:40:05 +02:00 |
|
Olivier Dehaene
|
f16f2f5ae1
|
v0.1.0
|
2022-10-20 19:14:44 +02:00 |
|
Olivier Dehaene
|
5e5d8766a2
|
feat: Improve error handling
|
2022-10-17 14:59:00 +02:00 |
|
Olivier Dehaene
|
4c693e6524
|
Refactored gRPC interface
Added validation logic
|
2022-10-11 16:50:54 +02:00 |
|
Olivier Dehaene
|
fa9a088467
|
Add load testing
|
2022-10-11 10:36:51 +02:00 |
|
Olivier Dehaene
|
295831a481
|
Init
|
2022-10-08 12:30:12 +02:00 |
|