OlivierDehaene
|
e74bd41e0f
|
feat(server): add paged attention to flash models (#516)
Closes #478
|
2023-06-30 19:09:59 +02:00 |
|
OlivierDehaene
|
bd3a9d8e85
|
fix(router): add timeout on flume sends (#488)
|
2023-06-23 14:58:28 +02:00 |
|
OlivierDehaene
|
218c9adaa5
|
feat: decrease IPC proto size (#367)
Closes #307 #308
|
2023-05-24 19:19:57 +02:00 |
|
Nicolas Patry
|
db2b4e0754
|
feat(router): new healthcheck that skips the queue (#244)
Co-authored-by: OlivierDehaene <23298448+OlivierDehaene@users.noreply.github.com>
Co-authored-by: OlivierDehaene <olivier@huggingface.co>
|
2023-04-26 20:23:54 +02:00 |
|
OlivierDehaene
|
ebc74d5666
|
feat(router): use number of tokens in batch as input for dynamic batching (#226)
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
|
2023-04-24 17:59:00 +02:00 |
|
OlivierDehaene
|
709d8936f6
|
feat(router): drop requests when client closes the channel (#202)
|
2023-04-20 11:07:40 +02:00 |
|
OlivierDehaene
|
9987960062
|
feat(router): make router input validation optional (#164)
|
2023-04-09 20:22:27 +02:00 |
|
OlivierDehaene
|
7dec65a244
|
fix(router): use buckets for metrics histograms (#163)
|
2023-04-09 20:13:28 +02:00 |
|
OlivierDehaene
|
f000068944
|
feat(server): clear cache on error (#143)
|
2023-03-28 11:29:35 +02:00 |
|
OlivierDehaene
|
b49dbf2d88
|
fix(server): use server tokenizer as gt (#128)
|
2023-03-16 12:12:26 +01:00 |
|
OlivierDehaene
|
cbd36aa4d1
|
fix(server): revert gpt-neox optims (#123)
|
2023-03-13 22:57:08 +01:00 |
|
OlivierDehaene
|
55bd4fed7d
|
feat(router): add best_of parameter (#117)
|
2023-03-09 15:30:54 +01:00 |
|
OlivierDehaene
|
3fef90d50f
|
feat(clients): Python client (#103)
|
2023-03-07 18:52:22 +01:00 |
|
OlivierDehaene
|
0ac184ce77
|
feat(server): add special token bool (#85)
|
2023-02-24 15:55:57 +01:00 |
|
OlivierDehaene
|
439fcaf810
|
feat(router): add prometheus metrics scrape endpoint (#71)
|
2023-02-16 17:18:53 +01:00 |
|
OlivierDehaene
|
9af454142a
|
feat: add distributed tracing (#62)
|
2023-02-13 13:02:45 +01:00 |
|
OlivierDehaene
|
20c3c5940c
|
feat(router): refactor API and add openAPI schemas (#53)
|
2023-02-03 12:43:37 +01:00 |
|
OlivierDehaene
|
7b870e1e18
|
feat(router): use background task to manage request queue (#52)
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
|
2023-02-02 14:59:27 +01:00 |
|
OlivierDehaene
|
017a2a8c2f
|
feat: Add token streaming using ServerSideEvents support (#41)
|
2023-01-31 17:04:00 +01:00 |
|
OlivierDehaene
|
4f9ac67cfa
|
Revert "feat: Add token streaming using ServerSideEvents support" (#40)
Reverts huggingface/text-generation-inference#36
|
2023-01-31 14:21:51 +01:00 |
|
OlivierDehaene
|
7fbfbb0dc5
|
feat: Add token streaming using ServerSideEvents support (#36)
Add token streaming using ServerSideEvents (SSE).
The signature of the SSE events is:
```rust
struct Details {
finish_reason: String,
generated_tokens: u32,
seed: Option<u64>,
}
struct StreamResponse {
token: Token,
generated_text: Option<String>,
details: Option<Details>,
}
struct ErrorResponse {
error: String,
}
```
|
2023-01-31 11:49:43 +01:00 |
|
Olivier Dehaene
|
fa9a088467
|
Add load testing
|
2022-10-11 10:36:51 +02:00 |
|
Olivier Dehaene
|
295831a481
|
Init
|
2022-10-08 12:30:12 +02:00 |
|