mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-24 00:12:08 +00:00
628d6a13da
Mostly straightforward, changes to existing code: * Wrap quantizer parameters in a small wrapper to avoid passing around untyped tuples and needing to repack them as a dict. * Move scratch space computation to warmup, because we need the maximum input sequence length to avoid allocating huge scratch buffers that OOM. |
||
|---|---|---|
| .. | ||
| layers | ||
| models | ||
| pb | ||
| utils | ||
| __init__.py | ||
| cache.py | ||
| cli.py | ||
| interceptor.py | ||
| server.py | ||
| tracing.py | ||