mirror of
https://github.com/huggingface/text-generation-inference.git
synced 2025-04-24 00:12:08 +00:00
628d6a13da
Mostly straightforward, changes to existing code: * Wrap quantizer parameters in a small wrapper to avoid passing around untyped tuples and needing to repack them as a dict. * Move scratch space computation to warmup, because we need the maximum input sequence length to avoid allocating huge scratch buffers that OOM. |
||
|---|---|---|
| .. | ||
| images | ||
| models | ||
| conftest.py | ||
| poetry.lock | ||
| pyproject.toml | ||
| pytest.ini | ||
| requirements.txt | ||