Daniël de Kok 5b6b74e21d Improve support for GPUs with capability < 8 (#2575) ... * Improve support for GPUs with capability < 8 - For models that cannot use flashinfer, use flash-attn v1 + paged attention for models with a compute capability older than 8. - Disable prefix caching when using paged attention. - When using flash-attn v1, pass the key/value, rather than the cache, since v1 cannot use block tables. * nix: add flash-attn-v1 to the server environment * Move disabling prefix caching into the block of exceptions * Capability as `usize`s		2024-09-27 16:19:42 +02:00
..
__init__.py	Improve support for GPUs with capability < 8 (#2575)	2024-09-27 16:19:42 +02:00
common.py	Lots of improvements (Still 2 allocators) (#2449)	2024-08-29 16:29:01 +02:00
cuda.py	Improve support for GPUs with capability < 8 (#2575)	2024-09-27 16:19:42 +02:00
flash_attn_triton.py	feat: add ruff and resolve issue (#2262)	2024-07-26 10:29:09 -04:00
flashinfer.py	More tensor cores. (#2558)	2024-09-24 23:57:26 +02:00
ipex.py	Improve support for GPUs with capability < 8 (#2575)	2024-09-27 16:19:42 +02:00
rocm.py	Improve support for GPUs with capability < 8 (#2575)	2024-09-27 16:19:42 +02:00