Daniël de Kok 653193a942 Improve support for GPUs with capability < 8 (#2575) ... * Improve support for GPUs with capability < 8 - For models that cannot use flashinfer, use flash-attn v1 + paged attention for models with a compute capability older than 8. - Disable prefix caching when using paged attention. - When using flash-attn v1, pass the key/value, rather than the cache, since v1 cannot use block tables. * nix: add flash-attn-v1 to the server environment * Move disabling prefix caching into the block of exceptions * Capability as `usize`s		2024-10-25 09:01:04 +00:00
..
client.nix	Preparing for release. (#2540)	2024-09-25 06:20:50 +00:00
crate-overrides.nix	nix: support Python tokenizer conversion in the router (#2515)	2024-09-25 06:15:35 +00:00
impure-shell.nix	Improve support for GPUs with capability < 8 (#2575)	2024-10-25 09:01:04 +00:00
server.nix	Improve support for GPUs with capability < 8 (#2575)	2024-10-25 09:01:04 +00:00