Commit Graph

7 Commits

Author SHA1 Message Date
Daniël de Kok
3e17cb7866 nix: add punica-kernels (#2477)
Enables LoRA support.
2024-09-25 06:13:11 +00:00
Daniël de Kok
622c9c367a nix: build Torch against MKL and various other improvements (#2469)
Updates tgi-nix input:

- Move Torch closer to upstream by building against MKL.
- Remove compute capability 8.7 from Torch (Jetson).
- Sync nixpkgs cumpute capabilities with Torch (avoids
  compiling too mana capabilities for MAGMA).
- Use nixpkgs configuration passed through by `tgi-nix`.
2024-09-25 06:11:21 +00:00
Daniël de Kok
b7d1adc3e9 nix: add awq-inference-engine as server dependency (#2442) 2024-09-25 06:10:59 +00:00
Nicolas Patry
6654c2d11b Adding eetq to flake. (#2438) 2024-09-25 06:10:59 +00:00
Daniël de Kok
a5af557359 nix: add text-generation-benchmark to pure devshell (#2431)
nix: add text-generation-benchmark to pure devshell
2024-09-25 06:10:59 +00:00
Daniël de Kok
516392d790 nix: add pure server to flake, add both pure and impure devshells (#2430)
* nix: pure server and support both pure and impure devShells

* nix: remove unused poetry2nix input

It is not wired up and we now have a pure server.

* nix: add ipdb to impure devshell
2024-09-25 06:10:59 +00:00
Daniël de Kok
20ed7b598e nix: try to reduce the number of Rust rebuilds (#2424)
Try to reduce the number of router/launcher rebuilds by filtering
sources. In this way, recompiles should only be triggered by changes
in Cargo or Rust files.
2024-09-25 06:08:38 +00:00