text-generation-inference/nix/docker.nix

{
  stdenv,
  dockerTools,
  cacert,
  text-generation-inference,
  stream ? false,
}:

let
  build = if stream then dockerTools.streamLayeredImage else dockerTools.buildLayeredImage;
in
build {
  name = "tgi-docker";
  tag = "latest";
  config = {
    EntryPoint = [ "${text-generation-inference}/bin/text-generation-inference" ];
    Env = [
      "HF_HOME=/data"
      "PORT=80"
    ];

  };
  contents = [
    cacert
    stdenv.cc
  ];
}
nix: experimental support for building a Docker container (#2470) * nix: experimental support for building a Docker image Run using something like: ``` docker run \ --device nvidia.com/gpu=all \ -it --rm -p 8080:80 \ -v $PWD/data:/data \ -v $PWD/tmp:/tmp \ tgi-docker:latest \ --model-id <model_id> ``` * Example of building the Docker image using Nix inside Docker * Stream to make the builder image smaller This avoids storing a Docker image tarball in the image. Instead, stream the layers while doing `docker run`. * Don't spam journalctl on Linux * Other dockerfile. --------- Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com> 2024-10-01 16:02:06 +00:00			`{`
We need gcc during runtime to enable triton to compile kernels. 2025-03-12 22:07:06 +00:00			`stdenv,`
nix: experimental support for building a Docker container (#2470) * nix: experimental support for building a Docker image Run using something like: ``` docker run \ --device nvidia.com/gpu=all \ -it --rm -p 8080:80 \ -v $PWD/data:/data \ -v $PWD/tmp:/tmp \ tgi-docker:latest \ --model-id <model_id> ``` * Example of building the Docker image using Nix inside Docker * Stream to make the builder image smaller This avoids storing a Docker image tarball in the image. Instead, stream the layers while doing `docker run`. * Don't spam journalctl on Linux * Other dockerfile. --------- Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com> 2024-10-01 16:02:06 +00:00			`dockerTools,`
			`cacert,`
			`text-generation-inference,`
			`stream ? false,`
			`}:`

			`let`
			`build = if stream then dockerTools.streamLayeredImage else dockerTools.buildLayeredImage;`
			`in`
			`build {`
			`name = "tgi-docker";`
			`tag = "latest";`
			`config = {`
			`EntryPoint = [ "${text-generation-inference}/bin/text-generation-inference" ];`
			`Env = [`
			`"HF_HOME=/data"`
			`"PORT=80"`
			`];`

			`};`
We need gcc during runtime to enable triton to compile kernels. 2025-03-12 22:07:06 +00:00			`contents = [`
			`cacert`
			`stdenv.cc`
			`];`
nix: experimental support for building a Docker container (#2470) * nix: experimental support for building a Docker image Run using something like: ``` docker run \ --device nvidia.com/gpu=all \ -it --rm -p 8080:80 \ -v $PWD/data:/data \ -v $PWD/tmp:/tmp \ tgi-docker:latest \ --model-id <model_id> ``` * Example of building the Docker image using Nix inside Docker * Stream to make the builder image smaller This avoids storing a Docker image tarball in the image. Instead, stream the layers while doing `docker run`. * Don't spam journalctl on Linux * Other dockerfile. --------- Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com> 2024-10-01 16:02:06 +00:00			`}`