huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-07-23 00:00:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
git_v3.3.3
text-generation-inference/backends/neuron
History
David Corvoysier 79183d1647
Bump neuron SDK version (#3260) 
* chore(neuron): bump version to 0.2.0

* refactor(neuron): use named parameters in inputs helpers

This allows to hide the differences between the two backends in terms of
input parameters.

* refactor(neuron): remove obsolete code paths

* fix(neuron): use neuron_config whenever possible

* fix(neuron): use new cache import path

* fix(neuron): neuron config is not stored in config anymore

* fix(nxd): adapt model retrieval to new APIs

* fix(generator): emulate greedy in sampling parameters

When on-device sampling is enabled, we need to emulate the greedy
behaviour using top-k=1, top-p=1, temperature=1.

* test(neuron): update models and expectations

* feat(neuron): support on-device sampling

* fix(neuron): adapt entrypoint

* tests(neuron): remove obsolete models

* fix(neuron): adjust test expectations for llama on nxd
2025-06-10 17:56:25 +02:00
..
server Bump neuron SDK version (#3260) 2025-06-10 17:56:25 +02:00
tests Bump neuron SDK version (#3260) 2025-06-10 17:56:25 +02:00
Cargo.toml feat: add support for HF_HUB_USER_AGENT_ORIGIN to add user-agent Origin field in Hub requests. (#3061) 2025-03-04 16:43:50 +01:00
Makefile Update neuron backend (#3098) 2025-03-12 09:53:15 +01:00
README.md Add Neuron backend (#3033) 2025-02-24 09:10:05 +01:00
tgi_entry_point.py Bump neuron SDK version (#3260) 2025-06-10 17:56:25 +02:00
tgi-entrypoint.sh Bump neuron SDK version (#3260) 2025-06-10 17:56:25 +02:00

README.md

Text-generation-inference - Neuron backend for AWS Trainium and inferentia2

Description

This is the TGI backend for AWS Neuron Trainium and Inferentia family of chips.

This backend is composed of:

  • the AWS Neuron SDK,
  • the legacy v2 TGI launcher and router,
  • a neuron specific inference server for text-generation.

Usage

Please refer to the official documentation.

Build your own image

The simplest way to build TGI with the neuron backend is to use the provided Makefile:

$ make -C backends/neuron image

Alternatively, you can build the image directly from the top directory using a command similar to the one defined in the Makefile under the image target.