huggingface/text-generation-inference
5
0
Fork 0
mirror of https://github.com/huggingface/text-generation-inference.git synced 2025-05-03 07:52:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

Skip {% generation %} and {% endgeneration %} template handling (#3204)

Browse Source 
* Add `.DS_Store` file to `.gitignore`

* Skip `{% generation %}` and `{% endgeneration %}`

Custom syntax within the chat template for the Phi4 Reasoning models
e.g. https://huggingface.co/microsoft/Phi-4-reasoning-plus, which is
AFAIK not handled natively yet, so skipping for now

* Update explanation on `{% generation %}` and `{% endgeneration %}` removal

* Revert "Add `.DS_Store` file to `.gitignore`"

This reverts commit d64d6d2f7f.
This commit is contained in:
Alvaro Bartolome 2025-05-01 12:13:17 +02:00 committed by GitHub
parent e7329fec18
commit 40dfce644a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
router/src/infer/chat_template.rs
View File

@ -45,6 +45,12 @@ impl ChatTemplate {
// It uses python notation to reverse lists, which do not exist in minijinja
// so we're using the reverse filter instead.
let mutated_template = mutated_template.replace("[::-1]", "|reverse");
// TODO: replace with a better solution
// Hack to remove the {% generation %} and {% endgeneration %} statements from
// the Jinja2 chat templates if there, since those are only using for assistant
// masking during training, and should be ignored during inference
let mutated_template = mutated_template.replace("{% generation %}", "");
let mutated_template = mutated_template.replace("{% endgeneration %}", "");
let template_str = mutated_template.into_boxed_str();
env.add_function("raise_exception", raise_exception);