From f9f23aaf2cab1de555cfbb943073fa70825eaebb Mon Sep 17 00:00:00 2001 From: moritzlaurer Date: Mon, 22 Apr 2024 15:47:49 +0200 Subject: [PATCH] add clarifying example for n-gram speculation --- docs/source/conceptual/speculation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/conceptual/speculation.md b/docs/source/conceptual/speculation.md index 071b7b68..f306c48a 100644 --- a/docs/source/conceptual/speculation.md +++ b/docs/source/conceptual/speculation.md @@ -36,7 +36,7 @@ In order to use medusa models in TGI, simply point to a medusa enabled model, an If you don't have a medusa model, or don't have the resource to fine-tune, you can try to use `n-gram`. -Ngram works by trying to find in the previous sequence existing tokens that match, and use those as speculation. +N-gram works by trying to find matching tokens in the previous sequence, and use those as speculation for generating new tokens. For example, if the tokens "np.mean" appear multiple times in the sequence, the model can speculate that the next continuation of the tokens "np." is probably also "mean". This is an extremely simple method, which works best for code, or highly repetitive text. This might not be beneficial, if the speculation misses too much.