Definition. Token-position entropy [ftip-0063]
Definition. Token-position entropy [ftip-0063]
Let \(T_{\rm tok}\) be the random number of generated tokens. For a one-based token position \(t\) satisfying \(\Pr (T_{\rm tok}\geq t)>0\), condition on \(T_{\rm tok}\geq t\) and let \(\xi _t\) be the token. The token-position entropy is
\[ H_t^{\rm pos} =-\sum _{v\in \mathcal V} \Pr (\xi _t=v\mid T_{\rm tok}\geq t) \log \Pr (\xi _t=v\mid T_{\rm tok}\geq t). \]This is the entropy of a position marginal. It is generally different from the history-conditioned entropy averaged in Definition [ftip-0062].