Remark. Why the entropy statistics are not interchangeable [ftip-0065]
Remark. Why the entropy statistics are not interchangeable [ftip-0065]
Shannon's finite entropy A mathematical theory of communication[shannon1948mathematical] can be aggregated over different random objects. Global and position-level entropy statistics use different laws. Selected-token surprisal is a sample statistic, while the other two are expectations under declared history or position laws.
A training intervention can raise one quantity while lowering another. Later claims about exploration or collapse must name the statistic, sampling law, horizon, and conditioning event.