Definition. Kullback--Leibler divergence [ftip-006X]
Definition. Kullback--Leibler divergence [ftip-006X]
For probability mass functions \(p\) and \(q\) on a finite or countable set \(\mathcal Y\), the Kullback--Leibler divergence of \(p\) relative to \(q\) is \[ D_{\mathrm {KL}}(p\Vert q) =\sum _{y\in \mathcal Y}p(y)\log \frac {p(y)}{q(y)}. \] We use \(0\log (0/q)=0\). If \(p(y)>0\) and \(q(y)=0\) for some \(y\), the value is \(+\infty \). On a countable set, the negative part of the displayed series is finite; a divergent positive part gives the value \(+\infty \). Since \(\mathcal V^*\) is countable, the definition applies to the response laws of Notation [ftip-002A].
The order of the arguments matters. In particular, finiteness of the displayed forward divergence requires \(p\) to be absolutely continuous with respect to \(q\).