Notation. Temperature and reference-policy log ratios [ftip-003M]
Notation. Temperature and reference-policy log ratios [ftip-003M]
Let \(\beta >0\) be the KL coefficient, \(\pi _{\mathrm {ref}}\) a fixed reference policy, and \(\pi _\theta \) a trainable policy with support contained in that of the reference on the responses under study. Define the reference-relative log likelihood
\[ \ell _\theta (x,y) =\log \frac {\pi _\theta (y\mid x)}{\pi _{\mathrm {ref}}(y\mid x)} \]and, for a comparison observation, the paired difference
\[ \Delta _\theta (x,y^+,y^-) =\ell _\theta (x,y^+)-\ell _\theta (x,y^-). \]