Definition. Optimal KL-regularized policy [rafailov2023direct, Section 4, equation (4)] [ftip-003O]
Definition. Optimal KL-regularized policy [rafailov2023direct, Section 4, equation (4)] [ftip-003O]
For a reward \(r(x,y)\), reference policy \(\pi _{\mathrm {ref}}\), and \(\beta >0\), the optimizer of the per-prompt KL-regularized reward objective has the form
\[ \pi _r(y\mid x) =\frac {1}{Z_r(x)}\, \pi _{\mathrm {ref}}(y\mid x) \exp \left (\frac {r(x,y)}{\beta }\right ), \]where
\[ Z_r(x)=\sum _y\pi _{\mathrm {ref}}(y\mid x) \exp \left (\frac {r(x,y)}{\beta }\right ) \]normalizes the policy on the response space. The identity assumes that the normalizer is finite and uses the same reference policy as the underlying objective.