Definition. Cross-reward gain under an alignment reward [ftip-008L]
Definition. Cross-reward gain under an alignment reward [ftip-008L]
The cross-reward gain from aligning with \(r\) and evaluating with \(s\) is
\[ G_p(r;s) =\mathbb E_{y\sim q_r}[s(y)] -\mathbb E_{y\sim p}[s(y)]. \]The semicolon records two roles. Its left argument changes the response law; its right argument scores both laws. Thus \(G_p(r;s)\) is not an independent evaluation unless \(s\) has separately been declared to serve that role.
This is the finite notation for \(\Delta (r,r')\) in Theorem 2, equation (3.4), of Theoretical limits of language model alignment[paes2026theoretical].