Theorem. Cross-reward gain is a base-law covariance [ftip-008Q]
✍️source

For a finite KL-alignment instance,

\[ G_p(r;s) =\operatorname {Cov}_p\left (s,\frac {a_r}{Z_r}\right ). \]

Proof.

The aligned expectation can be written under the reference law as \(\mathbb E_{q_r}[s]=\mathbb E_p[s a_r/Z_r]\). Also \(\mathbb E_p[a_r/Z_r]=1\). Substituting these two identities into Definition [ftip-008L] gives the covariance defined in Notation [ftip-008H].

This is the finite form of Theorem 2, equation (3.4), in Theoretical limits of language model alignment[paes2026theoretical]. Its proof is in Appendix B.1, equations (B.5)--(B.7).

Theorem. Cross-reward gain is a base-law covariance [ftip-008Q]
✍️source

Proof.

Reference. Theoretical limits of language model alignment [paes2026theoretical]

Finite KL-regularized alignment [ftip-008F]

Remark. Exact optimization on a finite response set [ftip-008G]

Notation. Translating the alignment source notation [ftip-008H]

Definition. Finite KL-alignment instance [ftip-008I]

Definition. Exponential weight and partition function [ftip-008J]

Definition. Aligned law in finite-source notation [ftip-008K]

Definition. Cross-reward gain under an alignment reward [ftip-008L]

Definition. Jeffreys divergence [ftip-008M]

Lemma. Log-density ratio of the exact tilt [ftip-008N]

Proof.

Theorem. Exact reward gain equals scaled Jeffreys divergence [ftip-008O]

Proof.

Remark. What the Jeffreys identity does and does not identify [ftip-008P]

Theorem. Cross-reward gain is a base-law covariance [ftip-008Q]
✍️source

Proof.

Remark. The covariance is predictive only for a declared reward [ftip-008R]

Example. A two-response alignment identity [ftip-008S]

Corollary. Averaging the fixed-query identity over prompts [ftip-008T]

Proof.

Lemma. Additive reward shifts leave the aligned law unchanged [ftip-008U]

Proof.

Remark. A penalty coefficient is not a hard KL budget [ftip-008V]

Remark. Fixed-query identities and their assumptions [ftip-008W]

Remark. A penalty coefficient is not a hard KL budget [ftip-008V]

Remark. The covariance is predictive only for a declared reward [ftip-008R]

Definition. Cross-reward gain under an alignment reward [ftip-008L]

Notation. Translating the alignment source notation [ftip-008H]