Lemma. Log-density ratio of the exact tilt [ftip-008N]
✍️source

For every \(y\in S_p\), the aligned law satisfies

\[ \log \frac {q_r(y)}{p(y)} =\frac {r(y)}{\beta }-\log Z_r. \]

Proof.

On \(S_p\), both \(p(y)\) and \(q_r(y)\) are positive. Dividing the formula of Definition [ftip-008K] by \(p(y)\) and taking logarithms gives the identity.

This is equation (B.1) followed by the logarithmic step in [paes2026theoretical, Appendix B.1, equations (B.1)--(B.2)].

Lemma. Log-density ratio of the exact tilt [ftip-008N]
✍️source

Proof.

Reference. Theoretical limits of language model alignment [paes2026theoretical]

Finite KL-regularized alignment [ftip-008F]

Remark. Exact optimization on a finite response set [ftip-008G]

Notation. Translating the alignment source notation [ftip-008H]

Definition. Finite KL-alignment instance [ftip-008I]

Definition. Exponential weight and partition function [ftip-008J]

Definition. Aligned law in finite-source notation [ftip-008K]

Definition. Cross-reward gain under an alignment reward [ftip-008L]

Definition. Jeffreys divergence [ftip-008M]

Lemma. Log-density ratio of the exact tilt [ftip-008N]
✍️source

Proof.

Theorem. Exact reward gain equals scaled Jeffreys divergence [ftip-008O]

Proof.

Remark. What the Jeffreys identity does and does not identify [ftip-008P]

Theorem. Cross-reward gain is a base-law covariance [ftip-008Q]

Proof.

Remark. The covariance is predictive only for a declared reward [ftip-008R]

Example. A two-response alignment identity [ftip-008S]

Corollary. Averaging the fixed-query identity over prompts [ftip-008T]

Proof.

Lemma. Additive reward shifts leave the aligned law unchanged [ftip-008U]

Proof.

Remark. A penalty coefficient is not a hard KL budget [ftip-008V]

Remark. Fixed-query identities and their assumptions [ftip-008W]

Theorem. Exact reward gain equals scaled Jeffreys divergence [ftip-008O]

Proof.

Definition. Aligned law in finite-source notation [ftip-008K]