Remark. Exact optimization on a finite response set [ftip-008G]
✍️source

Theorems 1 and 2 of the v1 preprint [paes2026theoretical] concern an exactly optimal KL-regularized policy and a fixed query. On a finite response set, their expectations and normalizers are ordinary finite sums.

These exact identities do not establish a comparison among best-of-\(N\), PPO, and GRPO, or a general guarantee for proxy rewards and reward ensembles. Those questions require assumptions beyond exact optimization of one reward.

Remark. Exact optimization on a finite response set [ftip-008G]
✍️source

Reference. Theoretical limits of language model alignment [paes2026theoretical]

Finite KL-regularized alignment [ftip-008F]

Remark. Exact optimization on a finite response set [ftip-008G]
✍️source

Notation. Translating the alignment source notation [ftip-008H]

Definition. Finite KL-alignment instance [ftip-008I]

Definition. Exponential weight and partition function [ftip-008J]

Definition. Aligned law in finite-source notation [ftip-008K]

Definition. Cross-reward gain under an alignment reward [ftip-008L]

Definition. Jeffreys divergence [ftip-008M]

Lemma. Log-density ratio of the exact tilt [ftip-008N]

Proof.

Theorem. Exact reward gain equals scaled Jeffreys divergence [ftip-008O]

Proof.

Remark. What the Jeffreys identity does and does not identify [ftip-008P]

Theorem. Cross-reward gain is a base-law covariance [ftip-008Q]

Proof.

Remark. The covariance is predictive only for a declared reward [ftip-008R]

Example. A two-response alignment identity [ftip-008S]

Corollary. Averaging the fixed-query identity over prompts [ftip-008T]

Proof.

Lemma. Additive reward shifts leave the aligned law unchanged [ftip-008U]

Proof.

Remark. A penalty coefficient is not a hard KL budget [ftip-008V]

Remark. Fixed-query identities and their assumptions [ftip-008W]