Remark. How the divergence enters alignment objectives [ftip-006Y]
✍️source

The KL-regularized reward objective appears as equation (3) in Section 3 of [rafailov2023direct]. Behavior-policy ratios, clipped ratios, and KL penalties enter alignment objectives for different purposes.

A coefficient multiplying \(D_{\mathrm {KL}}\) declares an optimization tradeoff. It is neither a guarantee that every sampled ratio is small nor an independent evaluation of the resulting policy.

Remark. How the divergence enters alignment objectives [ftip-006Y]
✍️source

Reference. Direct preference optimization: Your language model is secretly a reward model [rafailov2023direct]

Reinforcement learning from human feedback and proximal policy optimization [ftip-0039]

Definition. Language-model RLHF pipeline [ouyang2022training, Section 3.1, Figure 2, and Section 3.5] [ftip-003A]

Remark. Reward modeling, alignment criteria, and policy optimization [ftip-003B]

Definition. Kullback--Leibler divergence [ftip-006X]

Remark. How the divergence enters alignment objectives [ftip-006Y]
✍️source

Definition. KL-constrained RLHF objective [rafailov2023direct, Section 3, equation (3)] [ftip-003C]

Definition. KL-shaped PPO reward [ouyang2022training, Section 3.5, equation (2)] [ftip-003D]

Definition. Generalized advantage estimator [schulman2016gae, Section 3, equations (10) and (16)] [ftip-003E]

Definition. PPO clipped surrogate [schulman2017proximal, Section 3, equation (7)] [ftip-003F]

Definition. PPO value-and-entropy objective [schulman2017proximal, Section 5, equation (9)] [ftip-003G]

Definition. PPO rollout and minibatch update [schulman2017proximal, Section 5] [ftip-003H]

Example. The PPO clipping cases for positive and negative advantage [ftip-003I]

Example. The InstructGPT-style RLHF pipeline [ftip-003J]

Remark. PPO is an optimizer, not an alignment definition [ftip-003K]