Definition. KL-constrained RLHF objective [rafailov2023direct, Section 3, equation (3)] [ftip-003C]
✍️source

Let \(\mu \) be a prompt law, \(r(x,y)\) a scalar reward, and \(\pi _{\mathrm {ref}}\) a fixed reference policy. For \(\beta >0\), the KL-regularized RLHF objective, using forward KL, is

\[ \max _{\pi } \mathbb E_{x\sim \mu ,\,y\sim \pi (\cdot \mid x)}[r(x,y)] -\beta \, \mathbb E_{x\sim \mu } \left [D_{\mathrm {KL}}\left ( \pi (\cdot \mid x)\Vert \pi _{\mathrm {ref}}(\cdot \mid x) \right )\right ]. \]

The prompt law, reward, reference policy, and coefficient are part of the objective. The forward KL requires the optimized policy to be absolutely continuous with respect to the reference wherever the objective is finite.

Definition. KL-constrained RLHF objective [rafailov2023direct, Section 3, equation (3)] [ftip-003C]
✍️source

Reference. Direct preference optimization: Your language model is secretly a reward model [rafailov2023direct]

Reinforcement learning from human feedback and proximal policy optimization [ftip-0039]

Definition. Language-model RLHF pipeline [ouyang2022training, Section 3.1, Figure 2, and Section 3.5] [ftip-003A]

Remark. Reward modeling, alignment criteria, and policy optimization [ftip-003B]

Definition. Kullback--Leibler divergence [ftip-006X]

Remark. How the divergence enters alignment objectives [ftip-006Y]

Definition. KL-constrained RLHF objective [rafailov2023direct, Section 3, equation (3)] [ftip-003C]
✍️source

Definition. KL-shaped PPO reward [ouyang2022training, Section 3.5, equation (2)] [ftip-003D]

Definition. Generalized advantage estimator [schulman2016gae, Section 3, equations (10) and (16)] [ftip-003E]

Definition. PPO clipped surrogate [schulman2017proximal, Section 3, equation (7)] [ftip-003F]

Definition. PPO value-and-entropy objective [schulman2017proximal, Section 5, equation (9)] [ftip-003G]

Definition. PPO rollout and minibatch update [schulman2017proximal, Section 5] [ftip-003H]

Example. The PPO clipping cases for positive and negative advantage [ftip-003I]

Example. The InstructGPT-style RLHF pipeline [ftip-003J]

Remark. PPO is an optimizer, not an alignment definition [ftip-003K]