Example. The InstructGPT-style RLHF pipeline [ftip-003J]
✍️source

One prompt passes successively through supervised tuning, comparison-based reward modeling, and PPO in the InstructGPT-style RLHF pipeline.

$\text{[math]}$

For the displayed illustrative prompt, suppose the comparison-trained reward model scores the preferred response by $0.8$. If the candidate policy assigns it twice the probability assigned by the reference policy and the declared KL coefficient is $\beta =0.1$, its shaped scalar is \[ 0.8-\beta \log \frac {\pi (y_B\mid x)} {\pi _{\rm ref}(y_B\mid x)} =0.8-0.1\log 2\approx 0.731. \]

The stage ordering is grounded in [ouyang2022training, Sections 3.1--3.5]; the numbers are a schematic calculation rather than reported InstructGPT hyperparameters. The diagram records the interface used in that pipeline. Other RLHF systems may order or replace the stages differently.

Example. The InstructGPT-style RLHF pipeline [ftip-003J]
✍️source

Reference. Training language models to follow instructions with human feedback [ouyang2022training]

Reinforcement learning from human feedback and proximal policy optimization [ftip-0039]

Definition. Language-model RLHF pipeline [ouyang2022training, Section 3.1, Figure 2, and Section 3.5] [ftip-003A]

Remark. Reward modeling, alignment criteria, and policy optimization [ftip-003B]

Definition. Kullback--Leibler divergence [ftip-006X]

Remark. How the divergence enters alignment objectives [ftip-006Y]

Definition. KL-constrained RLHF objective [rafailov2023direct, Section 3, equation (3)] [ftip-003C]

Definition. KL-shaped PPO reward [ouyang2022training, Section 3.5, equation (2)] [ftip-003D]

Definition. Generalized advantage estimator [schulman2016gae, Section 3, equations (10) and (16)] [ftip-003E]

Definition. PPO clipped surrogate [schulman2017proximal, Section 3, equation (7)] [ftip-003F]

Definition. PPO value-and-entropy objective [schulman2017proximal, Section 5, equation (9)] [ftip-003G]

Definition. PPO rollout and minibatch update [schulman2017proximal, Section 5] [ftip-003H]

Example. The PPO clipping cases for positive and negative advantage [ftip-003I]

Example. The InstructGPT-style RLHF pipeline [ftip-003J]
✍️source

Remark. PPO is an optimizer, not an alignment definition [ftip-003K]