Example. The PPO clipping cases for positive and negative advantage [ftip-003I]
✍️source

Positive and negative advantages select different branches of the PPO clipped minimum.

$\text{[math]}$

With $\epsilon =0.2$, write $\bar r=\operatorname {clip}(r,0.8,1.2)$ and $L(r,A)=\min (rA,\bar rA)$. For positive advantage and $r=1.4$, the gain is capped at $2.4$. For negative advantage and $r=0.6$, the clipped term $-1.6$ is smaller than $-1.2$, so the objective retains the penalty.

The pointwise surrogate comes from [schulman2017proximal, Section 3]. The two sign cases check only its arithmetic; they provide no monotone-improvement result for a neural policy or finite minibatch.

Example. The PPO clipping cases for positive and negative advantage [ftip-003I]
✍️source

Reference. Proximal policy optimization algorithms [schulman2017proximal]

Reinforcement learning from human feedback and proximal policy optimization [ftip-0039]

Definition. Language-model RLHF pipeline [ouyang2022training, Section 3.1, Figure 2, and Section 3.5] [ftip-003A]

Remark. Reward modeling, alignment criteria, and policy optimization [ftip-003B]

Definition. Kullback--Leibler divergence [ftip-006X]

Remark. How the divergence enters alignment objectives [ftip-006Y]

Definition. KL-constrained RLHF objective [rafailov2023direct, Section 3, equation (3)] [ftip-003C]

Definition. KL-shaped PPO reward [ouyang2022training, Section 3.5, equation (2)] [ftip-003D]

Definition. Generalized advantage estimator [schulman2016gae, Section 3, equations (10) and (16)] [ftip-003E]

Definition. PPO clipped surrogate [schulman2017proximal, Section 3, equation (7)] [ftip-003F]

Definition. PPO value-and-entropy objective [schulman2017proximal, Section 5, equation (9)] [ftip-003G]

Definition. PPO rollout and minibatch update [schulman2017proximal, Section 5] [ftip-003H]

Example. The PPO clipping cases for positive and negative advantage [ftip-003I]
✍️source

Example. The InstructGPT-style RLHF pipeline [ftip-003J]

Remark. PPO is an optimizer, not an alignment definition [ftip-003K]