Example. Four rollout rewards, group advantages, and clipping [ftip-004N]
Example. Four rollout rewards, group advantages, and clipping [ftip-004N]
A rollout group is normalized to relative advantages and then passed to a clipped surrogate objective.
For a rollout group with rewards \(r_i\), form a group mean and scale and set \(\widehat A_i=(r_i-\bar r)/s\). For current-to-behaviour ratios \(\rho _i\) and clip interval \([1-\epsilon ,1+\epsilon ]\), define each pointwise term by
\[ \begin {aligned} \ell _i=\min \{&\rho _i\widehat A_i,\\ &\operatorname {clip}(\rho _i,1-\epsilon ,1+\epsilon )\widehat A_i\}. \end {aligned} \]The mean of the resulting terms is the group contribution to the update; its value depends on the sampled rewards, ratios, and clip width.
Group-relative normalization is grounded in [shao2024deepseekmath, Section 4.1], while clipping is grounded in [schulman2017proximal, Section 3]. The normalization convention and ratios must be declared by the training protocol; a positive surrogate mean alone gives no guarantee of improved held-out performance after an optimizer step.