Definition. KL-shaped PPO reward [ouyang2022training, Section 3.5, equation (2)] [ftip-003D]
Definition. KL-shaped PPO reward [ouyang2022training, Section 3.5, equation (2)] [ftip-003D]
For a prompt \(x\), sampled response \(y\), learned reward score \(r_\phi (x,y)\), and fixed reference policy \(\pi _{\mathrm {ref}}\), the sequence-level KL-shaped reward is
\[ R_{\mathrm {shape}}(x,y;\theta ) =r_\phi (x,y) -\beta \log \frac {\pi _\theta (y\mid x)} {\pi _{\mathrm {ref}}(y\mid x)}. \]Autoregressive factorization writes the logarithmic term as a sum of token-level log ratios. Ouyang et al. also mix a pretraining-gradient term into their reported PPO objective; that auxiliary term is separate from the shaped reward displayed here.