Definition. Outcome-supervised reward model [lightman2023verify, Section 2.5, ``Outcome-supervised reward models''] [ftip-003W]
Definition. Outcome-supervised reward model [lightman2023verify, Section 2.5, ``Outcome-supervised reward models''] [ftip-003W]
An outcome-supervised reward model assigns a score to a complete response and is trained from labels attached to the final outcome. For a prompt \(x\) and completed response \(y\), write its score as
\[ r_{\mathrm {out},\phi }(x,y)\in \mathbb R. \]The training label states whether the completed solution reaches the declared outcome. It need not identify the first invalid step or distinguish a sound derivation from an answer reached for an unsound reason.