Definition. Scalar reward-model score [ouyang2022training, Section 3.5 and Appendix C.2] [ftip-002P]
Definition. Scalar reward-model score [ouyang2022training, Section 3.5 and Appendix C.2] [ftip-002P]
A scalar reward model is a parameterized function
\[ r_\phi :\{(x,y):x\in \mathcal X_{\mathrm {pr}},\ y\in \mathcal Y(x)\} \longrightarrow \mathbb R, \]whose score is fitted from comparison observations. The score orders or weights responses for a specified training procedure; it is not automatically the independent utility of the response. Ouyang et al. initialize the model from the supervised policy and replace its language-model head with a scalar output for this stage.