Definition. Direct preference training stage [rafailov2023direct, Section 4] [ftip-003N]
Definition. Direct preference training stage [rafailov2023direct, Section 4] [ftip-003N]
Direct preference training starts from a fixed reference policy and a dataset of preferred and dispreferred responses. It evaluates a binary cross-entropy loss formed from the trainable policy's reference-relative log likelihoods and updates the policy directly.
This stage does not separately fit a scalar reward network, sample online rollouts for PPO, or train a value function. Its loss nevertheless comes from a particular reward-model and KL-regularized optimization derivation.