Definition. Preference query selection [christiano2017deep, Section 2.2.4] [ftip-002M]
Definition. Preference query selection [christiano2017deep, Section 2.2.4] [ftip-002M]
Preference learning may choose which trajectory segments or responses to show to a judge rather than sampling every pair uniformly. The query-selection procedure assigns candidate comparisons a priority derived from the current reward-model ensemble and requests labels for selected pairs.
Because selection depends on the current model and candidate pool, the resulting comparison data reflect an acquisition policy. Active selection can concentrate labels on uncertain pairs, but it does not make the collected comparisons representative of an undeclared target population.