Remark. Randomness, schedules, and stopping [ftip-001J]
Remark. Randomness, schedules, and stopping [ftip-001J]
A pretraining run is not determined by minimization of the empirical objective \(L_N\) in Definition [ftip-001E]. It includes the initialization, data order, window offsets, dropout and other model randomness, numerical precision, distributed reduction order, optimizer state, every hyperparameter schedule, and a stopping rule. These objects determine a distribution over final parameters even when the source mixture and nominal objective agree.
The training configuration of [vaswani2017attention, sec. 5.3] and the scaling experiments of [kaplan2020scaling, sec. 2.2] report concrete instances of these choices. An optimization result depends on which randomness it averages over and whether the stopping time is fixed or data-dependent.