Remark. Randomness, schedules, and stopping [ftip-001J]
AGENTDRAFTED
A pretraining run is not determined by minimization of the empirical
objective \(L_N\) in Definition [ftip-001E]. It includes the initialization, data order, window offsets, dropout and
other model randomness, numerical precision, distributed reduction order,
optimizer state, every hyperparameter schedule, and a stopping rule. These
objects determine a distribution over final parameters even when the source
mixture and nominal objective agree.
The training configuration of [vaswani2017attention, sec. 5.3] and
the scaling experiments of [kaplan2020scaling, sec. 2.2] report concrete
instances of these choices. An optimization result depends on
which randomness it averages over and whether the stopping time is fixed or
data-dependent.