Definition. Pretraining source mixture [ftip-0016]
Definition. Pretraining source mixture [ftip-0016]
Using Notation [ftip-0015], a pretraining source mixture is the pair \(((Q_s)_{s\in \mathcal S},\alpha )\). It induces the document law \[ Q_{\alpha }(A)=\sum _{s\in \mathcal S}\alpha _s Q_s(A) \] for every document event \(A\): first sample a source \(s\) with probability \(\alpha _s\), then sample a document from \(Q_s\).
The mixture weights specify sampling frequency. They need not equal the fraction of bytes stored in each source. Resampling, epoch boundaries, or temperature-based weights must therefore be recorded as part of \(\alpha \) or its schedule.