Definition. Population pretraining objective [ftip-001C]

✍️sourceAGENTDRAFTED

Let \(Q_T\) be the law of packed examples induced by Remark [ftip-0018]--Definition [ftip-001A], and assume that its expected number of predicted positions is positive. The population pretraining objective is the expected total token loss divided by the expected number of predicted tokens: \[ L_{\mathrm {pop}}(\theta ) =\frac { \mathbb E_{\zeta \sim Q_T} \left [\sum _{t=2}^{T}\ell _t(\theta ;\zeta )\right ] }{ \mathbb E_{\zeta \sim Q_T} \left [\sum _{t=2}^{T}m_t(\zeta )\right ] }, \] using expectation as fixed in Notation [ftip-000A].