Definition. Minibatch gradient estimator [ftip-001G]
Definition. Minibatch gradient estimator [ftip-001G]
At update \(k\), let \(\mathcal B_k\) be the minibatch named in Notation [ftip-0015], and let \(N_k=\sum _{\zeta \in \mathcal B_k}\sum _{t=2}^{T}m_t(\zeta )\) be its number of predicted tokens. For \(N_k>0\), the minibatch gradient estimator is \[ g_k=\frac 1{N_k} \sum _{\zeta \in \mathcal B_k}\sum _{t=2}^{T} \nabla _\theta \ell _t(\theta _k;\zeta ), \] where \(\ell _t\) is defined in Definition [ftip-001B]. Whether \(g_k\) is unbiased for the gradient of Definition [ftip-001C] depends on the window sampler and minibatch weighting. Other sampling schemes require their own bias statement.