Definition. Layer normalization [ba2016layer, sec. 3] [ftip-0011]

✍️sourceAGENTDRAFTED

For the residual width \(d\) of Notation [ftip-000L], a declared numerical stabilizer \(\varepsilon _{\mathrm {LN}}>0\), and \(z=(z_1,\ldots ,z_d)\in \mathbb R^d\), define \(\mu (z)=d^{-1}\sum _i z_i\) and \(\sigma ^2(z)=d^{-1}\sum _i(z_i-\mu (z))^2\). With learned vectors \(\gamma ,\beta \in \mathbb R^d\), layer normalization is \[ \operatorname {LN}(z) =\gamma \odot \frac {z-\mu (z)\mathbf 1} {\sqrt {\sigma ^2(z)+\varepsilon _{\mathrm {LN}}}}+\beta . \] It is applied independently to the hidden vector at each token position. The declared stabilizer makes the numerical operator total on \(\mathbb R^d\); the unstabilized expression is recovered where \(\sigma ^2(z)>0\) by setting \(\varepsilon _{\mathrm {LN}}=0\).