For the residual width \(d\) of Notation [ftip-000L], a declared
numerical stabilizer \(\varepsilon _{\mathrm {LN}}>0\), and
\(z=(z_1,\ldots ,z_d)\in \mathbb R^d\), define
\(\mu (z)=d^{-1}\sum _i z_i\) and
\(\sigma ^2(z)=d^{-1}\sum _i(z_i-\mu (z))^2\). With learned vectors
\(\gamma ,\beta \in \mathbb R^d\), layer normalization is
\[
\operatorname {LN}(z)
=\gamma \odot \frac {z-\mu (z)\mathbf 1}
{\sqrt {\sigma ^2(z)+\varepsilon _{\mathrm {LN}}}}+\beta .
\]
It is applied independently to the hidden vector at each token position. The
declared stabilizer makes the numerical operator total on \(\mathbb R^d\); the
unstabilized expression is recovered where \(\sigma ^2(z)>0\) by setting
\(\varepsilon _{\mathrm {LN}}=0\).