For \(z\in \mathbb R^d\), learned scale \(\gamma \in \mathbb R^d\), and a
declared numerical stabilizer \(\varepsilon _{\mathrm {RMS}}>0\),
root mean square layer normalization is
\[
\operatorname {RMSNorm}(z)
=\gamma \odot \frac {z}{\sqrt {d^{-1}\sum _{i=1}^{d}z_i^2+
\varepsilon _{\mathrm {RMS}}}}.
\]
Unlike layer normalization in Definition [ftip-0011], this operator does not
subtract the coordinate mean. The declared stabilizer makes it total at the
zero vector; setting the stabilizer to zero recovers the source expression on
nonzero inputs.