Definition. Root mean square layer normalization [zhang2019root, Section 4, equation (4)] [ftip-006F]
Definition. Root mean square layer normalization [zhang2019root, Section 4, equation (4)] [ftip-006F]
For \(z\in \mathbb R^d\), learned scale \(\gamma \in \mathbb R^d\), and a declared numerical stabilizer \(\varepsilon _{\mathrm {RMS}}>0\), root mean square layer normalization is
\[ \operatorname {RMSNorm}(z) =\gamma \odot \frac {z}{\sqrt {d^{-1}\sum _{i=1}^{d}z_i^2+ \varepsilon _{\mathrm {RMS}}}}. \]Unlike layer normalization in Definition [ftip-0011], this operator does not subtract the coordinate mean. The declared stabilizer makes it total at the zero vector; setting the stabilizer to zero recovers the source expression on nonzero inputs.