Remark. Normalization operator and placement are separate choices [ftip-006G]
Remark. Normalization operator and placement are separate choices [ftip-006G]
Ba et al. define layer normalization in [ba2016layer, Section 3]. The original Transformer applies it after residual addition in [vaswani2017attention, Section 3.1]. Xiong et al. distinguish post-normalized and pre-normalized Transformers in [xiong2020layer, Section 3.3]. Either the normalization operator or its placement can change while the other is held fixed.
A later stability statement must therefore name both choices. A result about the normalization formula alone does not determine the residual path.