Definition. Multi-head self-attention [vaswani2017attention, sec. 3.2.2] [ftip-000U]
Definition. Multi-head self-attention [vaswani2017attention, sec. 3.2.2] [ftip-000U]
For each head \(h\in \{1,\ldots ,H\}\), form \(Q_h,K_h,V_h\) as in Definition [ftip-000P] from the same input \(X\), and apply the masked attention of Definition [ftip-000S]: \[ Z_h=\operatorname {softmax}\left ( \frac {Q_hK_h^{\mathsf T}}{\sqrt {d_h}}+M \right )V_h. \] With \(W^O\in \mathbb R^{H d_v\times d}\), multi-head self-attention is \[ \operatorname {MHA}(X) =\operatorname {Concat}(Z_1,\ldots ,Z_H)W^O. \] It returns one width-\(d\) vector at every input position.