Definition. Scaled dot-product attention [vaswani2017attention, sec. 3.2.1, equation (1)] [ftip-000Q]

✍️sourceAGENTDRAFTED

Using the head widths in Definition [ftip-000P], let \(T_q,T_k\geq 1\) be query and key-value sequence lengths. For \(Q\in \mathbb R^{T_q\times d_h}\), \(K\in \mathbb R^{T_k\times d_h}\), and \(V\in \mathbb R^{T_k\times d_v}\), scaled dot-product attention is \[ \operatorname {Att}(Q,K,V) =\operatorname {softmax}\left (\frac {QK^{\mathsf T}}{\sqrt {d_h}}\right )V. \] The softmax is applied across each row of the \(T_q\times T_k\) score matrix, so every output row is a convex combination of the value rows.