Definition. Causal attention mask [vaswani2017attention, sec. 3.2.3] [ftip-000S]
Definition. Causal attention mask [vaswani2017attention, sec. 3.2.3] [ftip-000S]
For a length-\(T\) sequence, the causal attention mask is the matrix \(M\in (\mathbb R\cup \{-\infty \})^{T\times T}\) with \[ M_{ts}= \begin {cases} 0,&s\leq t,\\ -\infty ,&s>t. \end {cases} \] Masked self-attention replaces the score matrix in Definition [ftip-000Q] by \(QK^{\mathsf T}/\sqrt {d_h}+M\). Consequently, the output at position \(t\) has zero attention weight on every later position \(s>t\).