Example. A two-token attention calculation [ftip-000R]
Example. A two-token attention calculation [ftip-000R]
A scalar attention head on two positions makes the masked softmax and its resulting value average explicit.
Take head width \(d_h=1\), so the scale \(1/\sqrt {d_h}\) is one. At position two both keys are causally available. Therefore \[ \operatorname {softmax}(0,\log 3) =\frac {(1,3)}{1+3} =\left (\tfrac 14,\tfrac 34\right ), \qquad z_2=\tfrac 14\cdot 2+\tfrac 34\cdot 6=5. \]
Substitution into the scaled dot-product attention of Definition [ftip-000Q] produces the displayed value. Projection matrices, multiple heads, normalization, and residual connections remain outside this scalar calculation; it represents only one attention operation.