Remark. Where positional information enters [ftip-000O]
Remark. Where positional information enters [ftip-000O]
Definition [ftip-000N] covers only mechanisms acting on the hidden vector at the embedding stage. The original Transformer adds a fixed or learned vector \(p_t\) to the token embedding, so \(r_t(z)=z+p_t\); see [vaswani2017attention, §3.5].
Relative-bias methods instead alter an attention score as a function of the displacement \(t-s\). ALiBi gives one such construction in [press2021train, §3].
Rotary position embedding applies position-indexed rotations to queries and keys rather than adding a vector to the residual stream; see RoFormer, Section 3.2, equations (14)--(16) [⧉]. These mechanisms are not interchangeable choices of one tensor: they act at different points in the computation, so a model specification must name the mechanism and placement.