Definition. Token embedding [vaswani2017attention, sec. 3.4, Embeddings and Softmax] [ftip-000M]

✍️sourceAGENTDRAFTED

Let \(\mathcal V\) be the vocabulary of Definition [ftip-000D] and \(d\) the residual width of Notation [ftip-000L]. A token embedding matrix is a learned matrix \(E\in \mathbb R^{|\mathcal V|\times d}\). The hidden vector assigned to token \(a\in \mathcal V\) is the row \(E_a\in \mathbb R^d\).

For a sequence \(x_{1:n}\), embedding lookup produces the matrix \(X^{\mathrm {tok}}\in \mathbb R^{n\times d}\) with row \(X^{\mathrm {tok}}_t=E_{x_t}\). This operation maps token identifiers to vectors; it does not yet encode their positions.