Definition. Decoder-only Transformer language model [phuong2022formal, sec. 6, Decoder-only transformers] [ftip-0012]

✍️sourceAGENTDRAFTED

A decoder-only Transformer language model consists of:

1. the token interface in Definition [ftip-000E];
2. an initial representation from Definition [ftip-000M] and Definition [ftip-000N];
3. a stack of \(L\) causal decoder layers built from the attention, feed-forward, residual, and normalization constructions in Definition [ftip-000S]--Definition [ftip-0011]; and
4. an output map from the final hidden vector to token logits.

If the final hidden vector at position \(t\) is \(h_t\in \mathbb R^d\), an output matrix \(W_U\in \mathbb R^{|\mathcal V|\times d}\) and bias \(b_U\in \mathbb R^{|\mathcal V|}\) give logits \(z_t=W_Uh_t+b_U\). The softmax \[ \pi _\theta (a\mid x_{1:t}) =\frac {\exp (z_{t,a})} {\sum _{v\in \mathcal V}\exp (z_{t,v})} \] realizes the conditional next-token law of Definition [ftip-000H]. The parameter tuple \(\theta \) contains every learned tensor named by this architecture.