Decoder-only Transformer architecture [ftip-000K]
✍️source

The continuation law in Definition [ftip-000I] specifies what probabilities a language model exposes. A decoder-only Transformer specifies how a sequence of token identifiers is transformed into those probabilities. This section introduces one component at a time, with dimensions fixed before the component formulas are used.

Notation 1. Tensor dimensions, positions, heads, and layers [ftip-000L]

Let $T$ be the maximum token positions processed together, $d$ the residual-stream width, $d_{\mathrm {ff}}$ a feed-forward hidden width, $H$ the number of attention heads, $d_h$ the query-key width of one head, and $L$ the number of decoder layers. Positions are indexed by $t,s\in \{1,\ldots ,T\}$, heads by $h\in \{1,\ldots ,H\}$, and layers by $\ell \in \{1,\ldots ,L\}$.

A batch of $B$ hidden-state sequences is a tensor in $\mathbb R^{B\times T\times d}$. When the batch coordinate is irrelevant, write $X\in \mathbb R^{T\times d}$ and let $X_t\in \mathbb R^d$ denote its row at position $t$. Matrix products act on the final coordinate.

Definition 2. Token embedding [vaswani2017attention, sec. 3.4, Embeddings and Softmax] [ftip-000M]

Let $\mathcal V$ be the vocabulary of Definition [ftip-000D] and $d$ the residual width of Notation 1. A token embedding matrix is a learned matrix $E\in \mathbb R^{|\mathcal V|\times d}$. The hidden vector assigned to token $a\in \mathcal V$ is the row $E_a\in \mathbb R^d$.

For a sequence $x_{1:n}$, embedding lookup produces the matrix $X^{\mathrm {tok}}\in \mathbb R^{n\times d}$ with row $X^{\mathrm {tok}}_t=E_{x_t}$. This operation maps token identifiers to vectors; it does not yet encode their positions.

Definition 3. Embedding-stage positional transformation [ftip-000N]

Using the dimensions of Notation 1, an embedding-stage positional transformation is a specified family of position-dependent transformations $r_t:\mathbb R^d\to \mathbb R^d$, one for each $t\in \{1,\ldots ,T\}$. Applied after the token embedding of Definition 2, it produces the initial hidden vector \[ X^{(0)}_t=r_t(E_{x_t}). \] The family $(r_t)_{t=1}^T$ is part of the architecture, even when it has no learned parameters.

Remark 4. Where positional information enters [ftip-000O]

Definition 3 covers only mechanisms acting on the hidden vector at the embedding stage. The original Transformer adds a fixed or learned vector $p_t$ to the token embedding, so $r_t(z)=z+p_t$; see [vaswani2017attention, §3.5].

Relative-bias methods instead alter an attention score as a function of the displacement $t-s$. ALiBi gives one such construction in [press2021train, §3].

Rotary position embedding applies position-indexed rotations to queries and keys rather than adding a vector to the residual stream; see RoFormer, Section 3.2, equations (14)--(16) [⧉]. These mechanisms are not interchangeable choices of one tensor: they act at different points in the computation, so a model specification must name the mechanism and placement.

Definition 5. Query, key, and value projections [vaswani2017attention, sec. 3.2.1] [ftip-000P]

Let $X\in \mathbb R^{T\times d}$ use the notation of Notation 1. For attention head $h$, choose learned matrices \[ W_h^Q,W_h^K\in \mathbb R^{d\times d_h}, \qquad W_h^V\in \mathbb R^{d\times d_v}, \] where $d_v$ is the value width of one head. The query, key, and value matrices are \[ Q_h=XW_h^Q,\qquad K_h=XW_h^K,\qquad V_h=XW_h^V. \] Their rows associate a query, key, and value vector with each token position.

Definition 6. Scaled dot-product attention [vaswani2017attention, sec. 3.2.1, equation (1)] [ftip-000Q]

Using the head widths in Definition 5, let $T_q,T_k\geq 1$ be query and key-value sequence lengths. For $Q\in \mathbb R^{T_q\times d_h}$, $K\in \mathbb R^{T_k\times d_h}$, and $V\in \mathbb R^{T_k\times d_v}$, scaled dot-product attention is \[ \operatorname {Att}(Q,K,V) =\operatorname {softmax}\left (\frac {QK^{\mathsf T}}{\sqrt {d_h}}\right )V. \] The softmax is applied across each row of the $T_q\times T_k$ score matrix, so every output row is a convex combination of the value rows.

Example 7. A two-token attention calculation [ftip-000R]

A scalar attention head on two positions makes the masked softmax and its resulting value average explicit.

$\text{[math]}$

Take head width $d_h=1$, so the scale $1/\sqrt {d_h}$ is one. At position two both keys are causally available. Therefore \[ \operatorname {softmax}(0,\log 3) =\frac {(1,3)}{1+3} =\left (\tfrac 14,\tfrac 34\right ), \qquad z_2=\tfrac 14\cdot 2+\tfrac 34\cdot 6=5. \]

Substitution into the scaled dot-product attention of Definition 6 produces the displayed value. Projection matrices, multiple heads, normalization, and residual connections remain outside this scalar calculation; it represents only one attention operation.

Definition 8. Causal attention mask [vaswani2017attention, sec. 3.2.3] [ftip-000S]

For a length-$T$ sequence, the causal attention mask is the matrix $M\in (\mathbb R\cup \{-\infty \})^{T\times T}$ with \[ M_{ts}= \begin {cases} 0,&s\leq t,\\ -\infty ,&s>t. \end {cases} \] Masked self-attention replaces the score matrix in Definition 6 by $QK^{\mathsf T}/\sqrt {d_h}+M$. Consequently, the output at position $t$ has zero attention weight on every later position $s>t$.

Example 9. A length-four causal mask [ftip-000T]

For four positions, a causal mask admits the current and earlier keys and excludes every future key.

$\text{[math]}$

Rows index querying positions and columns index key positions. In row three, the first three entries remain available and the fourth receives $-\infty $; after softmax the fourth token therefore contributes exactly zero.

The construction is the autoregressive mask of [vaswani2017attention, Section 3.2.3]. It enforces a dependency restriction. Effective use of the available earlier positions is a separate model property.

Definition 10. Multi-head self-attention [vaswani2017attention, sec. 3.2.2] [ftip-000U]

For each head $h\in \{1,\ldots ,H\}$, form $Q_h,K_h,V_h$ as in Definition 5 from the same input $X$, and apply the masked attention of Definition 8: \[ Z_h=\operatorname {softmax}\left ( \frac {Q_hK_h^{\mathsf T}}{\sqrt {d_h}}+M \right )V_h. \] With $W^O\in \mathbb R^{H d_v\times d}$, multi-head self-attention is \[ \operatorname {MHA}(X) =\operatorname {Concat}(Z_1,\ldots ,Z_H)W^O. \] It returns one width-$d$ vector at every input position.

Remark 11. Multi-head, grouped-query, and compressed-attention variants [ftip-000V]

The multi-head construction in Definition 10 stores a key and value projection per query head. Grouped-query attention instead partitions query heads into groups that share key and value heads; see Ainslie et al., Section 2 [⧉]. This changes parameter and cache shapes while preserving the query-key-value semantics.

Multi-head latent attention first compresses key-value information through a lower-dimensional latent representation and reconstructs head-specific quantities; see DeepSeek-V2, Section 2.1 [⧉]. DeepSeek-V3, Section 2.1, records a later use of that architecture [⧉]. These architectural differences do not establish that their post-training effects equal those of ordinary multi-head attention.

Definition 12. Positionwise feed-forward sublayer [vaswani2017attention, sec. 3.3, equation (2)] [ftip-000W]

With widths $d$ and $d_{\mathrm {ff}}$ from Notation 1, a positionwise feed-forward sublayer is a map $F:\mathbb R^d\to \mathbb R^d$ of the form \[ F(z)=\phi (zW_1+b_1)W_2+b_2, \] where $W_1\in \mathbb R^{d\times d_{\mathrm {ff}}}$, $W_2\in \mathbb R^{d_{\mathrm {ff}}\times d}$, the vectors $b_1,b_2$ have the corresponding widths, and $\phi $ is applied coordinatewise. The same map is applied independently at each token position.

Definition 13. Sparse mixture-of-experts routing [ftip-000X]

Let $J\geq 1$ and $1\leq k\leq J$. Let $F_1,\ldots ,F_J:\mathbb R^d\to \mathbb R^d$ be feed-forward experts and let $g:\mathbb R^d\to \mathbb R^J$ be a router. For a hidden vector $z$, let $S_k(z)$ be the indices of the $k$ largest coordinates of $g(z)$, breaking equal scores by a fixed total order on expert indices. A sparse mixture-of-experts layer returns \[ F_{\mathrm {MoE}}(z) =\sum _{j\in S_k(z)}\alpha _j(z)F_j(z), \qquad \alpha _j(z)= \frac {\exp g_j(z)}{\sum _{i\in S_k(z)}\exp g_i(z)}. \] Only the selected experts are evaluated for that token.

Remark 14. Dense and routed feed-forward computation [ftip-000Y]

The dense sublayer in Definition 12 evaluates one learned map for every token. The sparsely gated construction of Shazeer et al., Section 2, selects among multiple experts and introduces an additional routing and load-balancing problem [⧉]. The top-$k$ selection rule is described in Definition 13.

Parameter count, active parameter count, and executed operations therefore need not agree. A post-training comparison involving routed models must state which quantity is held fixed and whether the router itself is updated.

Example 15. Top-k expert routing and its DeepSeek counterpart [ftip-000Z]

A routed mixture-of-experts layer uses router scores to activate only a declared subset of routed experts, then combines their outputs with any shared expert path.

$\text{[math]}$

The revision-pinned configuration [⧉] declares $256$ routed experts, $6$ experts selected per token, and one shared expert. This instantiates the selection interface while leaving the artifact's router scores, load-balancing mechanism, and expert outputs unspecified.

Definition 16. Residual connection [vaswani2017attention, sec. 3.1] [ftip-0010]

Using the sequence and residual dimensions of Notation 1, let $F:\mathbb R^{T\times d}\to \mathbb R^{T\times d}$ be a sublayer whose input and output have the same shape. A residual connection forms \[ R_F(X)=X+F(X). \] The addition is coordinatewise. Shape equality is part of the construction; the residual path is not a concatenation of features.

Definition 17. Layer normalization [ba2016layer, sec. 3] [ftip-0011]

For the residual width $d$ of Notation 1, a declared numerical stabilizer $\varepsilon _{\mathrm {LN}}>0$, and $z=(z_1,\ldots ,z_d)\in \mathbb R^d$, define $\mu (z)=d^{-1}\sum _i z_i$ and $\sigma ^2(z)=d^{-1}\sum _i(z_i-\mu (z))^2$. With learned vectors $\gamma ,\beta \in \mathbb R^d$, layer normalization is \[ \operatorname {LN}(z) =\gamma \odot \frac {z-\mu (z)\mathbf 1} {\sqrt {\sigma ^2(z)+\varepsilon _{\mathrm {LN}}}}+\beta . \] It is applied independently to the hidden vector at each token position. The declared stabilizer makes the numerical operator total on $\mathbb R^d$; the unstabilized expression is recovered where $\sigma ^2(z)>0$ by setting $\varepsilon _{\mathrm {LN}}=0$.

Definition 18. Normalization placement [ftip-006E]

For a residual stream $X$, a sublayer $F$, and a normalization operator $N$, post-normalization applies $N(X+F(X))$. In pre-normalization, the sublayer receives a normalized input and the residual update is $X+F(N(X))$.

Placement is an architecture coordinate, separate from the formula for $N$. Every decoder block must state which residual branches use which placement.

Definition 19. Root mean square layer normalization [zhang2019root, Section 4, equation (4)] [ftip-006F]

For $z\in \mathbb R^d$, learned scale $\gamma \in \mathbb R^d$, and a declared numerical stabilizer $\varepsilon _{\mathrm {RMS}}>0$, root mean square layer normalization is

\[ \operatorname {RMSNorm}(z) =\gamma \odot \frac {z}{\sqrt {d^{-1}\sum _{i=1}^{d}z_i^2+ \varepsilon _{\mathrm {RMS}}}}. \]

Unlike layer normalization in Definition 17, this operator does not subtract the coordinate mean. The declared stabilizer makes it total at the zero vector; setting the stabilizer to zero recovers the source expression on nonzero inputs.

Remark 20. Normalization operator and placement are separate choices [ftip-006G]

Ba et al. define layer normalization in [ba2016layer, Section 3]. The original Transformer applies it after residual addition in [vaswani2017attention, Section 3.1]. Xiong et al. distinguish post-normalized and pre-normalized Transformers in [xiong2020layer, Section 3.3]. Either the normalization operator or its placement can change while the other is held fixed.

A later stability statement must therefore name both choices. A result about the normalization formula alone does not determine the residual path.

Definition 21. Decoder-only Transformer language model [phuong2022formal, sec. 6, Decoder-only transformers] [ftip-0012]

A decoder-only Transformer language model consists of:

the token interface in Definition [ftip-000E];
an initial representation from Definition 2 and Definition 3;
a stack of $L$ causal decoder layers built from the attention, feed-forward, residual, and normalization constructions in Definition 8--Definition 17; and
an output map from the final hidden vector to token logits.

If the final hidden vector at position $t$ is $h_t\in \mathbb R^d$, an output matrix $W_U\in \mathbb R^{|\mathcal V|\times d}$ and bias $b_U\in \mathbb R^{|\mathcal V|}$ give logits $z_t=W_Uh_t+b_U$. The softmax \[ \pi _\theta (a\mid x_{1:t}) =\frac {\exp (z_{t,a})} {\sum _{v\in \mathcal V}\exp (z_{t,v})} \] realizes the conditional next-token law of Definition [ftip-000H]. The parameter tuple $\theta $ contains every learned tensor named by this architecture.

Example 22. Decoder-only dataflow and a DeepSeek V4-Flash provenance overlay [ftip-0013]

A decoder pipeline maps token identifiers through hidden states to logits; the released model instance and family report provide different evidence about that pipeline.

$\text{[math]}$

The pinned configuration [⧉] declares 43 layers, hidden width 4096, 64 attention heads, and one key--value head. Section 2 of the family report [⧉] describes the preview architecture. Neither source identifies a complete 0731-specific architecture or training delta. The diagram records the interface, not a small copy of the released model.

References

Reference. Formal algorithms for transformers [phuong2022formal]

2022
Mary Phuong, Marcus Hutter
http://arxiv.org/abs/2207.09238

@article{phuong2022formal,
 title = {Formal algorithms for transformers},
 author = {Phuong, Mary and Hutter, Marcus},
 year = {2022},
 url = {http://arxiv.org/abs/2207.09238},
 journal = {arXiv preprint arXiv:2207.09238}
}

Reference. Train short, test long: Attention with linear biases enables input length extrapolation [press2021train]

2021
Ofir Press, Noah A Smith, Mike Lewis
https://arxiv.org/abs/2108.12409

@article{press2021train,
 title = {Train short, test long: Attention with linear biases
enables input length extrapolation},
 author = {Press, Ofir and Smith, Noah A and Lewis, Mike},
 year = {2021},
 url = {https://arxiv.org/abs/2108.12409},
 journal = {arXiv preprint arXiv:2108.12409}
}

Reference. On Layer Normalization in the Transformer Architecture [xiong2020layer]

2020
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu
https://arxiv.org/abs/2002.04745

@inproceedings{xiong2020layer,
  title = {On Layer Normalization in the Transformer Architecture},
  author = {Xiong, Ruibin and Yang, Yunchang and He, Di and Zheng, Kai and Zheng, Shuxin and Xing, Chen and Zhang, Huishuai and Lan, Yanyan and Wang, Liwei and Liu, Tie-Yan},
  booktitle = {Proceedings of the 37th International Conference on Machine Learning},
  year = {2020},
  url = {https://arxiv.org/abs/2002.04745}
}

Reference. Root Mean Square Layer Normalization [zhang2019root]

2019
Biao Zhang, Rico Sennrich
https://arxiv.org/abs/1910.07467

@inproceedings{zhang2019root,
  title = {Root Mean Square Layer Normalization},
  author = {Zhang, Biao and Sennrich, Rico},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://arxiv.org/abs/1910.07467}
}

Reference. Attention is all you need [vaswani2017attention]

2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin
https://proceedings.neurips.cc/paper/7181-attention-is-all-you-need

@article{vaswani2017attention,
 title = {Attention is all you need},
 author = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and
Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and
Kaiser, {\L}ukasz and Polosukhin, Illia},
 year = {2017},
 url = {http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf},
 journal = {Advances in neural information processing systems},
 volume = {30}
}

Reference. Layer normalization [ba2016layer]

2016
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton
https://arxiv.org/abs/1607.06450

@article{ba2016layer,
 title = {Layer normalization},
 author = {Ba, Jimmy Lei and Kiros, Jamie Ryan and Hinton, Geoffrey
E},
 year = {2016},
 url = {https://arxiv.org/abs/1607.06450},
 journal = {arXiv preprint arXiv:1607.06450}
}

Context

Architecture-conditioned potential and computation [ftip-00JC]

This chapter refines the evaluation framework for claims that depend on architecture, optimization geometry or a systems implementation. It begins with the decoder-only Transformer, then introduces architecture-indexed frontiers and matched-compute comparisons, and studies how an optimizer can depend on the representation of its parameters.

The reference architecture specifies how token probabilities are computed; it does not replace the shared task and evaluation interfaces. Readers whose question does not depend on an architecture or optimizer can continue directly to conceptual discovery. Changes to retained context and rollout computation are treated with agent state.

Decoder-only Transformer architecture [ftip-000K]
✍️source

Notation. Tensor dimensions, positions, heads, and layers [ftip-000L]

Definition. Token embedding [vaswani2017attention, sec. 3.4, Embeddings and Softmax] [ftip-000M]

Let $\mathcal V$ be the vocabulary of Definition [ftip-000D] and $d$ the residual width of Notation [ftip-000L]. A token embedding matrix is a learned matrix $E\in \mathbb R^{|\mathcal V|\times d}$. The hidden vector assigned to token $a\in \mathcal V$ is the row $E_a\in \mathbb R^d$.

Definition. Embedding-stage positional transformation [ftip-000N]

Using the dimensions of Notation [ftip-000L], an embedding-stage positional transformation is a specified family of position-dependent transformations $r_t:\mathbb R^d\to \mathbb R^d$, one for each $t\in \{1,\ldots ,T\}$. Applied after the token embedding of Definition [ftip-000M], it produces the initial hidden vector \[ X^{(0)}_t=r_t(E_{x_t}). \] The family $(r_t)_{t=1}^T$ is part of the architecture, even when it has no learned parameters.

Remark. Where positional information enters [ftip-000O]

Definition [ftip-000N] covers only mechanisms acting on the hidden vector at the embedding stage. The original Transformer adds a fixed or learned vector $p_t$ to the token embedding, so $r_t(z)=z+p_t$; see [vaswani2017attention, §3.5].

Relative-bias methods instead alter an attention score as a function of the displacement $t-s$. ALiBi gives one such construction in [press2021train, §3].

Definition. Query, key, and value projections [vaswani2017attention, sec. 3.2.1] [ftip-000P]

Let $X\in \mathbb R^{T\times d}$ use the notation of Notation [ftip-000L]. For attention head $h$, choose learned matrices \[ W_h^Q,W_h^K\in \mathbb R^{d\times d_h}, \qquad W_h^V\in \mathbb R^{d\times d_v}, \] where $d_v$ is the value width of one head. The query, key, and value matrices are \[ Q_h=XW_h^Q,\qquad K_h=XW_h^K,\qquad V_h=XW_h^V. \] Their rows associate a query, key, and value vector with each token position.

Definition. Scaled dot-product attention [vaswani2017attention, sec. 3.2.1, equation (1)] [ftip-000Q]

Using the head widths in Definition [ftip-000P], let $T_q,T_k\geq 1$ be query and key-value sequence lengths. For $Q\in \mathbb R^{T_q\times d_h}$, $K\in \mathbb R^{T_k\times d_h}$, and $V\in \mathbb R^{T_k\times d_v}$, scaled dot-product attention is \[ \operatorname {Att}(Q,K,V) =\operatorname {softmax}\left (\frac {QK^{\mathsf T}}{\sqrt {d_h}}\right )V. \] The softmax is applied across each row of the $T_q\times T_k$ score matrix, so every output row is a convex combination of the value rows.

Example. A two-token attention calculation [ftip-000R]

A scalar attention head on two positions makes the masked softmax and its resulting value average explicit.

$\text{[math]}$

Substitution into the scaled dot-product attention of Definition [ftip-000Q] produces the displayed value. Projection matrices, multiple heads, normalization, and residual connections remain outside this scalar calculation; it represents only one attention operation.

Definition. Causal attention mask [vaswani2017attention, sec. 3.2.3] [ftip-000S]

For a length-$T$ sequence, the causal attention mask is the matrix $M\in (\mathbb R\cup \{-\infty \})^{T\times T}$ with \[ M_{ts}= \begin {cases} 0,&s\leq t,\\ -\infty ,&s>t. \end {cases} \] Masked self-attention replaces the score matrix in Definition [ftip-000Q] by $QK^{\mathsf T}/\sqrt {d_h}+M$. Consequently, the output at position $t$ has zero attention weight on every later position $s>t$.

Example. A length-four causal mask [ftip-000T]

For four positions, a causal mask admits the current and earlier keys and excludes every future key.

$\text{[math]}$

Definition. Multi-head self-attention [vaswani2017attention, sec. 3.2.2] [ftip-000U]

For each head $h\in \{1,\ldots ,H\}$, form $Q_h,K_h,V_h$ as in Definition [ftip-000P] from the same input $X$, and apply the masked attention of Definition [ftip-000S]: \[ Z_h=\operatorname {softmax}\left ( \frac {Q_hK_h^{\mathsf T}}{\sqrt {d_h}}+M \right )V_h. \] With $W^O\in \mathbb R^{H d_v\times d}$, multi-head self-attention is \[ \operatorname {MHA}(X) =\operatorname {Concat}(Z_1,\ldots ,Z_H)W^O. \] It returns one width-$d$ vector at every input position.

Remark. Multi-head, grouped-query, and compressed-attention variants [ftip-000V]

The multi-head construction in Definition [ftip-000U] stores a key and value projection per query head. Grouped-query attention instead partitions query heads into groups that share key and value heads; see Ainslie et al., Section 2 [⧉]. This changes parameter and cache shapes while preserving the query-key-value semantics.

Definition. Positionwise feed-forward sublayer [vaswani2017attention, sec. 3.3, equation (2)] [ftip-000W]

With widths $d$ and $d_{\mathrm {ff}}$ from Notation [ftip-000L], a positionwise feed-forward sublayer is a map $F:\mathbb R^d\to \mathbb R^d$ of the form \[ F(z)=\phi (zW_1+b_1)W_2+b_2, \] where $W_1\in \mathbb R^{d\times d_{\mathrm {ff}}}$, $W_2\in \mathbb R^{d_{\mathrm {ff}}\times d}$, the vectors $b_1,b_2$ have the corresponding widths, and $\phi $ is applied coordinatewise. The same map is applied independently at each token position.

Definition. Sparse mixture-of-experts routing [ftip-000X]

Remark. Dense and routed feed-forward computation [ftip-000Y]

The dense sublayer in Definition [ftip-000W] evaluates one learned map for every token. The sparsely gated construction of Shazeer et al., Section 2, selects among multiple experts and introduces an additional routing and load-balancing problem [⧉]. The top-$k$ selection rule is described in Definition [ftip-000X].

Example. Top-k expert routing and its DeepSeek counterpart [ftip-000Z]

A routed mixture-of-experts layer uses router scores to activate only a declared subset of routed experts, then combines their outputs with any shared expert path.

$\text{[math]}$

Definition. Residual connection [vaswani2017attention, sec. 3.1] [ftip-0010]

Using the sequence and residual dimensions of Notation [ftip-000L], let $F:\mathbb R^{T\times d}\to \mathbb R^{T\times d}$ be a sublayer whose input and output have the same shape. A residual connection forms \[ R_F(X)=X+F(X). \] The addition is coordinatewise. Shape equality is part of the construction; the residual path is not a concatenation of features.

Definition. Layer normalization [ba2016layer, sec. 3] [ftip-0011]

For the residual width $d$ of Notation [ftip-000L], a declared numerical stabilizer $\varepsilon _{\mathrm {LN}}>0$, and $z=(z_1,\ldots ,z_d)\in \mathbb R^d$, define $\mu (z)=d^{-1}\sum _i z_i$ and $\sigma ^2(z)=d^{-1}\sum _i(z_i-\mu (z))^2$. With learned vectors $\gamma ,\beta \in \mathbb R^d$, layer normalization is \[ \operatorname {LN}(z) =\gamma \odot \frac {z-\mu (z)\mathbf 1} {\sqrt {\sigma ^2(z)+\varepsilon _{\mathrm {LN}}}}+\beta . \] It is applied independently to the hidden vector at each token position. The declared stabilizer makes the numerical operator total on $\mathbb R^d$; the unstabilized expression is recovered where $\sigma ^2(z)>0$ by setting $\varepsilon _{\mathrm {LN}}=0$.

Definition. Normalization placement [ftip-006E]

Placement is an architecture coordinate, separate from the formula for $N$. Every decoder block must state which residual branches use which placement.

Definition. Root mean square layer normalization [zhang2019root, Section 4, equation (4)] [ftip-006F]

For $z\in \mathbb R^d$, learned scale $\gamma \in \mathbb R^d$, and a declared numerical stabilizer $\varepsilon _{\mathrm {RMS}}>0$, root mean square layer normalization is

\[ \operatorname {RMSNorm}(z) =\gamma \odot \frac {z}{\sqrt {d^{-1}\sum _{i=1}^{d}z_i^2+ \varepsilon _{\mathrm {RMS}}}}. \]

Unlike layer normalization in Definition [ftip-0011], this operator does not subtract the coordinate mean. The declared stabilizer makes it total at the zero vector; setting the stabilizer to zero recovers the source expression on nonzero inputs.

Remark. Normalization operator and placement are separate choices [ftip-006G]

A later stability statement must therefore name both choices. A result about the normalization formula alone does not determine the residual path.

Definition. Decoder-only Transformer language model [phuong2022formal, sec. 6, Decoder-only transformers] [ftip-0012]

A decoder-only Transformer language model consists of:

the token interface in Definition [ftip-000E];
an initial representation from Definition [ftip-000M] and Definition [ftip-000N];
a stack of $L$ causal decoder layers built from the attention, feed-forward, residual, and normalization constructions in Definition [ftip-000S]--Definition [ftip-0011]; and
an output map from the final hidden vector to token logits.

Example. Decoder-only dataflow and a DeepSeek V4-Flash provenance overlay [ftip-0013]

A decoder pipeline maps token identifiers through hidden states to logits; the released model instance and family report provide different evidence about that pipeline.

$\text{[math]}$

Architecture-indexed ceilings and matched compute [ftip-00JF]

This optional section refines the architecture-neutral capability framework only when the question names an architecture, optimizer, or systems implementation. The general model/interface formulation remains available when these fields are unnecessary.

Architecture comparisons below are conditional on a declared task family, evaluation law, intervention class, and scalar cost. Toy bounds are finite results under their displayed assumptions and do not transfer to concrete model instances without verified interface, task, and cost assumptions.

Architecture, optimizer, and systems variables [ftip-00JG]

Architecture, optimizer, and systems fields refine the model/interface description. A comparison depends only on the fields named in its assumptions and conclusion.

Definition. Optional architecture refinement [ftip-00JH]

Let $M$ denote an existing model intervention. An architecture refinement is a record $A=(\mathcal X,\mathcal Y,\operatorname {Map}_A)$ whose map realizes the same declared input and output interface. Forgetting $A$ returns $M$; no architecture claim is made when the field is omitted.

Definition. Optimizer and systems records [ftip-00JI]

When needed, extend the record by $O$ for optimizer and state-update rules, and $S$ for kernels, precision, memory, cache, and scheduling. The triplet $(A,O,S)$ is descriptive; it is not a scalar intelligence score.

Definition. Architecture-indexed evaluation functional [ftip-00JJ]

Fix an architecture $A$, its base artifact, a declared intervention set $\mathfrak I_A$, and the common evaluation interface of Convention [ftip-005D]. Each $\eta \in \mathfrak I_A$ specifies a protocol $P_{A,\eta }=\operatorname {PostTrain}(A,\eta )$ in that measurable and absolutely integrable evaluation domain, with real performance $J_{\rm ev}(P_{A,\eta })$. Let $c_A(\eta )\in [0,+\infty ]$ be its scalar cost under the study's declared accounting rule: a fixed measured cost or an expected nonnegative cost, as specified. For a finite budget $C\in \mathbb R_{\geq 0}$, define \[ V_A(C)=\sup \left \{ J_{\rm ev}(P_{A,\eta }): \eta \in \mathfrak I_A,\ c_A(\eta )\leq C \right \}\in \overline {\mathbb R}. \] Infinite-cost interventions are infeasible at every such budget.

As in Definition [ftip-005M], the empty feasible set has value $-\infty $; an unbounded-above feasible score set has value $+\infty $. The value is real exactly when that set is nonempty and bounded above. Increasing $C$ enlarges the feasible set, so $V_A$ is nondecreasing. A finite supremum need not be attained by any intervention.

Frontier differences and derivatives are ordinary real operations only where the relevant values are finite, with differentiability additionally required for a derivative. Cost ratios use the domain in Definition [ftip-00JP]. The allowed intervention set, evaluation law, and accounting rule are part of the definition, so $V_A$ is not a universal intelligence function.

Remark. Architecture-neutral projection and gradual complexity [ftip-00JK]

A statement that depends only on the model/interface map is invariant under changes to architecture $A$, optimizer $O$, or systems $S$ that leave that map fixed. A conclusion about one of those fields instead requires assumptions that distinguish its possible values.

Definition. Three ceiling layers [ftip-00JL]

For fixed task and evaluation laws, distinguish the representational ceiling of $A$, the optimizer-reachable ceiling of $(A,O)$, and the systems-feasible frontier of $(A,O,S,C)$. Each layer is conditional on the objects named; none implies that the next layer attains it.

Remark. Architecture effects are comparative estimands [ftip-00JM]

A difference between two architectures is meaningful only after the task, data, post-training, inference, evaluation, and cost records identify which coordinates are held fixed and which are allowed to vary.

Depth reuse and latent recurrence [ftip-00KG]

Repeated depth, latent feedback, and recirculation change the computation performed at inference. Relative to the fixed reference autoregressive Transformer, their costs include any extra passes, state, and prefill work.

Definition. Computation-axis record [ftip-00KH]

For an architecture record $A$, distinguish token steps, feed-forward depth passes, recurrent passes, and latent-state updates. Let $d(A,r)$ be the declared number of depth passes in run $r$; it is a computation coordinate, not a synonym for parameter count or reasoning quality.

Remark. Looped Transformers as a source result [ftip-00KI]

Giannou et al. show that a fixed shallow Transformer placed in a loop can execute programmed iterative computations, including conditional branches and in-context algorithms, in Looped Transformers as Programmable Computers[giannou2023looped]. This is a source result under its program, state, and precision assumptions; it is not a universal claim about language-model capability.

Remark. Parameter sharing and width tradeoffs [ftip-00KJ]

Xue et al. report that depth sharing can reduce trainable parameters while limiting modeling capacity, and study width and mixture-of-experts remedies in Go Wider Instead of Deeper[xue2021wider]. The comparison is empirical and task-specific; it does not identify a depth-independent ceiling.

Remark. Virtual logical depth and scaling [ftip-00KK]

Zhu et al. vary virtual logical depth by reusing weights and report reasoning gains at nearly fixed parameter count in Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws[zhu2025virtualdepth]. Their results motivate a depth coordinate in a scaling study, while leaving knowledge capacity, optimization, and transfer dependent on the declared training and evaluation protocol.

Remark. Recirculation as an inference intervention [ftip-00KL]

Mozer et al. introduce inference-time recirculation that feeds latent states back through an off-the-shelf model and report task improvements in Recirculation[mozer2026recirculation]. The added serial prefill and adaptive tuning belong in the systems and inference records; the observation is not a proof of a new representational ceiling.

Remark. Latent feedback with a preserved Transformer interface [ftip-00KM]

Wang et al. widen the feedback channel between decoding steps with latent feedback while retaining a Transformer and language-modeling interface in Full-bandwidth transformer[wang2026fullbandwidth]. Any reported token or accuracy savings are conditional on the scheduled multi-pass training and measured decoding cost.

Remark. Latent prediction as a training intervention [ftip-00KN]

Teoh et al. add next-latent prediction to next-token training and report compact predictive states without changing the Transformer interface in Next-Latent Prediction Transformers Learn Compact World Models[teoh2025nextlatent]. This belongs to the training intervention record, not to an architecture-only comparison.

Example. Equal parameters do not fix effective depth [ftip-00KO]

Two runs can share parameter count while using different numbers of recurrent or latent-feedback passes. A matched study must therefore report the computation-axis record and cannot infer an architecture ceiling from parameters alone.

Definition. Depth-reuse model-instance record [ftip-00KP]

For a depth-reuse comparison, record shared weights, pass count, stopping rule, recurrent state, training objective, token and pass FLOPs, prefill and decode latency, memory, and evaluation seeds. A pass-count change is an intervention even when the parameter tensor is unchanged.

Example. Fixed-recipe depth comparison [ftip-00KQ]

In a fixed-recipe arm, hold data, optimizer, feedback, stopping rule, and evaluation law fixed while varying only the declared depth-reuse intervention. The resulting difference is an estimand for that recipe, not a best-achievable comparison.

Remark. Scaling surfaces rather than a single law [ftip-00KR]

A depth-reuse study may fit a surface over parameters, tokens, passes, and cost. The fit is an empirical summary over its measured range; it does not establish an asymptotic law or a universal saturation point.

Remark. Architecture and training are separable records [ftip-00KS]

Weight sharing, latent objectives, and recirculation can alter optimization without changing the declared external interface. The comparison must retain architecture, training intervention, and systems coordinates separately.

Remark. Depth reuse across passes, parameters, and tasks [ftip-00KT]

The cited looped, virtual-depth, latent-feedback, and recirculation results do not show that more passes always improve capability, that parameter sharing dominates added parameters, or that inference-time gains transfer to training or to another task family. Each transfer requires a matched model-instance record.

Ceilings, frontiers, and saturation [ftip-00JN]

Definition. Fixed-recipe and tuned frontiers [ftip-00JO]

The fixed-recipe frontier evaluates one common $\eta $. The equal-tuning frontier permits architecture-specific choices from a predeclared trial set with the same tuning data and budget. The restricted envelope permits a predeclared class of interventions under $C$. These are distinct estimands.

Definition. Iso-quality cost ratio [ftip-00JP]

For a finite real target $q\in \mathbb R$ and the frontier of Definition [ftip-00JJ], define the inverse cost \[ C_A(q)=\inf \{C\in \mathbb R_{\geq 0}:V_A(C)\geq q\} \in [0,+\infty ], \qquad \inf \varnothing =+\infty . \] The iso-quality ratio is defined only on the domain \[ \rho _{A/B}(q)=\frac {C_A(q)}{C_B(q)}, \qquad 0\leq C_A(q)<+\infty , \quad 0<C_B(q)<+\infty . \] The architectures must use the same target, evaluation interface, cost units, and comparison arm. The ratio compares efficiency under these choices.

An inverse cost is a threshold infimum, not an executable minimum. The infimum over budgets may be unattained; even if a budget satisfies $V_A(C)\geq q$, its performance supremum may be unattained at $q$. An actual target-achieving intervention requires a separate witness.

Positive individual costs do not guarantee a positive inverse cost. For interventions $\eta _n$, $n\geq 1$, with score $1$ and cost $1/n$, the target $q=1$ has $C_A(1)=0$, although every intervention costs more than zero. If both architectures have this family, the putative ratio is $0/0$ and is excluded by the displayed domain.

Definition. Operational cost vector and scalarization [ftip-00JQ]

Record training FLOPs, inference FLOPs, wall time, memory, energy, and hardware separately as $\mathbf c$. A scalar cost $C=w\cdot \mathbf c$ is a declared study choice with nonnegative units $w$; changing $w$ changes the frontier and must not be hidden as architecture quality.

Theorem. Finite-window representational obstruction [ftip-00JR]

Consider a causal system whose state after each prefix has at most $K$ distinct values, and a task with $K+1$ prefixes requiring pairwise distinct continuation labels. By the pigeonhole principle two prefixes share a state, so at least one continuation label is wrong. This is a finite toy obstruction only; it does not bound a concrete Transformer or KDA instance without a proved reduction to this state model.

Theorem. A conditional saturation certificate [ftip-00JS]

Let $V_A$ be the nondecreasing frontier of Definition [ftip-00JJ], bounded above by a finite $U\in \mathbb R$. If an intervention reaches a real target $q<U$ at a finite cost $C_q\geq 0$, then at every finite $C\geq C_q$, $q\leq V_A(C)\leq U$. In particular $V_A(C)$ is real and $0\leq U-V_A(C)\leq U-q$. This elementary certificate is conditional on the bound $U$; it does not assert that real intelligence saturates.

Example. Equal scalar cost does not identify architecture [ftip-00JT]

Two systems can have the same scalar $C$ while differing in memory, latency, and training allocation. A cost-weight change can reverse their ordering without changing either system. Thus a one-factor comparison needs a fixed scalarization and a reported cost vector; no architecture effect follows from equal $C$ alone.

Matched-compute comparison protocol [ftip-00JU]

Definition. Common-recipe comparison arm [ftip-00JV]

A common-recipe arm fixes training data, optimizer family, schedule, post-training procedure and feedback, training and post-training budgets, inference budget, and evaluation protocol and draws. Interface-compatible parameter choices are declared in advance as part of each permitted architecture package. The comparison varies that package under the common recipe; any further deviation is recorded as a separate comparison coordinate. Its score difference measures performance under this recipe, not the best attainable result for either architecture.

Definition. Equal-tuning-budget comparison arm [ftip-00JW]

An equal-tuning-budget arm permits architecture-specific tuning under a predeclared protocol that fixes the tuning data and its volume, trial count, selection rule, stopping rule, and tuning cost for both architectures. The cost uses the same declared accounting rule. The resulting comparison measures practical performance under this equal optimization effort; it does not determine best possible training or a representation-only limit.

Definition. Restricted-envelope comparison arm [ftip-00JX]

A restricted-envelope arm declares an allowed intervention class $\mathcal E_A\subseteq \mathfrak I_A$ for each architecture, specifying the permitted training, post-training, and inference procedures. The classes expose exclusions, search budgets, seeds, and stopping rules. Restrict the evaluation functional of Definition [ftip-00JJ] to $\mathcal E_A$ and compare the resulting suprema under the same finite cost cap $C\geq 0$, common evaluation law, and common declared cost accounting.

The measurable, integrable evaluation domain and extended-real conventions of Definition [ftip-00JJ] apply: an empty feasible class has value $-\infty $, an unbounded-above feasible score set has value $+\infty $, and a finite supremum need not be attained. The allowed class is part of the comparison, so its envelope is a restriction-dependent quantity, not an unrestricted architectural ceiling.

Definition. Model-instance comparison evidence record [ftip-00JY]

A model-instance record contains architecture/checkpoint identity, parameter count, data and post-training recipe, optimizer, precision, hardware, cache policy, training and inference cost vectors, task suite, and independent evaluation seed. A missing field makes an architecture attribution unknown.

Remark. Systems optimizations are measured interventions [ftip-00JZ]

FlashAttention, KV-cache layouts, quantization, batching, and kernels can change feasible cost. If they approximate, truncate, or alter precision of the mathematical computation, the record must mark that as a systems intervention rather than as a pure architecture comparison.

Theorem. Matched-protocol difference is an estimand [ftip-00K0]

Given $n\geq 1$ evaluation draws, let $\widehat V_A(C)$ be the sample mean of finite real evaluation scores for architecture $A$ at a finite cost cap $C\geq 0$. Under a fixed evaluation law and common-recipe arm, the finite difference $\widehat V_A(C)-\widehat V_B(C)$ is a well-defined empirical estimand.

It does not identify a causal architecture effect when data, tuning, or systems coordinates differ. The conclusion follows directly from the declared record.

Remark. Conditions for applying a toy model [ftip-00K1]

A toy theorem applies to a concrete model only if an explicit map from its state and interface to the model preserves the theorem's assumptions, including its cost accounting. Without such a map, the conclusion concerns only the abstract model; its validity for the concrete system is unknown.

Kimi Delta Attention versus Transformer [ftip-00K2]

Definition. KDA model-instance record [ftip-00K3]

For a Kimi Delta Attention (KDA) versus full-attention baseline study, record the exact Kimi Linear checkpoint, layer mix, context length, hardware, kernel, precision, batch, and decoding workload. The paper describes KDA as a fine-grained gated delta-rule module in a hybrid architecture; those are source observations from Kimi Linear: An Expressive, Efficient Attention Architecture[kimi2025linear], not universal theorems.

If the baseline is called a Transformer, record whether it is the paper's MLA baseline or another full-attention implementation; the label alone does not identify a common architecture or cost.

Example. Matched-cost KDA and Transformer comparison [ftip-00K4]

A matched comparison uses the same task prompts, post-training data and feedback, optimizer family, evaluation protocol, and quality target. The corresponding measurements include parameter count, training FLOPs, inference FLOPs, wall time, memory, KV-cache bytes, and context length for each architecture. A reported speed or quality difference is conditional on these experimental conditions and measurements.

Remark. What the Kimi Linear paper establishes [ftip-00K5]

Kimi Linear reports a hybrid KDA/MLA model and fair-comparison experiments in Sections 3--5, with setup in Section 5.4 and efficiency comparisons in Sections 5.5--5.6 of Kimi Linear: An Expressive, Efficient Attention Architecture[kimi2025linear]. These include long-context efficiency.

The reported throughput and quality are empirical under its training recipe and hardware; they are lower bounds on demonstrated performance, not an architecture-independent ceiling.

Example. Parameter matching is not cost matching [ftip-00K6]

Two models with equal parameter count can have different attention FLOPs, KV-cache memory, kernel utilization, and attainable context. Conversely, equal wall time can hide different hardware and precision. Therefore a parameter-matched result cannot by itself identify a compute frontier.

Remark. Cache, throughput, and quality in KDA comparisons [ftip-00K7]

The KDA example does not prove that every linear-attention model beats every Transformer, that cache savings imply equal quality, or that a measured throughput gain is a capability gain. Any such statement requires a new matched study with the evidence record of Definition [ftip-00JY].

Definition. Model-instance comparison decision rule [ftip-00K8]

Call architecture A more cost-efficient than B at quality $q$ only when the compared threshold costs use the same declared protocol, the confidence procedure and stopping rule are fixed, and all cost-vector coordinates needed by the claim are present. A ratio additionally requires finite costs and a strictly positive denominator as in Definition [ftip-00JP]. State whether the costs are measured target-achieving witnesses or justified inverse-frontier values $C_A(q)$ and $C_B(q)$; observing one successful run does not identify those infima. Otherwise the comparison is descriptive or unknown.

Interface boundaries for alternative architectures [ftip-00K9]

Definition. JEPA comparability interface [ftip-00KA]

A JEPA-like architecture is comparable to an autoregressive model only after fixing the predictor, target representation, decoder or downstream interface, training objective, inference procedure, and evaluation law. A parameter-count match alone is not a typed comparison.

Remark. Representation and task interface are separate [ftip-00KB]

A latent predictor can be excellent under one decoder and unusable under another. The evaluation record must therefore include the interface that maps the representation to the declared task output.

Theorem. Interface-preserving architecture reduction [ftip-00KC]

If two architecture records induce the same conditional output law on a fixed task and evaluation interface, every evaluation functional $J$ gives the same value. This is an immediate pushforward identity under the stated output-law and interface assumptions. It does not imply that distinct architectures are generally equivalent.

Example. Equal benchmark score can hide different ceilings [ftip-00KD]

Two architectures may tie on a finite benchmark while differing on an unmeasured task slice or longer context. Thus a single score cannot establish equal representational ceilings; the task family and evaluation coverage must be declared.

Remark. Architecture-independent model interfaces [ftip-00KE]

A result stated solely in terms of a model/interface map applies to every architecture realizing that map and satisfying its hypotheses. An architecture-specific conclusion can require further assumptions, as in Remark [ftip-00JK].

Remark. Open model-instance questions [ftip-00KF]

The relative capability and cost of KDA, full attention, and JEPA candidates remain empirical questions. A reproducible comparison requires identified checkpoints or reproducible training, fixed evaluation seeds, and measured cost vectors. Architecture-specific ceilings remain estimands and hypotheses rather than established constants.

Published hybrid-attention comparison [ftip-00KU]

Kimi Linear combines recurrent KDA layers with periodic full MLA attention. Its published comparison shows task-dependent score changes and lower long-context decoding times. The selected scores in Example [ftip-00LC], the memory estimate in Example [ftip-00KY], and the discovery calculation in Example [ftip-00L3] connect these observations to a post-training question: when can a cheaper attempt compensate for a possible lower per-attempt success probability?

Example. Published Kimi Linear operating point [ftip-00KV]

The Kimi Linear report compares experimental models pretrained on 1.4 trillion tokens, each with 48 billion total and 3 billion active parameters. Kimi Linear uses three Kimi Delta Attention layers for each full Multi-head Latent Attention layer, with no positional encoding (NoPE) in its MLA layers; the reference uses full MLA attention [kimi2025linear, Sections 4 and 5.4]. These are properties of the reported hybrid architecture.

These experimental models are distinct from the released Kimi Linear model, pretrained on 5.7 trillion tokens and supporting up to one million tokens of context. Appendix D compares that release with Moonlight, which has 16 billion total and 3 billion active parameters [kimi2025linear, Section 5.4.1 and Appendix D].

Example. Published efficiency observations [ftip-00KW]

For batch size one at one million tokens, Figure 7 labels a $2.9\times $ prefill speedup and a $2.2\times $ decoding speedup for Kimi Linear against MLA. Section 6.3 gives $2.3\times $ for the latter point, so the graphic and prose differ slightly [kimi2025linear, Figure 7 and Sections 5.6 and 6.3].

The larger-batch comparison in Figure 1(b) gives decoding time per output token of 1.84 milliseconds for Kimi Linear and 11.48 milliseconds for MLA, with a reported $6.3\times $ speedup. Section 6.3 describes this as a theoretical speedup from reallocating saved KV-cache memory to larger batches. It illustrates a different use of the memory saving from the batch-one latency result [kimi2025linear, Figure 1(b) and Section 6.3].

Figure 1(a) reports RULER scores of 84.3 for Kimi Linear and 81.3 for MLA at 128k context, with $3.98\times $ decoding acceleration [kimi2025linear, Figure 1(a)]. The report also gives a reduction of up to 75 percent in KV-cache use [kimi2025linear, Abstract]. Together these observations motivate studying context length and batch size as separate cost coordinates. The unequal RULER scores and different timing regimes are retained when interpreting those points.

Definition. Quality target and cost measurement [ftip-00KX]

For a reported benchmark score q, use the same task, prompt law, scoring rule, and post-training condition for both systems. The comparison records the smallest measured cost at which each system reaches q, when such a measurement exists. A throughput or cache ratio is not substituted for this iso-quality cost.

Example. A cache-memory estimate for hybrid attention [ftip-00KY]

The cited FLA KDA implementation at commit 6b6f546f stores one active recurrent state per sequence, with dimensions determined by the value heads and key/value dimensions rather than accumulated context length. Its newly allocated state uses float32 [⧉]. The optional short-convolution state has one fixed-width kernel buffer per channel and sequence [⧉]. This supplies a concrete implementation model for the recurrent part of the 3:1 mixture in Example [ftip-00KV].

Consider a stylized comparison with $4\ell $ layers, batch size $b$, and context length $T$, where $\ell ,b,T$ are positive integers. Assume each MLA layer stores $\kappa T$ bytes per sequence, with the same $\kappa >0$ in both designs, and each KDA layer stores one terminal recurrent-plus-convolution state of $\sigma >0$ bytes. The active attention-state memories are

\[ M_{\rm MLA}=4b\ell \kappa T,\qquad M_{\rm hybrid}=b\ell (\kappa T+3\sigma ),\qquad \frac {M_{\rm hybrid}}{M_{\rm MLA}} =\frac {1}{4}+\frac {3\sigma }{4\kappa T}. \]

The hybrid uses less active state exactly when $T>\frac {\sigma }{\kappa }$, and the ratio tends to $1/4$ as context grows. This explains why replacing three quarters of the growing caches can approach a 75-percent saving, while fixed recurrent-state overhead can erase the advantage at short contexts. The retained MLA layers keep the hybrid's state memory linear in $T$; the whole model is not a fixed-memory recurrent system.

Weights, training activations, workspaces, allocator padding and saved prefix states are outside this active-state calculation. At a fixed device memory budget, the estimate predicts more room for simultaneous or longer rollouts when attention state dominates. It does not predict a fourfold speedup. Training and decoding also use different execution paths: FLA selects chunk kernels for gradient-enabled computation and fused recurrence for short inference calls [⧉]. The fixed-state estimate is therefore a useful serving hypothesis, not a training-memory or training-speed formula.

Example. Common-recipe reading of the Kimi result [ftip-00KZ]

The report holds its stated pretraining and SFT recipes common across the experimental models [kimi2025linear, Sections 5.4--5.5 and Table 4]. The selected observations in Example [ftip-00LC] compare the resulting trained designs, including the hybrid's 3:1 KDA/MLA pattern and NoPE choice. The mixed score directions suggest a task-dependent tradeoff rather than one ordering of the two architectures.

The theoretical mechanism is a change in the cost of maintaining and using the prefix: recurrent layers compress it into state, while the retained MLA layers preserve growing attention caches. The model in Example [ftip-00KY] predicts the largest memory advantage at long contexts. If this advantage reduces the cost of generating useful candidates, Example [ftip-00L3] predicts when more attempts can offset a reduction in per-attempt quality. This is a conditional explanation to test through rollout costs and verified success rates; it does not require attributing the whole result to KDA alone.

Example. Measured iso-quality cost ratio [ftip-00L0]

Let $c_K(q)$ and $c_T(q)$ be measured inference costs for Kimi Linear and the Transformer reference at the same finite real score $q$ and context length, under one accounting rule. If $0\leq c_K(q)<+\infty $ and $0<c_T(q)<+\infty $, the measured ratio $\widehat \rho _{K/T}(q)=c_K(q)/c_T(q)$ is reported together with the full cost vectors. These measured costs need not be the inverse frontier infima of Definition [ftip-00JP]. If either threshold is unmeasured or the denominator is zero, the ratio is left undefined rather than inferred from the paper's speedup plot.

Remark. Hybrid attention is not pure linear attention [ftip-00L1]

Kimi Linear combines KDA with periodic full-attention layers. A measured advantage therefore identifies the reported hybrid intervention, not a claim that a purely linear recurrent architecture has the same quality or ceiling. The layer pattern belongs in the architecture record.

Remark. Demonstrated performance is a lower bound [ftip-00L2]

The published benchmark scores demonstrate attainable points for the reported checkpoints and protocol. They do not establish a representational ceiling, a universal scaling law, or a claim that additional cost cannot improve either architecture.

Example. When cheaper attempts compensate for lower success [ftip-00L3]

Smaller caches can support more concurrent requests or avoid preemption and recomputation. These are actual serving mechanisms in vLLM's memory and scheduling guidance [⧉]. For post-training, the useful prediction is therefore about completed candidate attempts under a budget, including verification and optimization costs, rather than decoding speed alone.

Use the following cost model on the same hardware. One baseline attempt costs $c>0$; generation accounts for a fraction $f\in [0,1]$. Suppose generation becomes $s_{\rm gen}>0$ times as fast while all other cost per attempt stays fixed. The new cost and total attempt-throughput factor are

\[ c_K=c\left (1-f+\frac {f}{s_{\rm gen}}\right ),\qquad S=\frac {c}{c_K} =\frac {1}{1-f+f/s_{\rm gen}}. \]

For a fixed budget $B\geq 0$, this model permits $N_M=\lfloor B/c\rfloor $ baseline attempts and $N_K=\lfloor B/c_K\rfloor $ hybrid attempts. Assume $N_M,N_K\geq 1$ and verified-success events independent within each set of attempts, with respective probabilities $p_M,p_K\in [0,1]$. Then Theorem [ftip-0078] gives discovery probabilities $1-(1-p_M)^{N_M}$ and $1-(1-p_K)^{N_K}$. The hybrid matches or exceeds baseline discovery exactly when

\[ p_K\geq 1-(1-p_M)^{N_M/N_K}. \]

This follows by comparing the two failure probabilities and taking the nonnegative $N_K$-th root.

For an illustrative estimate, suppose generation consumes 80 percent of the baseline cost and use $s_{\rm gen}=2.2$, motivated by the batch-one decode figure in Example [ftip-00KW]. Applying that decode factor to the whole generation stage is a working approximation, not a measured rollout result. Then $S\approx 1.774$; a budget of $B=10c$ buys 10 baseline attempts or 17 hybrid attempts. If $p_M=0.10$ and $p_K=0.08$, the discovery probabilities are approximately $0.651$ and $0.758$. The break-even hybrid probability is about $0.0601$. Thus this model predicts that more attempts can outweigh a moderate per-attempt quality loss. The assumed probabilities are illustrative verified-success rates, not conversions of the benchmark scores in Example [ftip-00LC].

The prediction is strongest for long-context, generation-heavy work whose saved memory becomes usable rollout capacity. As verification or optimization dominates, $f$ shrinks and $S$ approaches one. Correlated attempts require the joint-law analysis of Convention [ftip-0077] instead of the independent product. Measure completed attempts, verified-success rates and total cost under a fixed prompt, decoding and verifier protocol to test the tradeoff. In the RLVR round of Example [ftip-004M], additional successful candidates can improve the available feedback; whether they produce useful accepted updates is a further optimizer-and-evaluation question, as Example [ftip-0085] demonstrates.

Paired measurements for model comparison [ftip-00L4]

A paired measurement specifies both model instances and their task, intervention, evaluation, and cost coordinates. These conditions determine the meaning of each point in the measured frontier.

Definition. Comparison pair identity [ftip-00L5]

A paired comparison identifies the two systems, exact checkpoint or training commit, architecture variant, parameter counts, tokenizer, and context limit. A label such as ``Transformer'' is insufficient when attention, cache, or layer patterns differ.

Definition. Common task and evaluation conditions [ftip-00L6]

Both systems use the same task family, prompt and data law, scoring rule, sampling seeds, stopping rule, and evaluator version. Any exception is a separate comparison arm, not an unrecorded architecture effect.

Definition. Training and post-training coordinates [ftip-00L7]

Record data volume, token order policy, optimizer and schedule, supervised tuning, feedback or reinforcement procedure, update count, and selection rule. The common-recipe arm fixes these coordinates; the equal-tuning arm allows predeclared alternatives with equal search resources.

Definition. Inference and systems coordinates [ftip-00L8]

Record hardware, software and kernel versions, precision, batch, cache policy, sequence lengths, warm-up procedure, concurrency, and decoding settings. A kernel or cache change that alters numerical computation is marked as an intervention rather than hidden as an implementation detail.

Definition. Per-system cost vector [ftip-00L9]

For each system, the cost vector $\mathbf c$ contains training FLOPs, inference FLOPs, wall time, memory, energy when available, and context length. The primary scalar $C$ and its weights are declared beside the vector; omitted coordinates are marked unknown.

Definition. Repeated measurement rule [ftip-00LA]

For each fixed workload and system, record warm-up runs, repetition count, random seeds, and the aggregation rule. Report mean and dispersion for time, memory, and score; a single fastest run cannot define a cost frontier.

Definition. Paired uncertainty estimates [ftip-00LB]

When the same evaluation items are used, report paired score differences and an uncertainty procedure fixed before inspecting the result. Confidence intervals quantify sampling variation; they do not repair unmatched training, systems, or task coordinates.

Example. Published Kimi Linear comparison [ftip-00LC]

Table 4 of the Kimi Linear report compares the experimental models pretrained on 1.4 trillion tokens in Example [ftip-00KV], after the same supervised fine-tuning (SFT) recipe. Both have 48 billion total and 3 billion active parameters. Three selected benchmark scores are [kimi2025linear, Sections 4 and 5.4 and Table 4]:

Arm	MMLU-Pro	LiveBench	EvalPlus
Kimi Linear	67.4	45.2	61.0
Full MLA	65.7	45.7	62.6

The entries use the paper's score units, with larger values better on each selected benchmark; LiveBench is reported as Pass@1. Kimi Linear is higher on MMLU-Pro and lower on LiveBench and EvalPlus in these observations. These are reported point estimates on different tasks, not a single capability score; the table supplies no uncertainty estimate.

The common recipe uses the K2 pretraining corpus, MuonClip and a shared training schedule. SFT proceeds from broad instruction data to targeted reasoning tasks; evaluation uses temperature 1.0 and an internal framework derived from LM-Harness [kimi2025linear, Section 5.4]. Thus the scores provide a concrete example of how the trained hybrid and full-MLA designs respond to the same recipe. The cost mechanism and its possible post-training consequences are developed in Example [ftip-00KZ].

Remark. Measured and assumed costs [ftip-00LD]

A useful cost model can combine measurements with assumptions justified by the architecture and its implementation. In Example [ftip-00L3], the generation share $f$ and speed factor $s_{\rm gen}$ determine a predicted attempt-throughput factor. Their assumed values can be varied without pretending they were measured in the original experiment.

For fixed $f\in [0,1]$, write $S(s)=1/(1-f+f/s)$ for $s>0$. This function is nondecreasing. An interval $s_{\rm gen}\in [s_-,s_+]$, with $0<s_-\leq s_+$, therefore gives the conditional range $S\in [S(s_-),S(s_+)]$. This is sensitivity to a chosen assumption range; it is a statistical confidence interval only if a sampling argument supplies that interpretation.

Definition. Measured contrasts and conditional predictions [ftip-00LE]

A measured paired contrast evaluates the same declared quantity for two observed model instances under a specified task, protocol and cost accounting. The common-recipe scores in Example [ftip-00LC] are one example. A confirmatory comparison fixes its matching and selection rules before examining the scores.

A conditional paired prediction instead uses an explicit model to supply one or more cost or response quantities, then derives the comparison under those assumptions. The discovery probabilities in Example [ftip-00L3] are an example. Such a prediction can be motivated by existing observations and guide a later experiment; its unmeasured inputs remain assumptions, and agreement with new matched measurements tests the model.

Example. How generation share changes the predicted gain [ftip-00LF]

Keep the illustrative generation factor $s_{\rm gen}=2.2$ from Example [ftip-00L3] and vary the baseline generation share $f$. Its cost model gives the following total attempt-throughput factors:

Generation share	Predicted throughput factor
20 percent	1.122
50 percent	1.375
80 percent	1.774
100 percent	2.200

These calculated values show why the same decoding improvement can matter much more for generation-heavy search than for a verifier- or optimizer-dominated workload. They suggest measuring the time spent in generation, verification and updates before selecting where to invest systems effort. A change in the bottleneck changes the useful intervention, even when the attention architecture stays fixed.

Remark. Testing the predicted architecture tradeoff [ftip-00LG]

The memory and discovery models predict the strongest benefit when long prefixes make attention state expensive, saved capacity improves useful rollout throughput, and per-attempt success remains above the break-even threshold. Short contexts, expensive verification, or strongly correlated attempts can weaken that benefit. These are distinct mechanisms to test, rather than a universal ranking of KDA and full attention.

A comparison across implementations can retain this reasoning while re-estimating state sizes, generation share and success probabilities for the new system. Report both the prediction and the matched measurements: their agreement or disagreement identifies which mechanism or assumption needs revision.

Conditional frontier and saturation analysis [ftip-00LH]

A finite matched study measures capability at selected values of one declared cost. Its frontier and saturation target are conditional on that protocol; they do not establish a universal intelligence law.

Definition. Finite cost grid [ftip-00LI]

Choose a finite ordered grid $0<C_1<\cdots <C_m$ and evaluate each architecture at every declared grid point under the same task and protocol. The resulting values $\widehat V_A(C_i)$ are observations of the restricted frontier, not its values between grid points.

Definition. Observed upper envelope [ftip-00LJ]

For an observed score $s_A(C_i)$ define the discrete upper envelope $U_A(C_i)=\max _{j\leq i}s_A(C_j)$. It is a descriptive monotone summary of the measured points; it is not evidence that unmeasured costs attain the envelope or that extra cost cannot reduce score.

Definition. Uncertainty bands on the frontier [ftip-00LK]

Attach a predeclared uncertainty interval $I_A(C_i)$ to each score and carry those intervals through score differences and threshold crossings. A finite band describes sampling and measurement variation; it does not cover unmeasured training procedures or architectures.

Definition. Declared saturation target [ftip-00LL]

Fix a quality target $q\in \mathbb R$, tolerance and cost increment $\varepsilon ,\delta \in (0,+\infty )$, and a budget $C\in \mathbb R_{\geq 0}$. Require both $V_A(C)$ and $V_A(C+\delta )$ to be finite real values. A study calls an architecture $\varepsilon $-saturated at $C$ only relative to its allowed intervention class when the restricted frontier has certified gain $V_A(C+\delta )-V_A(C)\leq \varepsilon $.

Theorem. Conditional saturation certificate [ftip-00LM]

Take finite real $U$, $C\geq 0$, $\delta \geq 0$, and $\varepsilon \geq 0$, with the nondecreasing frontier of Definition [ftip-00JJ]. If a justified upper bound gives $V_A(C+\delta )\leq U$ and a justified lower bound gives $V_A(C)\geq U-\varepsilon $, then $U-\varepsilon \leq V_A(C)\leq V_A(C+\delta )\leq U$. Both endpoint values are therefore finite real, and $0\leq V_A(C+\delta )-V_A(C)\leq \varepsilon $. This is a conditional bound for the named frontier; it proves no universal saturation of intelligence.

For the finite controller class with hard resource admission, Corollary [ftip-00MC] obtains the required bounds from a feasible controller and a uniform Bellman certificate.

Example. Finite observations need not identify a frontier [ftip-00LN]

Let $S\subset \mathbb R_{\geq 0}$ be a finite set of measured costs, and choose $C_0>0$ larger than every element of $S$. Suppose the exact frontier value observed at each cost in $S$ is zero. For finite budgets $C\geq 0$, suppose the declared class of possible frontiers permits both \[ V_0(C)=0, \qquad V_1(C)= \begin {cases} 0,&0\leq C<C_0,\\ 1,&C\geq C_0. \end {cases} \] These nondecreasing frontiers with scores in $[0,1]$ agree on every observed cost and differ at $C_0$. The observations alone do not distinguish them.

Both possibilities have finite realizations in the framework of Definition [ftip-00JJ]: allow two interventions with costs $0$ and $C_0$. Give the first score zero and the second score $\theta \in \{0,1\}$. For example, on a single deterministic evaluation task with utility equal to the output bit, let the two resulting protocols return $0$ and $\theta $. The two possible choices of $\theta $ give $V_0$ and $V_1$, respectively, under the same intervention and cost specification. Every score is finite, and the feasible score maximum is attained at each budget.

This example does not supply a strictly better agreeing frontier for every possible data set or every admissible class. If a proved global score bound is $1$ and an intervention attains it at a finite cost $C_*\geq 0$, monotonicity forces $V_A(C)=1$ for every $C\geq C_*$; no higher value is admissible. A singleton class of possible frontiers can also identify the frontier without such an alternative. A conditional certificate such as Theorem [ftip-00LM] therefore requires its stated upper-bound evidence; that evidence does not follow merely from the absence of an observed improvement.

Definition. One-factor cost interpretation [ftip-00LO]

The primary cost $C$ is a declared scalarization of the recorded cost vector. A frontier statement is conditional on its units and weights; the same paired measurements may produce a different ordering under a different scalarization.

Example. Crossing architecture frontiers [ftip-00LP]

Two architectures may alternate in the observed ordering across costs: one can score higher at small $C$ while the other catches up at larger $C$. Thus a single comparison point cannot establish a global ordering or a common ceiling.

Example. Frontier report for the KDA study [ftip-00LQ]

A Kimi Delta Attention versus Transformer study should publish the cost grid, paired scores, uncertainty intervals, scalarization weights, and the exact identities and configurations of the compared models. The report may then state which measured points are Pareto or iso-quality comparisons under that protocol.

Remark. Frontiers depend on the intervention and evaluation [ftip-00LR]

The conditional frontier is indexed by architecture, intervention class, task family, evaluation law, and cost scalarization. Changing any of these coordinates creates a new estimand; no ordering transfers automatically to a new checkpoint, optimizer, hardware stack, or task family.

Study design for comparisons across model instances [ftip-00LS]

This subsection specifies how a conditional architecture comparison may be instantiated without changing the architecture-neutral formulation. It is a study design, not a universal ranking of architectures.

Definition. Matched model-instance record [ftip-00LT]

A model-instance record names the architecture, parameterization or checkpoint, training and post-training recipe, optimizer, execution stack, inference budget, task family, evaluation law, and recorded cost vector. Two records are comparable only after the fields held fixed and the fields allowed to vary are declared.

Definition. Pair identity and comparison unit [ftip-00LU]

A comparison unit is a pair of model-instance records evaluated on the same task family and evaluation law, together with a declared cost scalarization. A Transformer reference and a Kimi Delta Attention instance may form such a pair only when the remaining coordinates are documented.

Remark. Paired responses to a common recipe [ftip-00LV]

For a pair of matched model-instance records, the common-recipe arm measures how the permitted architecture packages respond to one recipe. A score advantage in this comparison need not persist after either package is retuned.

Remark. Different recipes under equal tuning effort [ftip-00LW]

Under the equal-tuning-budget arm, the two selected model instances may use different recipes. That difference is compatible with equal predeclared tuning effort; forcing the selected recipes to match would answer a different comparison question.

Remark. How an allowed class changes its envelope [ftip-00LX]

In the restricted-envelope arm, enlarging an allowed intervention class at fixed evaluation law, cost accounting, and budget can raise its envelope and cannot lower it. A finite search over feasible interventions supplies a lower bound from the scores it attains. It certifies the supremum only if it covers every feasible intervention or is accompanied by a matching upper-bound argument.

Remark. Coordinate-change rule [ftip-00LY]

Changing the optimizer, post-training method, hardware or kernel, precision, context policy, or evaluation law changes a comparison coordinate. A score difference after such a change belongs to the new intervention, unless the study explicitly estimates the interaction.

Definition. Architecture-family transfer matrix [ftip-00LZ]

A transfer matrix records which interfaces and budgets permit a comparison among a Transformer reference, Kimi Delta Attention, looped or depth-reused variants, and JEPA-like systems. A blank or incompatible cell is an incomparability finding, not a missing score to be imputed.

Remark. Applying a finite model to an architecture [ftip-00M0]

A toy representation or recurrence result may motivate a hypothesis, but it is not transferred to a model instance until its interface, task, cost accounting, numerical regime, and evaluation protocol are instantiated and checked. If any required assumption is unverified, applicability to the model instance remains unestablished.

Definition. Negative-result and stopping report [ftip-00M1]

A completed comparison reports the tested cost grid, excluded configurations, stopping rule, uncertainty, and negative results. Stopping without a detected difference is evidence about the declared study, not proof that the architectures have equal potential.

Remark. Open comparison questions [ftip-00M2]

The remaining questions are whether a declared interface supports a fair Kimi Delta Attention--Transformer comparison, how looped computation changes the frontier under matched budgets, and which JEPA interface can be fixed without silently changing the task. None is settled by the study design alone.

Optimizer geometry and representation dependence [ftip-00AU]

A scalar objective does not by itself determine a training intervention. This section studies an exact finite-dimensional witness: one represented matrix can have many factor bases, the loss can be constant along that gauge orbit, and an update rule can nevertheless distinguish the bases.

The principal source is the version-one arXiv preprint The loss does not see the basis, but Adam does[singh2026lossbasis]. Its algebraic results assume the stated dimensions, rank conditions, state conventions, and real arithmetic. Experimental observations concern the reported configurations.

Factored objectives and gauge symmetry [ftip-00AV]

We first separate a represented matrix from a choice of factors and then separate its full product-preserving symmetry from the orthogonal subgroup that preserves the Euclidean parameter metric.

Notation. Factored objective and represented matrix [ftip-00AW]

Fix integers $d_1,d_2,k\geq 1$. Let $U\in \mathbb R^{d_1\times k}$ and $V\in \mathbb R^{d_2\times k}$ be factors, let

\[ W(U,V)=UV^{\mathsf T}\in \mathbb R^{d_1\times d_2}, \]

and let $f:\mathbb R^{d_1\times d_2}\to \mathbb R$ be differentiable. The corresponding factored objective is $L(U,V)=f(W(U,V))$. The pair $(U,V)$ is a parameterization; $W(U,V)$ is the represented object on which $f$ depends. This is the rectangular form noted in Section 3 of The loss does not see the basis, but Adam does[singh2026lossbasis].

Definition. General linear gauge action [ftip-00AX]

For $A\in \mathrm {GL}(k)$, define the general linear gauge action by

\[ \gamma _A(U,V)=(UA,VA^{-\mathsf T}). \]

The inverse transpose on the second factor is essential for a general invertible $A$. When $A=Q\in \mathrm O(k)$, one has $Q^{-\mathsf T}=Q$, and the action becomes $(UQ,VQ)$.

Lemma. The gauge action preserves the product and loss [ftip-00AY]

For every $A\in \mathrm {GL}(k)$,

\[ W(UA,VA^{-\mathsf T})=UV^{\mathsf T} \quad \hbox {and}\quad L(UA,VA^{-\mathsf T})=L(U,V). \]

Proof.

Since $(VA^{-\mathsf T})^{\mathsf T}=A^{-1}V^{\mathsf T}$, the first identity is $UAA^{-1}V^{\mathsf T}=UV^{\mathsf T}$. Applying $f$ gives the second.

This is the factored-model calculation in Section 3 of The loss does not see the basis, but Adam does[singh2026lossbasis].

Definition. Orthogonal gauge orbit [ftip-00AZ]

The orthogonal gauge orbit of $(U,V)$ is

\[ \mathcal O(U,V)=\{(UQ,VQ):Q\in \mathrm O(k)\}. \]

Every pair in the orbit represents the same $W$. This orbit is generally smaller than the complete fibre of the map $(U,V)\mapsto UV^{\mathsf T}$. The restriction to $\mathrm O(k)$ is geometric: it preserves the Euclidean metric on factor space.

Lemma. The orthogonal gauge is the maximal isometric subgroup [ftip-00B0]

The action $\gamma _A$ preserves $\lVert U\rVert _F^2+\lVert V\rVert _F^2$ for every pair $(U,V)$ if and only if $A\in \mathrm O(k)$.

Proof.

If the action is an isometry, take $V=0$ and let one row of $U$ be an arbitrary $u\in \mathbb R^{1\times k}$. Then $\lVert uA\rVert _2=\lVert u\rVert _2$ for every $u$, hence $AA^{\mathsf T}=I$. Conversely, right multiplication by an orthogonal matrix preserves both Frobenius norms.

See Lemma 3.2 and its proof in Appendix B.1 of The loss does not see the basis, but Adam does[singh2026lossbasis].

Lemma. Gradients transform covariantly along an orthogonal orbit [ftip-00B1]

For $Q\in \mathrm O(k)$,

\[ \nabla _U L(UQ,VQ)=\nabla _U L(U,V)Q, \qquad \nabla _V L(UQ,VQ)=\nabla _V L(U,V)Q. \]

Proof.

Put $G=\nabla f(W)$. Then $\nabla _U L=GV$ and $\nabla _V L=G^{\mathsf T}U$. The represented matrix, and hence $G$, is unchanged along the orbit. Substitution gives both identities.

See Lemma 4.1 and its proof in Appendix B.2 of The loss does not see the basis, but Adam does[singh2026lossbasis].

Definition. Optimizer state action [ftip-00B2]

Use the optimizer-state convention of Definition [ftip-001H]. For every $Q\in \mathrm O(k)$, an optimizer state action is a map $\sigma _Q:\mathcal S_{\rm opt}\to \mathcal S_{\rm opt}$ compatible with the shapes of the factor states and satisfying $\sigma _Q(s_0)=s_0$ for the declared initial state.

For example, a momentum buffer shaped like $U$ transforms by right multiplication by $Q$; a shared scalar second moment is fixed. An entrywise second-moment array need not admit a state action compatible with every orthogonal $Q$.

Definition. Gauge-equivariant optimizer [ftip-00B3]

Let $\mathcal A$ be a deterministic instance of the parameter-update rule of Definition [ftip-001I], acting on $(U,V,s)$. It is gauge-equivariant when for every $Q\in \mathrm O(k)$ there is a state action $\sigma _Q$ such that gauge-related initial states produce

\[ (\widetilde U_t,\widetilde V_t,\widetilde s_t) =(U_tQ,V_tQ,\sigma _Q(s_t)) \qquad (t\geq 0). \]

This is Definition 3.1 of The loss does not see the basis, but Adam does[singh2026lossbasis]. Any randomness, schedules, stopping rule, and mixed update blocks must also be coupled as required by Remark [ftip-001J]; the definition does not hide them.

Example. A commuting optimizer square [ftip-00B4]

One update of a gauge-equivariant optimizer makes the following square commute. The vertical arrows change factor basis and the horizontal arrows apply the same declared update.

$\text{[math]}$

Commutation is stronger than equality of the scalar losses before the step. It states equality of the represented product trajectory after aligning the parameter bases.

Remark. Product symmetry is larger than the isometric gauge [ftip-00B5]

The full $\mathrm {GL}(k)$ action in Definition [ftip-00AX] preserves $W$, but ordinary Euclidean gradients are covariant under the orthogonal specialization used in Lemma [ftip-00B1]. A theorem for $(UQ,VQ)$ must therefore not be silently promoted to every product-preserving reparameterization.

The source calls the orthogonal subgroup the gauge in its optimizer classification while also noting the larger function-preserving group. These are compatible statements only when their different geometric scopes remain explicit.

Equivariant and basis-sensitive update rules [ftip-00B6]

Gradient covariance supplies a test for update rules. Linear and full-matrix operations can commute with a latent rotation, whereas a fixed nonlinear map applied separately to coordinates selects a preferred basis.

Definition. Memoryless right-equivariant update [ftip-00B7]

A memoryless factor-update map is a function $\Phi :\mathbb R^{n\times k}\to \mathbb R^{n\times k}$ applied to a current gradient $G$ without a persistent optimizer state. It is right-equivariant when

\[ \Phi (GQ)=\Phi (G)Q \qquad (G\in \mathbb R^{n\times k},\ Q\in \mathrm O(k)). \]

This local definition is the input of Theorem 4.5 in The loss does not see the basis, but Adam does[singh2026lossbasis]. It does not classify stateful optimizers such as Adam.

proposition. Gradient descent and momentum preserve the orthogonal gauge [ftip-00B8]

Gradient descent and Polyak or Nesterov momentum are gauge-equivariant when their factor gradients, momentum buffers, schedules, and look-ahead points are transformed consistently.

Proof.

By Lemma [ftip-00B1], $G_t$ becomes $G_tQ$. Gradient descent therefore sends $U_tQ$ to $(U_t-\eta _tG_t)Q$. A momentum buffer initialized at zero and formed by linear combinations of covariant gradients transforms as $M_tQ$; a Nesterov look-ahead point is likewise the reference look-ahead right-multiplied by $Q$. Induction proves the claim.

Proposition 4.2(1) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis].

proposition. A shared scalar second moment preserves the orthogonal gauge [ftip-00B9]

Replace Adam's entrywise second-moment denominator by one scalar whose updates depend only on gauge-invariant quantities such as the pooled mean of the squared entries of both factor gradients. With a covariant first-moment buffer, the resulting stateful update is gauge-equivariant.

Proof.

The first moment is a linear exponential average and transforms by right multiplication by $Q$. The pooled squared-entry mean is the joint squared Frobenius norm divided by the entry count, hence is unchanged by $G\mapsto GQ$. Its bias correction and positive scalar denominator are unchanged, so the complete update transforms covariantly.

Proposition 4.2(2) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis]. The result concerns this declared shared-scalar variant, not stock Adam.

proposition. Full-matrix Muon and Shampoo preserve the orthogonal gauge [ftip-00BA]

Fix one factor, write $n\in \{d_1,d_2\}$ for its row dimension, and let $G_t,M_t\in \mathbb R^{n\times k}$ be its gradient and Muon momentum buffer. For $0\leq \beta <1$, $M_{-1}=0$, and the source's linear momentum update $M_t=\beta M_{t-1}+G_t$, Muon's direction is $\Delta _t=\operatorname {msign}(M_t)$. Under real arithmetic this stateful update and the source's damped Shampoo update are right-equivariant. The Muon claim includes the stated finite Newton--Schulz approximations; the Shampoo claim uses its complete left and right accumulators.

Proof.

By Lemma [ftip-00B1], the primed gradient is $G'_t=G_tQ$. Starting from $M'_{-1}=M_{-1}Q=0$, induction through the linear momentum update gives $M'_t=M_tQ$. If $M_t=A\Sigma B^{\mathsf T}$ is a compact singular-value decomposition, then $M_tQ=A\Sigma (Q^{\mathsf T}B)^{\mathsf T}$, so $\operatorname {msign}(M_tQ)=\operatorname {msign}(M_t)Q$. For $\epsilon _{\rm NS}>0$, the Newton--Schulz initialization $X_0=M_t/(\lVert M_t\rVert _F+\epsilon _{\rm NS})$ is covariant because its Frobenius normalization is invariant; each polynomial iterate formed from $X_j$ and $X_jX_j^{\mathsf T}$ preserves the same covariance. For Shampoo, the left accumulator is invariant while the right accumulator transforms by $R_t\mapsto Q^{\mathsf T} R_t Q$. Orthogonal functional calculus conjugates the right matrix function, so the transformed update is the original update right-multiplied by $Q$.

Proposition 4.2(3)--(4) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis]. Variable splitting, coordinatewise clipping, mixed update rules, finite precision, and different damping conventions lie outside this statement. The Muon part is stateful; it does not replace the momentum buffer by the current gradient or invoke the memoryless classification of Definition [ftip-00B7].

proposition. Coordinatewise equivariance forces linearity [ftip-00BB]

Let $k\geq 2$ and let a memoryless update have the fixed entrywise form $\Phi (G)_{ij}=\phi (G_{ij})$. If $\Phi (GQ)=\Phi (G)Q$ for every $G$ and $Q\in \mathrm O(k)$, then $\phi (x)=cx$ for some $c\in \mathbb R$.

Proof.

It suffices to use one row and rotations in the first two coordinates. Equivariance at zero gives $\phi (0)=0$. Rotating $(x,0)$ through an angle whose cosine is $a\in [-1,1]$ gives $\phi (ax)=a\phi (x)$. For nonzero $x,y$, compare each with a third number of at least their absolute magnitudes; the ratio $\phi (x)/x$ is constant.

Proposition 4.3 and its proof are in Appendix B.4 of The loss does not see the basis, but Adam does[singh2026lossbasis]. No continuity assumption is needed.

Corollary. Fixed nonlinear coordinatewise updates break the gauge [ftip-00BC]

A nonlinear fixed entrywise update applied at a zero optimizer state is not gauge-equivariant when $k\geq 2$. In particular, the source applies proposition [ftip-00BB] to the first-step maps of Adam, RMSProp, signSGD, and Lion; it checks Adafactor separately because its factored statistics are not an entrywise map.

This is a first-step obstruction. It suffices to disprove equivariance of the full run, but it does not classify every later state or say that such an optimizer cannot reach a particular solution.

Definition. First-step Adam map and gauge defect [ftip-00BD]

Fix $\epsilon >0$. For zero-state, bias-corrected Adam define the entrywise first-step direction

\[ D_\epsilon (G)=G\mathbin {\oslash }(|G|+\epsilon ) \]

and, for $Q\in \mathrm O(k)$, define its gauge-aligned defect by

\[ E_Q(G)=D_\epsilon (GQ)Q^{\mathsf T}-D_\epsilon (G). \]

Here $|G|$, addition, and division are entrywise. Bias correction makes the first direction independent of Adam's two decay coefficients under the declared convention [singh2026lossbasis, Proposition 4.4].

proposition. Exact represented-matrix defect after one Adam step [ftip-00BE]

Let $G_U=\nabla _UL(U_0,V_0)$, $G_V=\nabla _VL(U_0,V_0)$, $D_U=D_\epsilon (G_U)$, and $D_V=D_\epsilon (G_V)$. Let $W_1$ be the product after one Adam step from $(U_0,V_0)$ and $\widetilde W_1$ the product after one step from $(U_0Q,V_0Q)$. Then

\[ \begin {aligned} \widetilde W_1-W_1={}&-\eta \bigl (E_Q(G_U)V_0^{\mathsf T} +U_0E_Q(G_V)^{\mathsf T}\bigr )\\ &+\eta ^2\bigl (E_Q(G_U)D_V^{\mathsf T} +D_UE_Q(G_V)^{\mathsf T} +E_Q(G_U)E_Q(G_V)^{\mathsf T}\bigr ). \end {aligned} \]

Proof.

Gauge-align the rotated factors back by $Q^{\mathsf T}$. They are $U_0-\eta (D_U+E_Q(G_U))$ and $V_0-\eta (D_V+E_Q(G_V))$. Multiply, subtract $(U_0-\eta D_U)(V_0-\eta D_V)^{\mathsf T}$, and collect powers of $\eta $.

Proposition 4.4 and its proof are in Appendix B.5 of The loss does not see the basis, but Adam does[singh2026lossbasis].

Example. A two-coordinate Adam defect witness [ftip-00BF]

Take $d_1=d_2=1$, $k=2$, $f(w)=w^2/2$, $U_0=V_0=(1,0)$, and

\[ Q=2^{-1/2}\begin {pmatrix}1&1\\-1&1\end {pmatrix}. \]

The original and rotated products after one zero-state Adam step are

\[ W_1=\left (1-\frac {\eta }{1+\epsilon }\right )^2, \qquad \widetilde W_1=\left (1-\frac {\eta }{2^{-1/2}+\epsilon }\right )^2. \] $\text{[math]}$

They differ when $0<\eta <2^{-1/2}+\epsilon $. This is the explicit witness in Proposition 4.4 of The loss does not see the basis, but Adam does[singh2026lossbasis]; it is a statement about one factored quadratic, not about language-model performance.

Example. A rotated sign update fails to commute [ftip-00BG]

For $G=(1,1)$ and the 45-degree rotation matrix $Q$ of Example [ftip-00BF], one has $GQ=(0,\sqrt 2)$. With $\operatorname {sign}(0)=0$,

\[ \operatorname {sign}(GQ)=(0,1), \qquad \operatorname {sign}(G)Q=(0,\sqrt 2). \]

Thus even this two-coordinate update violates the commuting equation in Definition [ftip-00B7]. The calculation is the explicit Appendix B.4 witness of The loss does not see the basis, but Adam does[singh2026lossbasis].

Theorem. Full-rank memoryless equivariant rules are Gram-determined [ftip-00BH]

Let $k\leq n$, and let $\Phi $ be defined on full-column-rank matrices $G\in \mathbb R^{n\times k}$. Then $\Phi (GQ)=\Phi (G)Q$ for every $Q\in \mathrm O(k)$ if and only if there is a matrix-valued function $H$ of $GG^{\mathsf T}$ such that

\[ \Phi (G)=H(GG^{\mathsf T})G. \]

Proof.

The displayed form is immediately equivariant. Let $G^+$ denote the Moore--Penrose pseudoinverse. Conversely, set $X(G)=\Phi (G)G^+$. Then $X(G)G=\Phi (G)$ and $X(GQ)=X(G)$. Two full-column-rank matrices with the same left Gram matrix have the same range and differ by a right orthogonal factor. Hence $X(G)$ depends only on $GG^{\mathsf T}$; take $H=X$.

See Theorem 4.5 and its proof in Appendix B.6 of The loss does not see the basis, but Adam does[singh2026lossbasis]. It classifies the full-rank stratum only.

Corollary. The square invertible preconditioner representation [ftip-00BI]

In the specialization $k=n$ with $G$ invertible, let $P^{1/2}$ denote the unique positive-definite square root of $P$. The function in Theorem [ftip-00BH] has the unique canonical representative

\[ H(P)=\Phi (P^{1/2})P^{-1/2}, \qquad P=GG^{\mathsf T}>0. \]

Proof.

The polar decomposition writes $G=P^{1/2}Q$. Equivariance gives $\Phi (G)=\Phi (P^{1/2})Q=H(P)G$. Evaluating at $G=P^{1/2}$ proves uniqueness in this square invertible case.

For rectangular $G$, the action of $H(P)$ away from the range needed to multiply $G$ is not determined by $\Phi $. The proof above establishes uniqueness in the square invertible specialization.

Remark. What the structure theorem leaves unresolved at rank deficiency [ftip-00BJ]

In the ambient matrix space, full-column-rank matrices form an open dense subset whose complement has Lebesgue measure zero. A probability-one claim requires the random gradient law to be absolutely continuous and not confined to a lower-rank set; random initialization alone does not supply that premise.

Theorem [ftip-00BH] therefore makes no classification claim on a general rank-deficient stratum. Equivariance still constrains values within each orthogonal orbit, and it forces $\Phi (0)=0$, but it does not determine the rule there by continuity unless continuity is separately assumed. The source's generic-rank discussion therefore does not establish a classification at rank deficiency.

Common-scalar flows, clock changes, and momentum [ftip-00BK]

One narrow class of continuous-time preconditioners follows the gradient-flow path at a different speed. The clock change transfers path properties, but not rates, and it does not encompass a stateful momentum method.

Definition. Common-scalar preconditioned flow [ftip-00BL]

Let $\theta =(U,V)$ and let $a(t)>0$ be measurable with $1/a$ locally integrable. A common-scalar preconditioned flow is a locally absolutely continuous trajectory satisfying

\[ \dot \theta (t)=-\frac {\nabla L(\theta (t))}{a(t)} \]

for almost every $t$. The same scalar multiplies every coordinate of both factor gradients. For gauge-uniform conclusions, the source additionally requires $a(t)$ to be determined by gauge-invariant statistics of the trajectory up to time $t$.

Definition. Effective optimizer clock [ftip-00BM]

For the flow of Definition [ftip-00BL], define the effective optimizer clock

\[ \tau (t)=\int _0^t\frac {du}{a(u)}, \qquad \tau _{\max }=\int _0^\infty \frac {du}{a(u)}, \qquad 0<\tau _{\max }\leq \infty . \]

The clock is strictly increasing on its domain. If $\tau _{\max }=\infty $, it reaches every gradient-flow time; otherwise it traverses only the prefix with effective time below $\tau _{\max }$.

Theorem. Common-scalar flow is a time-reparameterized gradient flow [ftip-00BN]

Let $\theta (t)$ satisfy Definition [ftip-00BL], let $t(\tau )$ be the inverse of the clock in Definition [ftip-00BM], and put $\widetilde \theta (\tau )=\theta (t(\tau ))$. Then, for almost every $\tau $,

\[ \widetilde \theta '(\tau )=-\nabla L(\widetilde \theta (\tau )). \]

Proof.

The chain rule for absolutely continuous changes of variable gives $dt/d\tau =a(t)$ almost everywhere. Multiplying $\dot \theta =-\nabla L/a(t)$ by $dt/d\tau $ yields the displayed gradient flow.

See Theorem 4.6 and its proof in Appendix B.9 of The loss does not see the basis, but Adam does[singh2026lossbasis]. Gauge invariance of $a$ is not needed for this single-trajectory clock identity; it makes the clock common across a gauge orbit.

Corollary. Transfer of path and limit properties [ftip-00BO]

Every property depending only on the portion of the gradient-flow path traversed before $\tau _{\max }$ transfers to the common-scalar flow. If $\tau _{\max }=\infty $ and gradient flow converges, both flows have the same limit point. A sufficient condition for clock divergence is an eventual finite upper bound on $a(t)$.

For shallow factorization, the source cites Implicit regularization in matrix factorization[gunasekar2017implicit]. For deep factorization, it cites Implicit regularization in deep matrix factorization[arora2019implicit]. For greedy low-rank dynamics, it cites Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning[li2021greedy].

Those conclusions transfer only when both this clock theorem and every hypothesis of the original result hold. This corollary supplies no missing matrix-sensing assumption.

Remark. Rates and hitting times do not survive time reparameterization [ftip-00BP]

Two flows can traverse the same curve with arbitrarily different physical clocks. A convergence rate, finite-step budget, or hitting time stated in $t$ therefore does not transfer through Theorem [ftip-00BN] without quantitative bounds on $a$ and its integral. If $\tau _{\max }<\infty $, even the tail and limit of the gradient-flow path are not reached.

Remark. Scalar-Adam is outside the common-scalar flow theorem [ftip-00BQ]

The shared-scalar Adam variant in proposition [ftip-00B9] retains a first-moment exponential moving average. It is stateful and is not the memoryless flow in Definition [ftip-00BL]. Remark 4.7 of The loss does not see the basis, but Adam does[singh2026lossbasis] treats its agreement with gradient flow as empirical rather than as a consequence of Theorem 4.6.

The theorem likewise does not apply to discrete Adam, PPO, RLVR update loops, clipping, weight decay, and mixed optimizers. Gauge equivariance and time reparameterization are separate properties.

Example. Equivariance neither guarantees nor precludes recovery [ftip-00BR]

The source records controls on both sides of the proposed implication. Its ScaledGD-inspired control is equivariant but equalizes the spectral schedule and does not recover the planted low-rank target at the declared budget. A long-anneal sign update is non-equivariant but eventually reaches a low-recovery-error regime.

$\text{[math]}$

The controls are reported in Remark 4.7 and Appendix D.10 of The loss does not see the basis, but Adam does[singh2026lossbasis]. The source reports no tuned reproducibility baseline for the ScaledGD control. The diagram is not a convergence theorem.

Optimizer geometry in training interventions [ftip-00BS]

Equivalent parameterizations need not follow equivalent training trajectories. Their evolution also depends on the update geometry; the scalar loss and represented starting object do not always suffice.

Definition. Objective-equivalent parameterizations [ftip-00BT]

Two factor pairs $(U,V)$ and $(\widetilde U,\widetilde V)$ are objective-equivalent for $L=f\circ W$ when

\[ W(U,V)=W(\widetilde U,\widetilde V). \]

They then have the same scalar loss for every objective depending only on that represented matrix. Orthogonal gauge-related pairs are objective-equivalent by Lemma [ftip-00AY]; objective equivalence need not imply membership in the same orthogonal orbit.

Definition. Parameterization-stable training intervention [ftip-00BU]

Fix a factorized objective, an update algorithm, hyperparameter schedule, randomness coupling, stopping rule, and initial optimizer state. The resulting training intervention is parameterization-stable on an orthogonal orbit when every two gauge-related initial states have identical represented trajectories:

\[ \widetilde U_t\widetilde V_t^{\mathsf T}=U_tV_t^{\mathsf T} \qquad \hbox {at every compared step }t. \]

This is a local stability property of a fully declared intervention. It does not say that arbitrary objective-equivalent pairs outside the orbit must agree.

proposition. Gauge equivariance gives parameterization stability [ftip-00BV]

Every gauge-equivariant optimizer is parameterization-stable on each orthogonal orbit, provided the remaining protocol coordinates are coupled as in Definition [ftip-00BU].

Proof.

Definition [ftip-00B3] gives $(\widetilde U_t,\widetilde V_t)=(U_tQ,V_tQ)$. Hence

\[ \widetilde U_t\widetilde V_t^{\mathsf T} =U_tQQ^{\mathsf T}V_t^{\mathsf T}=U_tV_t^{\mathsf T}. \]

This is a sufficient condition on a declared orbit, not a claim that equivariance is necessary for every possible equality of represented trajectories.

proposition. A first-step defect falsifies a loss-only protocol specification [ftip-00BW]

Suppose two orthogonally gauge-related initializations have equal represented object and scalar objective, but a declared update produces different represented objects after one step. Then the training intervention is not determined by the represented initialization and scalar objective alone.

Proof.

If those two objects determined the intervention, equal inputs would give the same represented next state. The witnessed inequality contradicts that factorization.

Proposition proposition [ftip-00BE] and Example [ftip-00BF] instantiate its premise for one factored quadratic and zero-state Adam. It proves underspecification, not a capability difference.

Example. The orthogonal gauge of an attention head [ftip-00BX]

For the row-vector attention weights of Definition [ftip-000P], let $W^Q,W^K\in \mathbb R^{d\times d_h}$ and $Q_h=XW^Q$, $K_h=XW^K$. Attention logits depend on these weights through

\[ Q_hK_h^{\mathsf T}=XW^Q(W^K)^{\mathsf T}X^{\mathsf T}. \]

For $A\in \mathrm O(d_h)$, the right action $(W^Q,W^K)\mapsto (W^QA,W^KA)$ preserves the middle product.

$\text{[math]}$

Section 6 of The loss does not see the basis, but Adam does[singh2026lossbasis] uses column-vector weights and the invariant $W_Q^{\mathsf T}W_K$. Transposition gives the row-vector invariant $W^Q(W^K)^{\mathsf T}$, with the action on the right as displayed.

Remark. What the attention-twin experiment observes [ftip-00BY]

Section 6 and Table 3 of The loss does not see the basis, but Adam does[singh2026lossbasis] compare two function-equivalent initializations of a two-layer, four-head transformer on modular addition. The reported relative logit distance after one Adam step is $3.6\times 10^{-3}$, versus $2.9\times 10^{-7}$ for a same-basis noise twin; the final per-head invariant distance is reported as 56 percent.

The exact first-step defect has an algebraic explanation, but later drift, scale robustness, and downstream behavior are empirical. The tested small transformers, precision, optimizer allocation, seeds, and task do not establish a frontier-wide law.

Example. One loss quotient and two optimizer trajectories [ftip-00BZ]

A loss-only description collapses a gauge orbit to one represented starting point. A basis-sensitive optimizer can split that quotient back into different represented trajectories.

$\text{[math]}$

The right-hand split is possible only because the centre box omits the factor basis and optimizer geometry. An FTIP admissible protocol in Definition [ftip-005J] must retain those intervention coordinates when they affect the executable result.

Remark. Product symmetry and the scope of optimizer conclusions [ftip-00C0]

The relevant optimizer results appear in version 1 of [⧉], submitted 5 August 2026: Sections 3--4 and 6, Appendices B.1--B.6 and B.9, with limitations in Section 11.

The full product symmetry acts on the factors by

\[ (U,V)\longmapsto (UA,VA^{-\mathsf T}). \]

It does not use $(UA,VA)$. For row-vector attention weights, the corresponding action uses right multiplication, as in Example [ftip-00BX].

The general rectangular preconditioner representation need not have the square-case uniqueness. A probability-one full-rank claim additionally needs a law for the gradient; a generic full-rank observation does not specify one.

The coordinatewise theorem is a zero-state, memoryless obstruction. The clock theorem is continuous-time and excludes scalar-Adam's momentum. Experimental optimizer and attention results remain observations reported by the source. Equivariance is not identified with low-rank recovery.

Remark. Continuous-flow conclusions and discrete optimization [ftip-00C1]

The factored-model results prove that update geometry can be a genuine intervention coordinate even when objective and represented initialization are fixed. They do not prove transformer or RLVR convergence, practical optimizer superiority, a scaling law, or any change in reliable capability. Different represented trajectories are not by themselves an acquisition witness of Definition [ftip-005X].

The memoryless theorem does not describe Adam's full state, and the continuous-flow result does not supply discrete rates or finite-step bounds. The source also notes that fixed-rank LoRA changes the learned-rank mechanism; no LoRA transfer is made here.

A dynamic-compute comparison also depends on active research state, persistent harness state, retained history, evaluation configuration, and descendant and retry costs. One realized chain and an archive-best envelope are different outcomes, even at the same declared budget.

Backlinks

Foundations and interfaces [ftip-00J9]

A post-training comparison depends on the model, task, environment, and evaluation interfaces. These determine the possible interactions and the quantities by which their outcomes are compared.

The interfaces here require a token probability law and a specified starting artifact, without fixing the neural computation that implements them. The decoder-only Transformer appears with the later architecture analysis. Pretraining, tasks and interaction provide the shared starting point for both parameter updates and agent adaptation.

Research question and scope [ftip-0002]

The question is how much independently evaluated capability a specified learning system can develop from its starting model, information and tools under stated resource bounds. Post-training changes model parameters using examples, comparisons, rewards or interaction. Agent procedures also change contexts, search and retained experience. Their effects depend on the tasks, available feedback, model computation and evaluation procedure.

A score increase can have several explanations. Elicitation improves access to behavior already observable under the starting procedure. Acquisition requires improvement on an independent transfer evaluation. More deployment computation can improve performance through additional search or sampling, while exploitation of an evaluator can improve its score without improving the intended capability. The conventions below distinguish these possibilities.

The post-training questions concern what different objectives, update rules and feedback sources enable a model to learn. The agent questions concern what persistent state, tools, interaction and recursive computation enable a system to do reliably. A retained library and an updated set of model weights can both affect later performance, but they are different interventions with different costs.

Architecture-dependent questions concern how token probabilities are computed, how parameterization interacts with optimization, and how model implementations compare under matched resources. Evaluation determines which comparisons the evidence supports. Finite probability bounds, counterexamples and feedback analyses establish consequences under their stated conditions; they do not by themselves bound every possible learning procedure.

The same distinctions extend to successive model generations. An autonomous lineage may change its representations, curricula and research procedures; an independently developed contribution may change what it can affordably acquire. That comparison includes both development and learning by the system receiving the contribution. The mathematical research settings study this direction within the broader capability question. A plateau in one recipe is evidence about that recipe, and the general discovery-cost separation remains a conjecture.

Economic conditions extend resource accounting to the productive stocks, institutions and allocations that sustain learning. Physical feasibility, financing, knowledge maintenance and competitive equilibrium impose different restrictions. A bound on attainable resources must cover the admitted development policies, including investment and deliberate maintenance. Such a bound becomes decisive for capability only when combined with an independent work lower bound; a contributor advantage also needs a realizable acquisition protocol under the same constraints.

Convention. Capability claims, procedures, evaluations, and resource bounds [ftip-0003]

A capability claim in this note names four objects:

a starting model artifact $M_0$, with its accompanying information, libraries, tools and initially available controllers disclosed;
a procedure $A$ that may query models, interact, retain experience and train successor artifacts, with its feedback sources specified;
an evaluation rule $E$, including the task law, retained artifact and deployment procedure being evaluated; and
a resource bound $b$.

The claim concerns the observable outcome of running $A$ from $M_0$ and judging the result with $E$ while respecting $b$. It is not a claim about an unbounded or unspecified model.

For a comparison across generations, use the complete lineage specification to make these initial resources and permitted operations explicit. Newly generated controllers and learned artifacts belong to the charged process. An externally supplied contribution has a declared producer and acquisition procedure; it is not silently added to the starting model's resources.

The bound $b$ may contain several coordinates, such as training tokens, rollouts, accelerator work, wall-clock time, tool calls, or evaluation-time samples. Distinguish the complete development budget from the deployment budget used to compare frozen artifacts on fresh problems. Coordinates with different units remain separate unless a declared cost model converts them to a common unit.

This convention follows the experimental separation between training and inference procedures used in [shao2024deepseekmath, secs. 3--4] and [guo2025deepseek, secs. 2--3]. Those studies motivate the objects in a claim; they do not imply that every capability is summarized by one benchmark or one scalar budget.

Example. Same model under two procedures and budgets [ftip-0004]

Holding the model fixed, the comparison isolates how inference procedure and budget alter measured success.

$\text{[math]}$

Assume independent draws and a declared perfect checker that recognizes a correct candidate when one appears. The one-draw procedure succeeds with $J_1=p$. A procedure allowed up to $k$ draws succeeds whenever not all draws fail: \[ J_k=1-(1-p)^k. \] Both procedures query the same conditional distribution.

Policy performance under a specified interaction and evaluation procedure is standard in [sutton2018reinforcement, Section 3.5]. The formula demonstrates procedural dependence. Independence and a perfect checker are declared toy assumptions, not properties attributed to a frontier language model.

Definition. Elicitation [ftip-0005]

Fix a capability claim in Convention [ftip-0003], a prespecified successful behaviour event $B$, and a tolerance $\varepsilon >0$. Let $p_0(B)$ be the probability of $B$ under the declared starting procedure and $p_1(B)$ its probability after the intervention, both measured by the same evaluator. An elicitation witness is an observed increase \[ p_1(B)>p_0(B)\quad \text {with}\quad p_0(B)\geq \varepsilon . \] It witnesses improved access to behaviour that was already observable at the declared starting budget.

The event $B$, tolerance $\varepsilon $, sampling procedure, and confidence rule are part of the claim. Changing any of them changes what the witness means.

Remark. Elicitation, starting probability, and finite evidence [ftip-0006]

Reinforcement learning can move probability toward successful traces that a base model already emits; this possibility is tested empirically in [liu2025understanding, secs. 2.3 and 3.3--3.4]. A finite experiment cannot establish that an unrestricted generative model assigns exactly zero probability to a behaviour. The positive threshold $\varepsilon $ therefore belongs to the declared experiment rather than to an ontological claim about what the model ``contains.''

An elicitation witness depends on the sampling budget, decoding rule, and reward. Bounding the attainable increase in $p_1(B)$ requires assumptions about the starting probability $p_0(B)$ and the information supplied by post-training.

Definition. Capability acquisition [ftip-0007]

Fix a capability claim in Convention [ftip-0003], a prespecified transfer evaluation $E^{+}$, and thresholds $0\leq \varepsilon _0<\varepsilon _1\leq 1$. Let $u_0$ and $u_1$ be the probabilities of passing $E^{+}$ before and after the intervention, using the same evaluation protocol and budget. An acquisition witness is the pair of inequalities \[ u_0\leq \varepsilon _0 \qquad \text {and}\qquad u_1\geq \varepsilon _1. \] The transfer evaluation must not be used to select training examples, rewards, or checkpoints.

An acquisition witness is operational. It records a large, independently measured change under one declared procedure; it does not prove that the starting model assigned mathematical probability zero to every successful continuation.

Remark. A transfer contrast as evidence of acquisition [ftip-0008]

Training success alone does not distinguish learning from memorization, selection, or evaluator exploitation. The base-versus-reinforcement-learning comparisons in [yue2025does, secs. 3--5] motivate this distinction, but their empirical criteria are not a universal definition of acquisition.

The thresholds in Definition [ftip-0007] require enough evaluation samples to certify both inequalities. Whether the witness survives a distribution shift depends on the task law; evaluator leakage invalidates its interpretation as independent evidence.

Tokens, text, and probability [ftip-0009]

A language model assigns probabilities to tokens rather than directly to semantic answers. We therefore fix the elementary probability notation, the text-to-token interface, and the factorization of a continuation before describing the neural computation that realizes those probabilities.

Notation. Finite sets, maps, distributions, random variables, and expectation [ftip-000A]

For a finite set $X$, write $|X|$ for its cardinality and $X^*$ for the set of finite sequences with entries in $X$. A map $f:X\to Y$ sends $x\in X$ to $f(x)\in Y$; the inverse image of $A\subseteq Y$ is $f^{-1}(A)=\{x\in X:f(x)\in A\}$.

Write \[ \Delta (X)=\left \{p:X\to [0,1]:\sum _{x\in X}p(x)=1\right \} \] for the probability simplex on $X$. If $p\in \Delta (X)$ and $Z:X\to \mathbb R$, then \[ \mathbb E_{x\sim p}[Z(x)]=\sum _{x\in X}p(x)Z(x). \] A random variable is a map from the sample set to its value set. Random variables $Z$ and $W$ are independent under $p$ when $p(Z=z,W=w)=p(Z=z)p(W=w)$ for every pair of values $z,w$.

For nonfinite interaction spaces, probability laws and conditional kernels require the measurable structure described in Remark [ftip-000C].

Definition. Finite probability law [ftip-000B]

Using the notation of Notation [ftip-000A], a finite probability law on $X$ is an element $p\in \Delta (X)$. For an event $A\subseteq X$, its probability is \[ p(A)=\sum _{x\in A}p(x). \] For $p(A)>0$, the conditional law on $B\subseteq X$ is $p(B\mid A)=p(A\cap B)/p(A)$.

Remark. Finite and measurable probability laws [ftip-000C]

A finite vocabulary and a bounded token sequence admit probability laws expressed as ordinary sums, with explicit normalization and conditioning. The finite-sequence interface matches [phuong2022formal, sec. 3, Sequence modelling].

States, observations, and tool outputs need not be finite. In that setting $\Delta (X)$ is replaced by probability measures on a declared measurable space and conditional laws are probability kernels. Results on finite spaces need additional measurable-space hypotheses before they apply in this setting.

Definition. Finite vocabulary [phuong2022formal, sec. 3, Sequence modelling] [ftip-000D]

A finite vocabulary is a nonempty finite set $\mathcal V$ of token identifiers. A token identifier is an atomic symbol for the sequence model; it need not be a word or a character.

When termination is represented in-band, choose a distinguished token $\mathtt {eos}\in \mathcal V$. Other control tokens, if present, are named as elements of the same vocabulary rather than assumed implicitly.

Definition. Tokenizer and detokenizer [kudo2018sentencepiece, secs. 3.1 and 3.5] [ftip-000E]

Let $\Sigma $ be a finite character or byte alphabet and let $\mathcal V$ be the vocabulary of Definition [ftip-000D]. A tokenizer is a specified map $\tau :\Sigma ^*\to \mathcal V^*$. A detokenizer is a specified map $\delta :\mathcal V^*\to \Sigma ^*$.

A tokenizer package also fixes its text normalization rule $N$. When the package promises normalized round trips, the relevant condition is $\delta (\tau (s))=N(s)$ for text $s\in \Sigma ^*$; it is not the claim that $\tau $ and $\delta $ are inverse bijections on all sequences. SentencePiece keeps the segmentation model and detokenization convention together, which is why both maps belong to the model interface.

Byte-pair encoding gives another construction of $\tau $. Learned merge operations turn an initial symbol sequence into subword units; see [sennrich2016neural, §3.2]. The learned merges and base alphabet are therefore part of the tokenizer specification.

Example. A tokenizer round trip and normalization failure [ftip-000F]

A four-stage toy tokenizer separates token-level round-trip stability from recovery of the original string.

$\text{[math]}$

Here underscores display spaces. Declare $N(A\_\_B)=a\_b$, $T(a\_b)=(a,b)$, and $D(a,b)=a\_b$. For $s=A\_\_B$, \[ D(T(N(s)))=a\_b\neq s, \qquad T(N(D(T(N(s)))))=(a,b). \] The token sequence is stable after normalization although the original case and repeated space are not recovered.

The source-grounded tokenizer interface is Definition [ftip-000E]. The displayed normalization rule belongs only to this finite construction; no such lossy transformation is attributed to SentencePiece.

Convention. Token sequences, prefixes, positions, and end-of-sequence [ftip-000G]

Let $\mathcal V$ be the vocabulary of Definition [ftip-000D]. A token sequence of length $n\geq 0$ is written $x_{1:n}=(x_1,\ldots ,x_n)\in \mathcal V^n$; positions are one-based. Its length is $|x_{1:n}|=n$, its prefix before position $t$ is $x_{<t}=x_{1:t-1}$, and $x_{<1}$ is the empty sequence. Juxtaposition $xy$ denotes concatenation.

If $\mathtt {eos}$ is distinguished, a completed generated continuation $y_{1:m}$ ends with $y_m=\mathtt {eos}$ and contains no earlier $\mathtt {eos}$. A fixed length limit can also stop generation; the stopping mechanism must state which convention it uses.

Definition. Conditional next-token law [phuong2022formal, sec. 3, Sequence modelling] [ftip-000H]

Let $\Theta $ be a parameter set. With the probability notation of Notation [ftip-000A] and the sequence convention of Convention [ftip-000G], a conditional next-token law is a family of maps \[ \pi _\theta :\mathcal V^*\longrightarrow \Delta (\mathcal V), \qquad \theta \in \Theta . \] For a prefix $x_{<t}$ and token $a\in \mathcal V$, the number $\pi _\theta (a\mid x_{<t})$ is the probability assigned to choosing $a$ at position $t$.

This is the observable probabilistic interface. It does not yet specify the neural computation that produces the distribution.

Definition. Autoregressive continuation law [phuong2022formal, sec. 3, Sequence modelling] [ftip-000I]

Fix the next-token law of Definition [ftip-000H], a prompt $x\in \mathcal V^*$, and a continuation $y_{1:m}\in \mathcal V^m$. Its autoregressive continuation probability is \[ P_\theta (y_{1:m}\mid x) =\prod _{t=1}^{m} \pi _\theta \left (y_t\mid x y_{<t}\right ). \] Each factor conditions only on the prompt and earlier continuation tokens.

If generation stops on $\mathtt {eos}$, the law of a completed continuation is restricted to the sequences specified in Convention [ftip-000G]. If generation instead stops at a length limit, the terminal event and its probability must be recorded separately.

Example. A three-symbol autoregressive continuation [ftip-000J]

Along one branch of a three-symbol prefix tree, conditional token laws determine both continuation probability and negative log-likelihood.

$\text{[math]}$

Let the vocabulary be $\{a,b,\mathtt {EOS}\}$ and let $p_i$ be the displayed conditional probability at step $i$. The continuation $(b,a,\mathtt {EOS})$ has \[ \Pr (b,a,\mathtt {EOS}\mid a) =p_1p_2p_3, \qquad -\log \Pr =-\sum _{i=1}^{3}\log p_i. \]

The multiplication follows the causal product in [phuong2022formal, Section 3]. Its scope is one fixed finite continuation, leaving both the decoding algorithm and the law over all variable-length outputs unspecified.

Pretraining [ftip-0014]

Pretraining fits the parameters of a language model to a large corpus by predicting tokens from preceding tokens. This description leaves the data law, window sampler, loss mask, population objective, finite-sample objective, and update rule unspecified. Each receives a separate definition below.

Notation. Pretraining data, masks, parameters, and updates [ftip-0015]

Let $\mathcal D_{\rm doc}$ be a measurable document space and $\mathcal S$ a finite set of data sources. Source $s\in \mathcal S$ has a document law $Q_s$ on $\mathcal D_{\rm doc}$, and mixture weights $\alpha \in \Delta (\mathcal S)$ use the simplex notation of Notation [ftip-000A]. Let $T$ be the context length. A token window is $x_{1:T}\in \mathcal V^T$; a prediction mask is $m_{1:T}\in \{0,1\}^T$, where $m_t=1$ means that the token at position $t$ contributes to the loss.

Let $\Theta $ be the parameter set used by the next-token law in Definition [ftip-000H]. The model parameters at optimizer update $k\in \{0,1,\ldots ,K\}$ are $\theta _k\in \Theta $; $\theta _0$ is the initialization and $\theta _K$ the stopped parameter value. A minibatch at update $k$ is $\mathcal B_k$. Random document draws, window offsets, masks, batch order, and optimizer noise are part of the training run even when the notation suppresses them.

Definition. Pretraining source mixture [ftip-0016]

Using Notation [ftip-0015], a pretraining source mixture is the pair $((Q_s)_{s\in \mathcal S},\alpha )$. It induces the document law \[ Q_{\alpha }(A)=\sum _{s\in \mathcal S}\alpha _s Q_s(A) \] for every document event $A$: first sample a source $s$ with probability $\alpha _s$, then sample a document from $Q_s$.

The mixture weights specify sampling frequency. They need not equal the fraction of bytes stored in each source. Resampling, epoch boundaries, or temperature-based weights must therefore be recorded as part of $\alpha $ or its schedule.

Definition. Document filter [ftip-0070]

Let $\dagger $ denote rejection. A document filter is a declared map \[ F:\mathcal D_{\rm doc}\longrightarrow \mathcal D_{\rm doc}\cup \{\dagger \}. \] It may retain a document, return a transformed document, or reject it. The lift $F_{\#}$ applies $F$ to a finite corpus and removes every $\dagger $.

A filter specification includes the features it reads, thresholds, transformation rules, and implementation revision. An acceptance rate alone does not determine the retained document law.

Definition. Deduplication operator [ftip-0071]

Let $\mathcal M_{\rm doc}$ be the set of finite multisets of documents. A deduplication operator is a map \[ D:\mathcal M_{\rm doc}\longrightarrow \mathcal M_{\rm doc} \] that selects retained representatives according to a declared duplicate relation. The relation may compare complete documents or spans.

The declaration includes normalization, similarity threshold, cluster construction, and representative tie-breaking. Different choices can remove different training events even when they report the same duplicate rate.

Definition. Evaluation decontamination operator [ftip-0072]

For a protected evaluation collection $\mathcal E$, an evaluation decontamination operator is a map \[ C_{\mathcal E}:\mathcal M_{\rm doc}\longrightarrow \mathcal M_{\rm doc} \] that removes documents or spans matching $\mathcal E$ under a declared matching rule.

The protected collection, normalization, match granularity, threshold, and removal policy are part of the operator. Decontamination against one released benchmark does not establish independence from every related task.

Definition. Ordered pretraining data pipeline [ftip-0017]

Let $\mathcal C_{\mathrm {raw}}\in \mathcal M_{\rm doc}$ be a finite corpus drawn from the source mixture in Definition [ftip-0016]. Using the declared document filter, deduplication operator, and evaluation decontamination operator, an ordered pretraining data pipeline returns \[ \mathcal C_{\mathrm {clean}} =C_{\mathcal E}\left (D\left (F_{\#}(\mathcal C_{\mathrm {raw}})\right )\right ). \] Changing the order can change the retained corpus. The pipeline therefore records the order and exact revisions in addition to the three operators.

Remark. From a data pipeline to a corpus law [ftip-0018]

The source mixture and cleaning pipeline determine different parts of the data law. The mixture in Definition [ftip-0016] describes how raw documents are proposed. The construction in Definition [ftip-0017] determines which proposals survive and how they are transformed. Together with all random seeds and thresholds, they induce a cleaned document law $Q_{\mathrm {clean}}$.

Kaplan et al. describe the concrete WebText2 dataset in [kaplan2020scaling, sec. 2.3]. Hoffmann et al. report the MassiveText sources and mixture in [hoffmann2022training, Appendix A, Table A1]. A finite released corpus is one realization of such choices, and an undocumented change in them is not identified by a loss--compute curve.

Definition. Context-window sampler [ftip-0019]

Fix the context length $T$ from Notation [ftip-0015]. A context-window sampler draws cleaned text from the law described in Remark [ftip-0018] and tokenizes it with Definition [ftip-000E]. It then chooses a declared offset and returns a length-$T$ token sequence $x_{1:T}$ together with metadata that identifies document boundaries.

If fewer than $T$ tokens remain, the sampler must specify whether it pads, drops the suffix, or joins another document. These choices change both the distribution of contexts and the prediction mask.

Definition. Sequence packing and document mask [ftip-001A]

A packed training record is a tuple $\zeta =(x,m,c,M^{\mathrm {doc}})$. It places tokenized segments $x^{(1)},\ldots ,x^{(r)}$ into the length-$T$ token array $x$, records a prediction mask $m$, and records a segment identifier $c_t\in \{0,1,\ldots ,r\}$ at each position. The value $c_t=0$ denotes padding. Its document mask is \[ M^{\mathrm {doc}}_{ts}= \begin {cases} 0,&c_t=c_s\ne 0\text { and }s\leq t,\\ -\infty ,&\text {otherwise}. \end {cases} \] Replacing the causal mask in Definition [ftip-000S] by this mask prevents one packed document from attending to another. We write $x_t(\zeta )$, $m_t(\zeta )$, and $M^{\mathrm {doc}}(\zeta )$ for the corresponding fields.

The prediction mask $m_t(\zeta )$ from Notation [ftip-0015] is one only at positions whose target token is retained for training. Padding, a segment's first token when no predecessor is present, and any deliberately excluded control token receive mask value zero.

Remark. Window sampling, document masks, and packing [ftip-006K]

[kaplan2020scaling], Section 2.2, uses fixed-length sequence batches.

[phuong2022formal], Section 7, gives the causal language-model objective.

The proposed data interface additionally specifies the sampler, segment identifiers, document mask, and prediction mask. Those boundary choices are not determined by the high-level training accounts.

These objects describe one admissible data interface. A concrete training run must replace them with its offset, padding, joining, masking, and packing rules.

Definition. Token negative log-likelihood [ftip-001B]

Let $\zeta =(x,m,c,M^{\mathrm {doc}})$ be a packed training record from Definition [ftip-001A]. Write $\pi _{\theta ,M^{\mathrm {doc}}(\zeta )}(\,\cdot \mid x_{<t}(\zeta ))$ for the next-token law of Definition [ftip-000H] when every decoder attention layer uses the declared document mask. For a predicted position $t\geq 2$, its masked token negative log-likelihood is \[ \ell _t(\theta ;\zeta ) =-m_t(\zeta )\log \pi _{\theta ,M^{\mathrm {doc}}(\zeta )} \left (x_t(\zeta )\mid x_{<t}(\zeta )\right ), \] A masked position has zero contribution; an unmasked position penalizes the log probability computed under the same segment boundaries that defined the packed example.

Definition. Population pretraining objective [ftip-001C]

Let $Q_T$ be the law of packed examples induced by Remark [ftip-0018]--Definition [ftip-001A], and assume that its expected number of predicted positions is positive. The population pretraining objective is the expected total token loss divided by the expected number of predicted tokens: \[ L_{\mathrm {pop}}(\theta ) =\frac { \mathbb E_{\zeta \sim Q_T} \left [\sum _{t=2}^{T}\ell _t(\theta ;\zeta )\right ] }{ \mathbb E_{\zeta \sim Q_T} \left [\sum _{t=2}^{T}m_t(\zeta )\right ] }, \] using expectation as fixed in Notation [ftip-000A].

Example. Packing two documents without cross-document attention [ftip-001D]

Packing documents into one row requires a block-causal mask so that each token attends only within its own document and to earlier positions there.

$\text{[math]}$

For a packed row, set $M_{ij}=0$ only when tokens $i,j$ belong to the same document and $j\leq i$; otherwise set $M_{ij}=-\infty $. Thus a token in document $B$ cannot inspect any token in document $A$, even when $A$ precedes $B$ in storage.

The within-document triangle instantiates the causal mask in Definition [ftip-000S]. The block boundary instantiates the packing convention in Definition [ftip-001A]. Document-isolated attention is the declared convention in this packed row, not a universal description of pretraining systems.

Definition. Empirical pretraining objective [ftip-001E]

For a finite training sample of packed records $\mathcal D_N=(\zeta ^{(i)})_{i=1}^{N}$ with at least one unmasked target token, the empirical pretraining objective is \[ L_N(\theta ) =\frac { \sum _{i=1}^{N}\sum _{t=2}^{T} \ell _t(\theta ;\zeta ^{(i)}) }{ \sum _{i=1}^{N}\sum _{t=2}^{T}m_t(\zeta ^{(i)}) }. \] This is a per-predicted-token average. Reusing or resampling examples changes the stochastic optimization path even when the displayed finite-sample function is unchanged.

Remark. Normalization of masked pretraining loss [ftip-006L]

The autoregressive token loss follows [phuong2022formal, Section 7]. Kaplan et al. report language-model loss with parameter and compute variables in [kaplan2020scaling, sec. 2.1]. The population ratio-of-expectations in Definition [ftip-001C] and finite per-token ratio in Definition [ftip-001E] are proposed normalization conventions with explicit prediction masks. Neither cited source defines those exact ratios.

A theorem or experiment using another record weighting, length weighting, or expectation order must state that change rather than reuse these symbols.

Definition. Minibatch gradient estimator [ftip-001G]

At update $k$, let $\mathcal B_k$ be the minibatch named in Notation [ftip-0015], and let $N_k=\sum _{\zeta \in \mathcal B_k}\sum _{t=2}^{T}m_t(\zeta )$ be its number of predicted tokens. For $N_k>0$, the minibatch gradient estimator is \[ g_k=\frac 1{N_k} \sum _{\zeta \in \mathcal B_k}\sum _{t=2}^{T} \nabla _\theta \ell _t(\theta _k;\zeta ), \] where $\ell _t$ is defined in Definition [ftip-001B]. Whether $g_k$ is unbiased for the gradient of Definition [ftip-001C] depends on the window sampler and minibatch weighting. Other sampling schemes require their own bias statement.

Example. A two-record pretraining loss and minibatch estimate [ftip-001F]

Two equal-length records make a full empirical loss directly comparable with a minibatch estimate, while exposing the assumptions behind unbiasedness.

$\text{[math]}$

Let $\ell _{ij}$ denote the token loss at position $j$ of record $i$. The full token-average loss is the mean of all $\ell _{ij}$, while a one-record minibatch averages only the losses from its sampled record. Uniform sampling of equal-length records makes the latter unbiased for the former.

The preceding loss and estimator definitions apply to this equal-length record setup. Equal record lengths make the estimator unbiased here; unequal lengths, padding, dependence, and optimizer noise remain outside its scope.

Definition. Optimizer state [kingma2015adam, Algorithm 1] [ftip-001H]

An optimizer state $s_k$ is the collection of persistent variables, other than the model parameters $\theta _k$, that an update rule uses after step $k$. For Adam, $s_k$ contains the step index and exponential moving averages of the gradient and squared gradient.

The state is initialized by a declared value $s_0$. Two runs with the same checkpoint $\theta _k$ but different optimizer states need not have the same next update.

Definition. Parameter update [kingma2015adam, Algorithm 1] [ftip-001I]

A parameter update rule is a specified map $U_k$ that takes the current parameters $\theta _k$, optimizer state $s_k$, gradient estimate $g_k$, and scheduled hyperparameters $\eta _k$, and returns \[ (\theta _{k+1},s_{k+1}) =U_k(\theta _k,s_k,g_k,\eta _k). \] The learning rate, momentum coefficients, numerical stabilizers, clipping, and weight decay belong to $\eta _k$ or to the declared form of $U_k$; they are not determined by the loss alone.

Remark. Randomness, schedules, and stopping [ftip-001J]

A pretraining run is not determined by minimization of the empirical objective $L_N$ in Definition [ftip-001E]. It includes the initialization, data order, window offsets, dropout and other model randomness, numerical precision, distributed reduction order, optimizer state, every hyperparameter schedule, and a stopping rule. These objects determine a distribution over final parameters even when the source mixture and nominal objective agree.

The training configuration of [vaswani2017attention, sec. 5.3] and the scaling experiments of [kaplan2020scaling, sec. 2.2] report concrete instances of these choices. An optimization result depends on which randomness it averages over and whether the stopping time is fixed or data-dependent.

Definition. Base-model artifact [ftip-001K]

A base-model artifact is an executable tuple \[ M_0=(\tau ,\delta ,\mathsf {cfg},\theta _K,\mathsf {num}), \] where $\tau ,\delta $ are the tokenizer and detokenizer of Definition [ftip-000E]; $\mathsf {cfg}$ fixes the architecture of Definition [ftip-0012]; $\theta _K$ is the stopped pretraining checkpoint; and $\mathsf {num}$ fixes the numerical and inference conventions required to evaluate its next-token law.

The tokenizer model is part of the executable system by Definition [ftip-000E]. A weight file without its vocabulary, architecture configuration, or output convention does not determine the law $\pi _{\theta _K}$ in Definition [ftip-000H]. Training-data and optimizer provenance may accompany the artifact, but they are not runtime inputs.

Remark. The runtime ingredients of a model artifact [ftip-006M]

Section 6 of [phuong2022formal] describes decoder architecture. Sections 3.1 and 3.5 of [kudo2018sentencepiece] treat the tokenizer as a separate model. Public model releases additionally distribute weights, configuration, and numerical conventions. The proposed artifact model in Definition [ftip-001K] includes these runtime ingredients because none alone determines an executable next-token law.

Training provenance accompanies the artifact but is not itself a runtime argument. Later causal claims must distinguish the executable endpoint from the process that produced it.

Example. Anatomy of a base-model artifact and V4 optimizer/data provenance [ftip-001L]

An executable-artifact manifest separates runtime components from evidence about how the corresponding weights were trained.

$\text{[math]}$

The pinned DeepSeek-V4-Flash-0731 configuration [⧉] declares architecture and numeric-format fields. Sections 2.4 and 4 of the V4 family report [⧉] describe a mixed Muon/AdamW optimizer allocation and pretraining data construction. The exact 0731 data mixture, optimizer schedule, and post-preview training delta remain undisclosed by those sources.

Tasks and evaluation interfaces [ftip-001M]

A model does not have a benchmark score without a task distribution, an inference procedure, and an evaluator. Keeping these objects separate is essential for post-training: a training change and an evaluation-time search change can raise the same reported score while answering different research questions.

Notation. Tasks, task laws, inference protocols, and utility [ftip-001N]

Write $\mathsf T$ for a task interface, $\mathcal X_{\mathsf T}$ for its instance set, and $\mathcal O_{\mathsf T}$ for its candidate-outcome set. A task instance is $x\in \mathcal X_{\mathsf T}$ and a candidate outcome is $o\in \mathcal O_{\mathsf T}$. The symbol $\mu _{\mathsf T}$ denotes a probability law on instances. Equip the instance and outcome sets with declared sigma-algebras; $\mu _{\mathsf T}$ is a probability measure on the instance sigma-algebra.

Fix an evaluation-resource dimension $m_{\mathrm {eval}}\geq 1$ and write $\mathcal B_{\mathrm {eval}}=\mathbb R_+^{m_{\mathrm {eval}}}$ with its coordinatewise order. An evaluation inference protocol is written $\mathsf I$; its declared evaluation budget is $b_{\mathrm {eval}}\in \mathcal B_{\mathrm {eval}}$. Its inference-seed set is $\Omega _I$, with realized seed $\xi \in \Omega _I$. An evaluator's random-seed set is $\Omega _E$; its realized seed is $\omega \in \Omega _E$. It returns utility $u_{\mathsf T}(x,o;\omega )\in \mathbb R^d$. The dimension $d\geq 1$ is declared by the evaluation: $d=1$ gives scalar utility, while $d>1$ retains several outcomes or costs without an implicit weighting. Equip both seed sets with declared sigma-algebras and $\mathbb R^d$ with its Borel sigma-algebra. Products below carry the product sigma-algebra; finite or countable discrete spaces may use all subsets. A seed space alone does not specify its sampling law.

Definition. Task [ftip-001O]

A task is a triple \[ \mathsf T=(\mathcal X_{\mathsf T},\mathcal O_{\mathsf T}, \operatorname {Adm}_{\mathsf T}), \] where $\mathcal X_{\mathsf T}$ is an instance set, $\mathcal O_{\mathsf T}$ is a candidate-outcome set, and $\operatorname {Adm}_{\mathsf T}(x,o)$ is a declared well-formedness predicate for an instance-outcome pair.

Admissibility says that an outcome can be interpreted for the instance; it does not say that the outcome is correct, useful, or safe. Those judgments belong to the evaluation utility introduced later.

Definition. Task law [ftip-001Q]

For the instance set in Definition [ftip-001O], a task law is a probability law $\mu _{\mathsf T}$ on $\mathcal X_{\mathsf T}$. An evaluation sample of size $N$ is a declared joint law of instances $(X_1,\ldots ,X_N)$ whose marginals are $\mu _{\mathsf T}$; independent sampling is a further assumption, not part of the term ``task law.''

In an episodic control problem this law includes the distribution of starting situations. In a fixed benchmark it may instead be the uniform law on a held-out finite set. Reweighting instances changes the task law even when the instance files are unchanged.

Remark. Response tasks and interactive tasks [ftip-001P]

A task and a task law specify admissible instance--outcome pairs and how instances are sampled. Benchmark datasets usually present an input and expect an answer in a parseable format. Reinforcement-learning formulations instead name states, actions, and reward; see [sutton2018reinforcement, Chapter 3]. The admissibility predicate captures parseability, while the law records how instances are sampled. Neither choice already identifies parsing with success.

One-shot inference and an interactive environment can share the same task interface. Claims about an interaction's temporal structure additionally depend on the environment and the observations available at each decision.

Definition. Evaluation inference protocol [ftip-001R]

Let $M$ be an executable model artifact of the form in Definition [ftip-001K]. An evaluation inference protocol is a specified procedure \[ \mathsf I(M,x,b_{\mathrm {eval}};\xi )=(o,c) \in \mathcal O_{\mathsf T}\times \mathcal B_{\mathrm {eval}}, \] where $x\in \mathcal X_{\mathsf T}$, $b_{\mathrm {eval}}\in \mathcal B_{\mathrm {eval}}$, and $\xi \in \Omega _I$ use the notation of Notation [ftip-001N]; $o$ is a candidate outcome and $c\in \mathcal B_{\mathrm {eval}}$ is the realized resource-use vector. A valid execution satisfies $c\preceq b_{\mathrm {eval}}$ coordinatewise.

The protocol fixes prompt construction, decoding, sampling temperature, number of candidates, candidate selection, allowed tools, stopping, and any test-time search. None of these choices is determined by the checkpoint alone.

Remark. Decoding and test-time compute belong to evaluation [ftip-001S]

One next-token law supports many evaluation procedures. Greedy decoding, temperature sampling, majority voting, verifier-guided selection, and multi-round tool use can produce different outcome laws from the same checkpoint. The sampling and selection procedures in [shao2024deepseekmath, secs. 3--4] are concrete examples.

Accordingly, a comparison that changes $b_{\mathrm {eval}}$ or $\mathsf I$ does not estimate a training-only effect. A potential frontier therefore depends on evaluation-time compute as well as the trained model.

Definition. Evaluation utility [ftip-001T]

With the notation of Notation [ftip-001N], an evaluation utility is a declared measurable function \[ u_{\mathsf T}:\mathcal X_{\mathsf T}\times \mathcal O_{\mathsf T}\times \Omega _E\longrightarrow \mathbb R^d, \] where $\Omega _E$ is the evaluator's measurable seed space. The evaluator applies it only to admissible pairs from Definition [ftip-001O]. A deterministic evaluator is the special case in which the value does not depend on $\omega \in \Omega _E$.

Binary pass-fail evaluation takes $d=1$ and values in $\{0,1\}$. A cumulative return is another scalar case. Vector utility keeps qualities such as correctness, safety, latency, and tool cost distinct until a later rule declares how to compare them.

Whenever an expected utility vector is used, its composite evaluation random variable must be measurable and absolutely integrable in every coordinate under the declared joint law. Thus its expectation lies in $\mathbb R^d$. Bounded measurable utility is a sufficient specialization; so is finite support for the complete evaluation random variable with finite values on that support. Merely taking a finite real value at each seed is insufficient for an unbounded evaluator. The scalar protocol comparison in Convention [ftip-005D] states its joint law explicitly.

Remark. Scalar returns and multiple evaluation quantities [ftip-006N]

Sutton and Barto define scalar episodic return in [sutton2018reinforcement, Section 3.3]. Benchmarks also use exact-match, test, latency, safety, and cost measurements. A proposed seeded vector utility keeps those coordinates separate, with evaluator randomness included in the declared law.

The vector is not a claim that its coordinates share a natural scale. Scalarization, Pareto order, and worst-group summaries are introduced only after the attainable evaluation set is defined.

Example. One policy under two evaluation inference budgets [ftip-001U]

One fixed policy can receive different measured success rates when the evaluation horizon changes.

$\text{[math]}$

Partition complete trajectories into those that finish within the short horizon, those requiring additional tokens, and failures. The short protocol counts only the first class; the longer protocol may count the first two. Thus the measured score depends on the declared horizon even though the policy is unchanged.

Making an evaluation horizon explicit follows finite-horizon policy evaluation in [sutton2018reinforcement, Chapter 3]. Only truncation is varied here. Larger budgets need not help once decoding rules, tool costs, or selection errors are allowed to change with the budget.

Remark. Public task data and evaluator-only information [ftip-001V]

The inference protocol receives only the information designated as public. The public instance $x$ may include a prompt, tools, and a response schema. Evaluator-only information may include a reference answer, hidden tests, a simulator state, or the evaluator seed $\omega $. The protocol in Definition [ftip-001R] receives the former and not the latter.

If evaluator-only information influences training, decoding, or candidate selection, the experiment no longer measures performance under the declared information interface. A theorem that assumes an independent evaluator must name the sigma-field or finite data record from which the protocol is excluded.

Interactive environments and policies [ftip-001W]

Some tasks end with one generated response. Agentic tasks instead alternate between an acting system and an environment: the system calls a tool, receives an observation, and chooses what to do next. The following objects make that temporal and informational structure explicit before any reinforcement-learning objective is introduced.

Notation. State, observation, action, time, history, and stopping [ftip-001X]

Let $\mathcal S,\mathcal O,\mathcal A$ be declared measurable spaces of environment states, public observations, and agent actions. Time is discrete, $t\in \{0,1,\ldots ,T_{\max }\}$, where $T_{\max }<\infty $ is a hard horizon. Random states, observations, and actions are $S_t,O_t,A_t$; lowercase $s_t,o_t,a_t$ denote realized values. The same convention makes $h_t$ a realization of $H_t$ and $z$ a realization of $Z$.

The public history available before action $A_t$ is $H_t$; the full environment trajectory is $Z$; and the stopping time is $\tau _{\mathrm {stop}}\in \{0,1,\ldots ,T_{\max }\}$. Probability kernels are written $K(dy\mid x)$; on finite spaces this means the probability law $K(y\mid x)$ and reduces to the notation of Notation [ftip-000A].

Definition. Initial-state law [ftip-001Y]

With the spaces in Notation [ftip-001X], an initial-state law $\rho _0(ds_0,do_0)$ is a probability law on $\mathcal S\times \mathcal O$. It jointly samples the hidden initial state $S_0$ and the first public observation $O_0$.

Allowing a joint law covers both a deterministic rendering $O_0=\operatorname {obs}(S_0)$ and a noisy initial observation. A fixed task instance can be included as a coordinate of $O_0$; its sampling law is then the task law of Definition [ftip-001Q].

Definition. Transition-observation kernel [ftip-001Z]

A transition-observation kernel at time $t$ is a probability kernel \[ K_t(ds_{t+1},do_{t+1}\mid s_t,a_t) \] from the current hidden state and action to the next hidden state and public observation. After $A_t=a_t$, the environment draws $(S_{t+1},O_{t+1})$ from this kernel.

The factorization $K_t(ds',do'\mid s,a)=P_t(ds'\mid s,a)\,Q_t(do'\mid s',a)$ for declared kernels $P_t$ and $Q_t$ recovers the usual separate state-transition and observation kernels when such a factorization is declared. The joint-kernel form does not assume it.

Remark. Deterministic tools and stochastic environments [ftip-0020]

The kernel in Definition [ftip-001Z] describes both tools and broader environments. A deterministic tool has a measurable update map $F_t(s,a)=(s',o')$ and the point-mass kernel concentrated at $F_t(s,a)$. A stochastic environment permits several next state-observation pairs for the same input.

This distinction matters for replay. Re-executing a deterministic tool call under an unchanged state reproduces its observation, whereas a stored rollout from a stochastic environment is one sample from a conditional law. Any analysis that replaces one with the other needs a coupling or concentration assumption.

Definition. Public history [ftip-0021]

At time $t$, the public history is the alternating sequence \[ H_t=(O_0,A_0,O_1,A_1,\ldots ,A_{t-1},O_t). \] Thus $H_0=(O_0)$. It contains exactly the observations received and actions taken before the next action $A_t$; it does not contain the hidden state $S_t$ unless that state was itself revealed as an observation.

Remark. The information boundary of a history [ftip-0022]

The history in Definition [ftip-0021] contains the agent's available information. The hidden state sequence $(S_0,\ldots ,S_t)$, future observations, evaluator-only tests from Remark [ftip-001V], and future random seeds are absent unless the environment has already exposed them through $O_0,\ldots ,O_t$.

This boundary determines which policies are implementable. It also separates legitimate memory from leakage: a summary of earlier public observations may be stored in the history, while a hidden reference answer may not be supplied under the same declaration.

Definition. Non-anticipating policy [ftip-0023]

A non-anticipating policy is a family of probability kernels \[ \pi _t(da\mid H_t), \qquad t=0,1,\ldots ,T_{\max }-1, \] from public histories in Definition [ftip-0021] to the action space $\mathcal A$. At time $t$, the action may depend on $H_t$ and fresh policy randomness, but not on a later observation or an unobserved state.

A decoder-only language model becomes such a policy only after an inference protocol serializes $H_t$ into tokens, decodes tokens, and parses them as an action. The next-token law in Definition [ftip-000H] alone does not specify those maps.

Definition. Stopping rule [ftip-0024]

A stopping rule is a random time $\tau _{\mathrm {stop}}\in \{0,1,\ldots ,T_{\max }\}$ such that, for each $t$, the decision whether $\tau _{\mathrm {stop}}=t$ is determined by the public history $H_t$ and any declared stopping randomness available by time $t$. It cannot inspect future observations.

Stopping can be triggered by a terminal environment observation, an agent action, a resource limit, or the hard horizon. The trigger and the owner of the decision are part of the rule.

Definition. Stopped trajectory [ftip-0025]

If $\tau _{\mathrm {stop}}=n$, the stopped trajectory is \[ Z=(S_0,O_0,A_0,S_1,O_1,A_1,\ldots , A_{n-1},S_n,O_n). \] It contains the environment states as well as the public action-observation record. Its public projection is the history $H_n$ of Definition [ftip-0021].

For $n=0$, the trajectory is $(S_0,O_0)$ and contains no action. A terminal utility may depend on the full trajectory when evaluated inside the environment, even though the policy is restricted to its public projection.

Definition. Interactive realization of a task [ftip-006T]

For a task $\mathsf T=(\mathcal X_{\mathsf T},\mathcal O_{\mathsf T}, \operatorname {Adm}_{\mathsf T})$ from Definition [ftip-001O], an interactive realization is

\[ \mathsf {Env}_{\mathsf T} =\left (\mathcal S_{\mathsf T},\mathcal O^{\rm obs}_{\mathsf T}, \mathcal A_{\mathsf T},T_{\max }, (\rho _0^x)_x,(K_t^x)_{x,t},\operatorname {out}_{\mathsf T}\right ). \]

For each $x\in \mathcal X_{\mathsf T}$, the law $\rho _0^x$ is on $\mathcal S_{\mathsf T}\times \mathcal O^{\rm obs}_{\mathsf T}$, and $K_t^x$ maps a state and action to the next state-observation law for $0\leq t<T_{\max }$. The outcome map sends a stopped trajectory $Z$ to $\operatorname {out}_{\mathsf T}(x,Z)\in \mathcal O_{\mathsf T}$ and satisfies $\operatorname {Adm}_{\mathsf T}(x,\operatorname {out}_{\mathsf T}(x,Z))$. For one fixed realization, write its three spaces as the unadorned $\mathcal S,\mathcal O,\mathcal A$ of Notation [ftip-001X].

The agent policy and stopping rule are not environment fields. They are combined with this realization only when an interaction law is formed.

Remark. How a task acquires interactive semantics [ftip-006U]

Section 3.1 of [sutton2018reinforcement] specifies the agent--environment interaction. Section 2 of [kaelbling1998planning] separates hidden state from observation. We combine those ingredients with the task interface of Definition [ftip-001O] so each task instance selects an initial law and transition kernels.

The outcome-extraction map is an additional interface choice; neither cited source supplies it in this task-indexed form. This map connects a stopped interaction to the candidate outcome that the independent evaluator will score.

Example. A finite agent--environment interaction loop [ftip-0026]

A deterministic three-action episode records each observation before the next policy decision and ends with an explicit terminal observation.

$\text{[math]}$

Let $H_0=(\text {door locked})$. After action $A_0=\text {take key}$, the environment returns $O_1=\text {key held}$, so $H_1=(H_0,A_0,O_1)$. Repeating the same update with unlock and door open, then choosing $A_2=\text {enter}$, produces $O_3=\text {terminal success}$ and $H_3=(H_2,A_2,O_3)$. No later observation is available to an earlier action choice.

The alternating action--observation interface is the finite-history form of the agent setup in [kaelbling1998planning, Section 2]. The trace checks chronology and termination; uncertain state, partial observability, and language-valued actions lie beyond this deterministic instance.

Example. Token generation as a stopped environment interaction [ftip-0027]

A prefix tree represents token emission as a finite interaction stopped either by an end token or by a hard length cap.

$\text{[math]}$

Take vocabulary $\{a,\mathtt {EOS}\}$. The terminal histories are an immediate end token, an end token after $a$, and a length-capped history $aa$. Their probabilities are products of the conditional token laws and sum to one. The last history terminates because the declared maximum length is reached, not because the policy emitted $\mathtt {EOS}$.

The causal token product is given in [phuong2022formal, Section 3]. The finite representation distinguishes two stopping mechanisms. Token steps remain distinct from tool calls, and unbounded generation is outside the construction.

Remark. The induced law of a stopped interaction [ftip-0028]

Sutton and Barto's agent--environment interface in [sutton2018reinforcement, sec. 3.1] supplies alternating states, actions, and rewards. Kaelbling et al. separate hidden state and public observation in [kaelbling1998planning, Section 2]. A stopped interaction is determined by its initial law, transition--observation kernel, policy, and public stopping rule.

The induced interaction law uses four independently declared objects: the initial law $\rho _0$ in Definition [ftip-001Y], the environment kernels $K_t$ in Definition [ftip-001Z], the policy $\pi _t$ in Definition [ftip-0023], and the stopping rule in Definition [ftip-0024]. On finite spaces, the probability of a length-$n$ trajectory before applying its stopping indicator factors as \[ \rho _0(s_0,o_0) \prod _{t=0}^{n-1} \pi _t(a_t\mid h_t) K_t(s_{t+1},o_{t+1}\mid s_t,a_t). \] The event $\{\tau _{\mathrm {stop}}=n\}$ selects the stopped trajectories.

This factorization supports later expectations, likelihood ratios, and off-policy reuse. It also exposes an identifiability limit: an outcome distribution by itself generally does not reveal which of the initial law, environment, policy, or stopping rule changed.

Definition. Autoregressive continuation law [phuong2022formal, sec. 3, Sequence modelling] [ftip-000I]