Equivariant and basis-sensitive update rules [ftip-00B6]
✍️source

    Gradient covariance supplies a test for update rules. Linear and full-matrix operations can commute with a latent rotation, whereas a fixed nonlinear map applied separately to coordinates selects a preferred basis.

    Definition 1. Memoryless right-equivariant update [ftip-00B7]

      A memoryless factor-update map is a function \(\Phi :\mathbb R^{n\times k}\to \mathbb R^{n\times k}\) applied to a current gradient \(G\) without a persistent optimizer state. It is right-equivariant when

      \[ \Phi (GQ)=\Phi (G)Q \qquad (G\in \mathbb R^{n\times k},\ Q\in \mathrm O(k)). \]

      This local definition is the input of Theorem 4.5 in The loss does not see the basis, but Adam does[singh2026lossbasis]. It does not classify stateful optimizers such as Adam.

      proposition 2. Gradient descent and momentum preserve the orthogonal gauge [ftip-00B8]

        Gradient descent and Polyak or Nesterov momentum are gauge-equivariant when their factor gradients, momentum buffers, schedules, and look-ahead points are transformed consistently.

        Proof. 

          By Lemma [ftip-00B1], \(G_t\) becomes \(G_tQ\). Gradient descent therefore sends \(U_tQ\) to \((U_t-\eta _tG_t)Q\). A momentum buffer initialized at zero and formed by linear combinations of covariant gradients transforms as \(M_tQ\); a Nesterov look-ahead point is likewise the reference look-ahead right-multiplied by \(Q\). Induction proves the claim.

          Proposition 4.2(1) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis].

          proposition 3. A shared scalar second moment preserves the orthogonal gauge [ftip-00B9]

            Replace Adam's entrywise second-moment denominator by one scalar whose updates depend only on gauge-invariant quantities such as the pooled mean of the squared entries of both factor gradients. With a covariant first-moment buffer, the resulting stateful update is gauge-equivariant.

            Proof. 

              The first moment is a linear exponential average and transforms by right multiplication by \(Q\). The pooled squared-entry mean is the joint squared Frobenius norm divided by the entry count, hence is unchanged by \(G\mapsto GQ\). Its bias correction and positive scalar denominator are unchanged, so the complete update transforms covariantly.

              Proposition 4.2(2) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis]. The result concerns this declared shared-scalar variant, not stock Adam.

              proposition 4. Full-matrix Muon and Shampoo preserve the orthogonal gauge [ftip-00BA]

                Fix one factor, write \(n\in \{d_1,d_2\}\) for its row dimension, and let \(G_t,M_t\in \mathbb R^{n\times k}\) be its gradient and Muon momentum buffer. For \(0\leq \beta <1\), \(M_{-1}=0\), and the source's linear momentum update \(M_t=\beta M_{t-1}+G_t\), Muon's direction is \(\Delta _t=\operatorname {msign}(M_t)\). Under real arithmetic this stateful update and the source's damped Shampoo update are right-equivariant. The Muon claim includes the stated finite Newton--Schulz approximations; the Shampoo claim uses its complete left and right accumulators.

                Proof. 

                  By Lemma [ftip-00B1], the primed gradient is \(G'_t=G_tQ\). Starting from \(M'_{-1}=M_{-1}Q=0\), induction through the linear momentum update gives \(M'_t=M_tQ\). If \(M_t=A\Sigma B^{\mathsf T}\) is a compact singular-value decomposition, then \(M_tQ=A\Sigma (Q^{\mathsf T}B)^{\mathsf T}\), so \(\operatorname {msign}(M_tQ)=\operatorname {msign}(M_t)Q\). For \(\epsilon _{\rm NS}>0\), the Newton--Schulz initialization \(X_0=M_t/(\lVert M_t\rVert _F+\epsilon _{\rm NS})\) is covariant because its Frobenius normalization is invariant; each polynomial iterate formed from \(X_j\) and \(X_jX_j^{\mathsf T}\) preserves the same covariance. For Shampoo, the left accumulator is invariant while the right accumulator transforms by \(R_t\mapsto Q^{\mathsf T} R_t Q\). Orthogonal functional calculus conjugates the right matrix function, so the transformed update is the original update right-multiplied by \(Q\).

                  Proposition 4.2(3)--(4) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis]. Variable splitting, coordinatewise clipping, mixed update rules, finite precision, and different damping conventions lie outside this statement. The Muon part is stateful; it does not replace the momentum buffer by the current gradient or invoke the memoryless classification of Definition 1.

                  proposition 5. Coordinatewise equivariance forces linearity [ftip-00BB]

                    Let \(k\geq 2\) and let a memoryless update have the fixed entrywise form \(\Phi (G)_{ij}=\phi (G_{ij})\). If \(\Phi (GQ)=\Phi (G)Q\) for every \(G\) and \(Q\in \mathrm O(k)\), then \(\phi (x)=cx\) for some \(c\in \mathbb R\).

                    Proof. 

                      It suffices to use one row and rotations in the first two coordinates. Equivariance at zero gives \(\phi (0)=0\). Rotating \((x,0)\) through an angle whose cosine is \(a\in [-1,1]\) gives \(\phi (ax)=a\phi (x)\). For nonzero \(x,y\), compare each with a third number of at least their absolute magnitudes; the ratio \(\phi (x)/x\) is constant.

                      Proposition 4.3 and its proof are in Appendix B.4 of The loss does not see the basis, but Adam does[singh2026lossbasis]. No continuity assumption is needed.

                      Corollary 6. Fixed nonlinear coordinatewise updates break the gauge [ftip-00BC]

                        A nonlinear fixed entrywise update applied at a zero optimizer state is not gauge-equivariant when \(k\geq 2\). In particular, the source applies proposition 5 to the first-step maps of Adam, RMSProp, signSGD, and Lion; it checks Adafactor separately because its factored statistics are not an entrywise map.

                        This is a first-step obstruction. It suffices to disprove equivariance of the full run, but it does not classify every later state or say that such an optimizer cannot reach a particular solution.

                        Definition 7. First-step Adam map and gauge defect [ftip-00BD]

                          Fix \(\epsilon >0\). For zero-state, bias-corrected Adam define the entrywise first-step direction

                          \[ D_\epsilon (G)=G\mathbin {\oslash }(|G|+\epsilon ) \]

                          and, for \(Q\in \mathrm O(k)\), define its gauge-aligned defect by

                          \[ E_Q(G)=D_\epsilon (GQ)Q^{\mathsf T}-D_\epsilon (G). \]

                          Here \(|G|\), addition, and division are entrywise. Bias correction makes the first direction independent of Adam's two decay coefficients under the declared convention [singh2026lossbasis, Proposition 4.4].

                          proposition 8. Exact represented-matrix defect after one Adam step [ftip-00BE]

                            Let \(G_U=\nabla _UL(U_0,V_0)\), \(G_V=\nabla _VL(U_0,V_0)\), \(D_U=D_\epsilon (G_U)\), and \(D_V=D_\epsilon (G_V)\). Let \(W_1\) be the product after one Adam step from \((U_0,V_0)\) and \(\widetilde W_1\) the product after one step from \((U_0Q,V_0Q)\). Then

                            \[ \begin {aligned} \widetilde W_1-W_1={}&-\eta \bigl (E_Q(G_U)V_0^{\mathsf T} +U_0E_Q(G_V)^{\mathsf T}\bigr )\\ &+\eta ^2\bigl (E_Q(G_U)D_V^{\mathsf T} +D_UE_Q(G_V)^{\mathsf T} +E_Q(G_U)E_Q(G_V)^{\mathsf T}\bigr ). \end {aligned} \]

                            Proof. 

                              Gauge-align the rotated factors back by \(Q^{\mathsf T}\). They are \(U_0-\eta (D_U+E_Q(G_U))\) and \(V_0-\eta (D_V+E_Q(G_V))\). Multiply, subtract \((U_0-\eta D_U)(V_0-\eta D_V)^{\mathsf T}\), and collect powers of \(\eta \).

                              Proposition 4.4 and its proof are in Appendix B.5 of The loss does not see the basis, but Adam does[singh2026lossbasis].

                              Example 9. A two-coordinate Adam defect witness [ftip-00BF]

                                Take \(d_1=d_2=1\), \(k=2\), \(f(w)=w^2/2\), \(U_0=V_0=(1,0)\), and

                                \[ Q=2^{-1/2}\begin {pmatrix}1&1\\-1&1\end {pmatrix}. \]

                                The original and rotated products after one zero-state Adam step are

                                \[ W_1=\left (1-\frac {\eta }{1+\epsilon }\right )^2, \qquad \widetilde W_1=\left (1-\frac {\eta }{2^{-1/2}+\epsilon }\right )^2. \]

                                They differ when \(0<\eta <2^{-1/2}+\epsilon \). This is the explicit witness in Proposition 4.4 of The loss does not see the basis, but Adam does[singh2026lossbasis]; it is a statement about one factored quadratic, not about language-model performance.

                                Example 10. A rotated sign update fails to commute [ftip-00BG]

                                  For \(G=(1,1)\) and the 45-degree rotation matrix \(Q\) of Example 9, one has \(GQ=(0,\sqrt 2)\). With \(\operatorname {sign}(0)=0\),

                                  \[ \operatorname {sign}(GQ)=(0,1), \qquad \operatorname {sign}(G)Q=(0,\sqrt 2). \]

                                  Thus even this two-coordinate update violates the commuting equation in Definition 1. The calculation is the explicit Appendix B.4 witness of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                  Theorem 11. Full-rank memoryless equivariant rules are Gram-determined [ftip-00BH]

                                    Let \(k\leq n\), and let \(\Phi \) be defined on full-column-rank matrices \(G\in \mathbb R^{n\times k}\). Then \(\Phi (GQ)=\Phi (G)Q\) for every \(Q\in \mathrm O(k)\) if and only if there is a matrix-valued function \(H\) of \(GG^{\mathsf T}\) such that

                                    \[ \Phi (G)=H(GG^{\mathsf T})G. \]

                                    Proof. 

                                      The displayed form is immediately equivariant. Let \(G^+\) denote the Moore--Penrose pseudoinverse. Conversely, set \(X(G)=\Phi (G)G^+\). Then \(X(G)G=\Phi (G)\) and \(X(GQ)=X(G)\). Two full-column-rank matrices with the same left Gram matrix have the same range and differ by a right orthogonal factor. Hence \(X(G)\) depends only on \(GG^{\mathsf T}\); take \(H=X\).

                                      See Theorem 4.5 and its proof in Appendix B.6 of The loss does not see the basis, but Adam does[singh2026lossbasis]. It classifies the full-rank stratum only.

                                      Corollary 12. The square invertible preconditioner representation [ftip-00BI]

                                        In the specialization \(k=n\) with \(G\) invertible, let \(P^{1/2}\) denote the unique positive-definite square root of \(P\). The function in Theorem 11 has the unique canonical representative

                                        \[ H(P)=\Phi (P^{1/2})P^{-1/2}, \qquad P=GG^{\mathsf T}>0. \]

                                        Proof. 

                                          The polar decomposition writes \(G=P^{1/2}Q\). Equivariance gives \(\Phi (G)=\Phi (P^{1/2})Q=H(P)G\). Evaluating at \(G=P^{1/2}\) proves uniqueness in this square invertible case.

                                          For rectangular \(G\), the action of \(H(P)\) away from the range needed to multiply \(G\) is not determined by \(\Phi \). The proof above establishes uniqueness in the square invertible specialization.

                                          Remark 13. What the structure theorem leaves unresolved at rank deficiency [ftip-00BJ]

                                            In the ambient matrix space, full-column-rank matrices form an open dense subset whose complement has Lebesgue measure zero. A probability-one claim requires the random gradient law to be absolutely continuous and not confined to a lower-rank set; random initialization alone does not supply that premise.

                                            Theorem 11 therefore makes no classification claim on a general rank-deficient stratum. Equivariance still constrains values within each orthogonal orbit, and it forces \(\Phi (0)=0\), but it does not determine the rule there by continuity unless continuity is separately assumed. The source's generic-rank discussion therefore does not establish a classification at rank deficiency.

                                            References

                                              Reference. The loss does not see the basis, but Adam does [singh2026lossbasis]

                                              @article{singh2026lossbasis,
  title = {The Loss Does Not See the Basis, but Adam Does},
  author = {Singh, Devender},
  year = {2026},
  url = {https://arxiv.org/abs/2608.05136v1},
  journal = {arXiv preprint arXiv:2608.05136}
}

                                              Context

                                                Optimizer geometry and representation dependence [ftip-00AU]

                                                  A scalar objective does not by itself determine a training intervention. This section studies an exact finite-dimensional witness: one represented matrix can have many factor bases, the loss can be constant along that gauge orbit, and an update rule can nevertheless distinguish the bases.

                                                  The principal source is the version-one arXiv preprint The loss does not see the basis, but Adam does[singh2026lossbasis]. Its algebraic results assume the stated dimensions, rank conditions, state conventions, and real arithmetic. Experimental observations concern the reported configurations.

                                                  Factored objectives and gauge symmetry [ftip-00AV]

                                                    We first separate a represented matrix from a choice of factors and then separate its full product-preserving symmetry from the orthogonal subgroup that preserves the Euclidean parameter metric.

                                                    Notation. Factored objective and represented matrix [ftip-00AW]

                                                      Fix integers \(d_1,d_2,k\geq 1\). Let \(U\in \mathbb R^{d_1\times k}\) and \(V\in \mathbb R^{d_2\times k}\) be factors, let

                                                      \[ W(U,V)=UV^{\mathsf T}\in \mathbb R^{d_1\times d_2}, \]

                                                      and let \(f:\mathbb R^{d_1\times d_2}\to \mathbb R\) be differentiable. The corresponding factored objective is \(L(U,V)=f(W(U,V))\). The pair \((U,V)\) is a parameterization; \(W(U,V)\) is the represented object on which \(f\) depends. This is the rectangular form noted in Section 3 of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                                      Definition. General linear gauge action [ftip-00AX]

                                                        For \(A\in \mathrm {GL}(k)\), define the general linear gauge action by

                                                        \[ \gamma _A(U,V)=(UA,VA^{-\mathsf T}). \]

                                                        The inverse transpose on the second factor is essential for a general invertible \(A\). When \(A=Q\in \mathrm O(k)\), one has \(Q^{-\mathsf T}=Q\), and the action becomes \((UQ,VQ)\).

                                                        Lemma. The gauge action preserves the product and loss [ftip-00AY]

                                                          For every \(A\in \mathrm {GL}(k)\),

                                                          \[ W(UA,VA^{-\mathsf T})=UV^{\mathsf T} \quad \hbox {and}\quad L(UA,VA^{-\mathsf T})=L(U,V). \]

                                                          Proof. 

                                                            Since \((VA^{-\mathsf T})^{\mathsf T}=A^{-1}V^{\mathsf T}\), the first identity is \(UAA^{-1}V^{\mathsf T}=UV^{\mathsf T}\). Applying \(f\) gives the second.

                                                            This is the factored-model calculation in Section 3 of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                                            Definition. Orthogonal gauge orbit [ftip-00AZ]

                                                              The orthogonal gauge orbit of \((U,V)\) is

                                                              \[ \mathcal O(U,V)=\{(UQ,VQ):Q\in \mathrm O(k)\}. \]

                                                              Every pair in the orbit represents the same \(W\). This orbit is generally smaller than the complete fibre of the map \((U,V)\mapsto UV^{\mathsf T}\). The restriction to \(\mathrm O(k)\) is geometric: it preserves the Euclidean metric on factor space.

                                                              Lemma. The orthogonal gauge is the maximal isometric subgroup [ftip-00B0]

                                                                The action \(\gamma _A\) preserves \(\lVert U\rVert _F^2+\lVert V\rVert _F^2\) for every pair \((U,V)\) if and only if \(A\in \mathrm O(k)\).

                                                                Proof. 

                                                                  If the action is an isometry, take \(V=0\) and let one row of \(U\) be an arbitrary \(u\in \mathbb R^{1\times k}\). Then \(\lVert uA\rVert _2=\lVert u\rVert _2\) for every \(u\), hence \(AA^{\mathsf T}=I\). Conversely, right multiplication by an orthogonal matrix preserves both Frobenius norms.

                                                                  See Lemma 3.2 and its proof in Appendix B.1 of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                                                  Lemma. Gradients transform covariantly along an orthogonal orbit [ftip-00B1]

                                                                    For \(Q\in \mathrm O(k)\),

                                                                    \[ \nabla _U L(UQ,VQ)=\nabla _U L(U,V)Q, \qquad \nabla _V L(UQ,VQ)=\nabla _V L(U,V)Q. \]

                                                                    Proof. 

                                                                      Put \(G=\nabla f(W)\). Then \(\nabla _U L=GV\) and \(\nabla _V L=G^{\mathsf T}U\). The represented matrix, and hence \(G\), is unchanged along the orbit. Substitution gives both identities.

                                                                      See Lemma 4.1 and its proof in Appendix B.2 of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                                                      Definition. Optimizer state action [ftip-00B2]

                                                                        Use the optimizer-state convention of Definition [ftip-001H]. For every \(Q\in \mathrm O(k)\), an optimizer state action is a map \(\sigma _Q:\mathcal S_{\rm opt}\to \mathcal S_{\rm opt}\) compatible with the shapes of the factor states and satisfying \(\sigma _Q(s_0)=s_0\) for the declared initial state.

                                                                        For example, a momentum buffer shaped like \(U\) transforms by right multiplication by \(Q\); a shared scalar second moment is fixed. An entrywise second-moment array need not admit a state action compatible with every orthogonal \(Q\).

                                                                        Definition. Gauge-equivariant optimizer [ftip-00B3]

                                                                          Let \(\mathcal A\) be a deterministic instance of the parameter-update rule of Definition [ftip-001I], acting on \((U,V,s)\). It is gauge-equivariant when for every \(Q\in \mathrm O(k)\) there is a state action \(\sigma _Q\) such that gauge-related initial states produce

                                                                          \[ (\widetilde U_t,\widetilde V_t,\widetilde s_t) =(U_tQ,V_tQ,\sigma _Q(s_t)) \qquad (t\geq 0). \]

                                                                          This is Definition 3.1 of The loss does not see the basis, but Adam does[singh2026lossbasis]. Any randomness, schedules, stopping rule, and mixed update blocks must also be coupled as required by Remark [ftip-001J]; the definition does not hide them.

                                                                          Example. A commuting optimizer square [ftip-00B4]

                                                                            One update of a gauge-equivariant optimizer makes the following square commute. The vertical arrows change factor basis and the horizontal arrows apply the same declared update.

                                                                            Commutation is stronger than equality of the scalar losses before the step. It states equality of the represented product trajectory after aligning the parameter bases.

                                                                            Remark. Product symmetry is larger than the isometric gauge [ftip-00B5]

                                                                              The full \(\mathrm {GL}(k)\) action in Definition [ftip-00AX] preserves \(W\), but ordinary Euclidean gradients are covariant under the orthogonal specialization used in Lemma [ftip-00B1]. A theorem for \((UQ,VQ)\) must therefore not be silently promoted to every product-preserving reparameterization.

                                                                              The source calls the orthogonal subgroup the gauge in its optimizer classification while also noting the larger function-preserving group. These are compatible statements only when their different geometric scopes remain explicit.

                                                                              Equivariant and basis-sensitive update rules [ftip-00B6]
                                                                              ✍️source

                                                                                Gradient covariance supplies a test for update rules. Linear and full-matrix operations can commute with a latent rotation, whereas a fixed nonlinear map applied separately to coordinates selects a preferred basis.

                                                                                Definition. Memoryless right-equivariant update [ftip-00B7]

                                                                                  A memoryless factor-update map is a function \(\Phi :\mathbb R^{n\times k}\to \mathbb R^{n\times k}\) applied to a current gradient \(G\) without a persistent optimizer state. It is right-equivariant when

                                                                                  \[ \Phi (GQ)=\Phi (G)Q \qquad (G\in \mathbb R^{n\times k},\ Q\in \mathrm O(k)). \]

                                                                                  This local definition is the input of Theorem 4.5 in The loss does not see the basis, but Adam does[singh2026lossbasis]. It does not classify stateful optimizers such as Adam.

                                                                                  proposition. Gradient descent and momentum preserve the orthogonal gauge [ftip-00B8]

                                                                                    Gradient descent and Polyak or Nesterov momentum are gauge-equivariant when their factor gradients, momentum buffers, schedules, and look-ahead points are transformed consistently.

                                                                                    Proof. 

                                                                                      By Lemma [ftip-00B1], \(G_t\) becomes \(G_tQ\). Gradient descent therefore sends \(U_tQ\) to \((U_t-\eta _tG_t)Q\). A momentum buffer initialized at zero and formed by linear combinations of covariant gradients transforms as \(M_tQ\); a Nesterov look-ahead point is likewise the reference look-ahead right-multiplied by \(Q\). Induction proves the claim.

                                                                                      Proposition 4.2(1) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                                                                      proposition. A shared scalar second moment preserves the orthogonal gauge [ftip-00B9]

                                                                                        Replace Adam's entrywise second-moment denominator by one scalar whose updates depend only on gauge-invariant quantities such as the pooled mean of the squared entries of both factor gradients. With a covariant first-moment buffer, the resulting stateful update is gauge-equivariant.

                                                                                        Proof. 

                                                                                          The first moment is a linear exponential average and transforms by right multiplication by \(Q\). The pooled squared-entry mean is the joint squared Frobenius norm divided by the entry count, hence is unchanged by \(G\mapsto GQ\). Its bias correction and positive scalar denominator are unchanged, so the complete update transforms covariantly.

                                                                                          Proposition 4.2(2) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis]. The result concerns this declared shared-scalar variant, not stock Adam.

                                                                                          proposition. Full-matrix Muon and Shampoo preserve the orthogonal gauge [ftip-00BA]

                                                                                            Fix one factor, write \(n\in \{d_1,d_2\}\) for its row dimension, and let \(G_t,M_t\in \mathbb R^{n\times k}\) be its gradient and Muon momentum buffer. For \(0\leq \beta <1\), \(M_{-1}=0\), and the source's linear momentum update \(M_t=\beta M_{t-1}+G_t\), Muon's direction is \(\Delta _t=\operatorname {msign}(M_t)\). Under real arithmetic this stateful update and the source's damped Shampoo update are right-equivariant. The Muon claim includes the stated finite Newton--Schulz approximations; the Shampoo claim uses its complete left and right accumulators.

                                                                                            Proof. 

                                                                                              By Lemma [ftip-00B1], the primed gradient is \(G'_t=G_tQ\). Starting from \(M'_{-1}=M_{-1}Q=0\), induction through the linear momentum update gives \(M'_t=M_tQ\). If \(M_t=A\Sigma B^{\mathsf T}\) is a compact singular-value decomposition, then \(M_tQ=A\Sigma (Q^{\mathsf T}B)^{\mathsf T}\), so \(\operatorname {msign}(M_tQ)=\operatorname {msign}(M_t)Q\). For \(\epsilon _{\rm NS}>0\), the Newton--Schulz initialization \(X_0=M_t/(\lVert M_t\rVert _F+\epsilon _{\rm NS})\) is covariant because its Frobenius normalization is invariant; each polynomial iterate formed from \(X_j\) and \(X_jX_j^{\mathsf T}\) preserves the same covariance. For Shampoo, the left accumulator is invariant while the right accumulator transforms by \(R_t\mapsto Q^{\mathsf T} R_t Q\). Orthogonal functional calculus conjugates the right matrix function, so the transformed update is the original update right-multiplied by \(Q\).

                                                                                              Proposition 4.2(3)--(4) and its proof are in Appendix B.3 of The loss does not see the basis, but Adam does[singh2026lossbasis]. Variable splitting, coordinatewise clipping, mixed update rules, finite precision, and different damping conventions lie outside this statement. The Muon part is stateful; it does not replace the momentum buffer by the current gradient or invoke the memoryless classification of Definition [ftip-00B7].

                                                                                              proposition. Coordinatewise equivariance forces linearity [ftip-00BB]

                                                                                                Let \(k\geq 2\) and let a memoryless update have the fixed entrywise form \(\Phi (G)_{ij}=\phi (G_{ij})\). If \(\Phi (GQ)=\Phi (G)Q\) for every \(G\) and \(Q\in \mathrm O(k)\), then \(\phi (x)=cx\) for some \(c\in \mathbb R\).

                                                                                                Proof. 

                                                                                                  It suffices to use one row and rotations in the first two coordinates. Equivariance at zero gives \(\phi (0)=0\). Rotating \((x,0)\) through an angle whose cosine is \(a\in [-1,1]\) gives \(\phi (ax)=a\phi (x)\). For nonzero \(x,y\), compare each with a third number of at least their absolute magnitudes; the ratio \(\phi (x)/x\) is constant.

                                                                                                  Proposition 4.3 and its proof are in Appendix B.4 of The loss does not see the basis, but Adam does[singh2026lossbasis]. No continuity assumption is needed.

                                                                                                  Corollary. Fixed nonlinear coordinatewise updates break the gauge [ftip-00BC]

                                                                                                    A nonlinear fixed entrywise update applied at a zero optimizer state is not gauge-equivariant when \(k\geq 2\). In particular, the source applies proposition [ftip-00BB] to the first-step maps of Adam, RMSProp, signSGD, and Lion; it checks Adafactor separately because its factored statistics are not an entrywise map.

                                                                                                    This is a first-step obstruction. It suffices to disprove equivariance of the full run, but it does not classify every later state or say that such an optimizer cannot reach a particular solution.

                                                                                                    Definition. First-step Adam map and gauge defect [ftip-00BD]

                                                                                                      Fix \(\epsilon >0\). For zero-state, bias-corrected Adam define the entrywise first-step direction

                                                                                                      \[ D_\epsilon (G)=G\mathbin {\oslash }(|G|+\epsilon ) \]

                                                                                                      and, for \(Q\in \mathrm O(k)\), define its gauge-aligned defect by

                                                                                                      \[ E_Q(G)=D_\epsilon (GQ)Q^{\mathsf T}-D_\epsilon (G). \]

                                                                                                      Here \(|G|\), addition, and division are entrywise. Bias correction makes the first direction independent of Adam's two decay coefficients under the declared convention [singh2026lossbasis, Proposition 4.4].

                                                                                                      proposition. Exact represented-matrix defect after one Adam step [ftip-00BE]

                                                                                                        Let \(G_U=\nabla _UL(U_0,V_0)\), \(G_V=\nabla _VL(U_0,V_0)\), \(D_U=D_\epsilon (G_U)\), and \(D_V=D_\epsilon (G_V)\). Let \(W_1\) be the product after one Adam step from \((U_0,V_0)\) and \(\widetilde W_1\) the product after one step from \((U_0Q,V_0Q)\). Then

                                                                                                        \[ \begin {aligned} \widetilde W_1-W_1={}&-\eta \bigl (E_Q(G_U)V_0^{\mathsf T} +U_0E_Q(G_V)^{\mathsf T}\bigr )\\ &+\eta ^2\bigl (E_Q(G_U)D_V^{\mathsf T} +D_UE_Q(G_V)^{\mathsf T} +E_Q(G_U)E_Q(G_V)^{\mathsf T}\bigr ). \end {aligned} \]

                                                                                                        Proof. 

                                                                                                          Gauge-align the rotated factors back by \(Q^{\mathsf T}\). They are \(U_0-\eta (D_U+E_Q(G_U))\) and \(V_0-\eta (D_V+E_Q(G_V))\). Multiply, subtract \((U_0-\eta D_U)(V_0-\eta D_V)^{\mathsf T}\), and collect powers of \(\eta \).

                                                                                                          Proposition 4.4 and its proof are in Appendix B.5 of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                                                                                          Example. A two-coordinate Adam defect witness [ftip-00BF]

                                                                                                            Take \(d_1=d_2=1\), \(k=2\), \(f(w)=w^2/2\), \(U_0=V_0=(1,0)\), and

                                                                                                            \[ Q=2^{-1/2}\begin {pmatrix}1&1\\-1&1\end {pmatrix}. \]

                                                                                                            The original and rotated products after one zero-state Adam step are

                                                                                                            \[ W_1=\left (1-\frac {\eta }{1+\epsilon }\right )^2, \qquad \widetilde W_1=\left (1-\frac {\eta }{2^{-1/2}+\epsilon }\right )^2. \]

                                                                                                            They differ when \(0<\eta <2^{-1/2}+\epsilon \). This is the explicit witness in Proposition 4.4 of The loss does not see the basis, but Adam does[singh2026lossbasis]; it is a statement about one factored quadratic, not about language-model performance.

                                                                                                            Example. A rotated sign update fails to commute [ftip-00BG]

                                                                                                              For \(G=(1,1)\) and the 45-degree rotation matrix \(Q\) of Example [ftip-00BF], one has \(GQ=(0,\sqrt 2)\). With \(\operatorname {sign}(0)=0\),

                                                                                                              \[ \operatorname {sign}(GQ)=(0,1), \qquad \operatorname {sign}(G)Q=(0,\sqrt 2). \]

                                                                                                              Thus even this two-coordinate update violates the commuting equation in Definition [ftip-00B7]. The calculation is the explicit Appendix B.4 witness of The loss does not see the basis, but Adam does[singh2026lossbasis].

                                                                                                              Theorem. Full-rank memoryless equivariant rules are Gram-determined [ftip-00BH]

                                                                                                                Let \(k\leq n\), and let \(\Phi \) be defined on full-column-rank matrices \(G\in \mathbb R^{n\times k}\). Then \(\Phi (GQ)=\Phi (G)Q\) for every \(Q\in \mathrm O(k)\) if and only if there is a matrix-valued function \(H\) of \(GG^{\mathsf T}\) such that

                                                                                                                \[ \Phi (G)=H(GG^{\mathsf T})G. \]

                                                                                                                Proof. 

                                                                                                                  The displayed form is immediately equivariant. Let \(G^+\) denote the Moore--Penrose pseudoinverse. Conversely, set \(X(G)=\Phi (G)G^+\). Then \(X(G)G=\Phi (G)\) and \(X(GQ)=X(G)\). Two full-column-rank matrices with the same left Gram matrix have the same range and differ by a right orthogonal factor. Hence \(X(G)\) depends only on \(GG^{\mathsf T}\); take \(H=X\).

                                                                                                                  See Theorem 4.5 and its proof in Appendix B.6 of The loss does not see the basis, but Adam does[singh2026lossbasis]. It classifies the full-rank stratum only.

                                                                                                                  Corollary. The square invertible preconditioner representation [ftip-00BI]

                                                                                                                    In the specialization \(k=n\) with \(G\) invertible, let \(P^{1/2}\) denote the unique positive-definite square root of \(P\). The function in Theorem [ftip-00BH] has the unique canonical representative

                                                                                                                    \[ H(P)=\Phi (P^{1/2})P^{-1/2}, \qquad P=GG^{\mathsf T}>0. \]

                                                                                                                    Proof. 

                                                                                                                      The polar decomposition writes \(G=P^{1/2}Q\). Equivariance gives \(\Phi (G)=\Phi (P^{1/2})Q=H(P)G\). Evaluating at \(G=P^{1/2}\) proves uniqueness in this square invertible case.

                                                                                                                      For rectangular \(G\), the action of \(H(P)\) away from the range needed to multiply \(G\) is not determined by \(\Phi \). The proof above establishes uniqueness in the square invertible specialization.

                                                                                                                      Remark. What the structure theorem leaves unresolved at rank deficiency [ftip-00BJ]

                                                                                                                        In the ambient matrix space, full-column-rank matrices form an open dense subset whose complement has Lebesgue measure zero. A probability-one claim requires the random gradient law to be absolutely continuous and not confined to a lower-rank set; random initialization alone does not supply that premise.

                                                                                                                        Theorem [ftip-00BH] therefore makes no classification claim on a general rank-deficient stratum. Equivariance still constrains values within each orthogonal orbit, and it forces \(\Phi (0)=0\), but it does not determine the rule there by continuity unless continuity is separately assumed. The source's generic-rank discussion therefore does not establish a classification at rank deficiency.

                                                                                                                        Common-scalar flows, clock changes, and momentum [ftip-00BK]

                                                                                                                          One narrow class of continuous-time preconditioners follows the gradient-flow path at a different speed. The clock change transfers path properties, but not rates, and it does not encompass a stateful momentum method.

                                                                                                                          Definition. Common-scalar preconditioned flow [ftip-00BL]

                                                                                                                            Let \(\theta =(U,V)\) and let \(a(t)>0\) be measurable with \(1/a\) locally integrable. A common-scalar preconditioned flow is a locally absolutely continuous trajectory satisfying

                                                                                                                            \[ \dot \theta (t)=-\frac {\nabla L(\theta (t))}{a(t)} \]

                                                                                                                            for almost every \(t\). The same scalar multiplies every coordinate of both factor gradients. For gauge-uniform conclusions, the source additionally requires \(a(t)\) to be determined by gauge-invariant statistics of the trajectory up to time \(t\).

                                                                                                                            Definition. Effective optimizer clock [ftip-00BM]

                                                                                                                              For the flow of Definition [ftip-00BL], define the effective optimizer clock

                                                                                                                              \[ \tau (t)=\int _0^t\frac {du}{a(u)}, \qquad \tau _{\max }=\int _0^\infty \frac {du}{a(u)}, \qquad 0<\tau _{\max }\leq \infty . \]

                                                                                                                              The clock is strictly increasing on its domain. If \(\tau _{\max }=\infty \), it reaches every gradient-flow time; otherwise it traverses only the prefix with effective time below \(\tau _{\max }\).

                                                                                                                              Theorem. Common-scalar flow is a time-reparameterized gradient flow [ftip-00BN]

                                                                                                                                Let \(\theta (t)\) satisfy Definition [ftip-00BL], let \(t(\tau )\) be the inverse of the clock in Definition [ftip-00BM], and put \(\widetilde \theta (\tau )=\theta (t(\tau ))\). Then, for almost every \(\tau \),

                                                                                                                                \[ \widetilde \theta '(\tau )=-\nabla L(\widetilde \theta (\tau )). \]

                                                                                                                                Proof. 

                                                                                                                                  The chain rule for absolutely continuous changes of variable gives \(dt/d\tau =a(t)\) almost everywhere. Multiplying \(\dot \theta =-\nabla L/a(t)\) by \(dt/d\tau \) yields the displayed gradient flow.

                                                                                                                                  See Theorem 4.6 and its proof in Appendix B.9 of The loss does not see the basis, but Adam does[singh2026lossbasis]. Gauge invariance of \(a\) is not needed for this single-trajectory clock identity; it makes the clock common across a gauge orbit.

                                                                                                                                  Corollary. Transfer of path and limit properties [ftip-00BO]

                                                                                                                                    Every property depending only on the portion of the gradient-flow path traversed before \(\tau _{\max }\) transfers to the common-scalar flow. If \(\tau _{\max }=\infty \) and gradient flow converges, both flows have the same limit point. A sufficient condition for clock divergence is an eventual finite upper bound on \(a(t)\).

                                                                                                                                    For shallow factorization, the source cites Implicit regularization in matrix factorization[gunasekar2017implicit]. For deep factorization, it cites Implicit regularization in deep matrix factorization[arora2019implicit]. For greedy low-rank dynamics, it cites Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning[li2021greedy].

                                                                                                                                    Those conclusions transfer only when both this clock theorem and every hypothesis of the original result hold. This corollary supplies no missing matrix-sensing assumption.

                                                                                                                                    Remark. Rates and hitting times do not survive time reparameterization [ftip-00BP]

                                                                                                                                      Two flows can traverse the same curve with arbitrarily different physical clocks. A convergence rate, finite-step budget, or hitting time stated in \(t\) therefore does not transfer through Theorem [ftip-00BN] without quantitative bounds on \(a\) and its integral. If \(\tau _{\max }<\infty \), even the tail and limit of the gradient-flow path are not reached.

                                                                                                                                      Remark. Scalar-Adam is outside the common-scalar flow theorem [ftip-00BQ]

                                                                                                                                        The shared-scalar Adam variant in proposition [ftip-00B9] retains a first-moment exponential moving average. It is stateful and is not the memoryless flow in Definition [ftip-00BL]. Remark 4.7 of The loss does not see the basis, but Adam does[singh2026lossbasis] treats its agreement with gradient flow as empirical rather than as a consequence of Theorem 4.6.

                                                                                                                                        The theorem likewise does not apply to discrete Adam, PPO, RLVR update loops, clipping, weight decay, and mixed optimizers. Gauge equivariance and time reparameterization are separate properties.

                                                                                                                                        Example. Equivariance neither guarantees nor precludes recovery [ftip-00BR]

                                                                                                                                          The source records controls on both sides of the proposed implication. Its ScaledGD-inspired control is equivariant but equalizes the spectral schedule and does not recover the planted low-rank target at the declared budget. A long-anneal sign update is non-equivariant but eventually reaches a low-recovery-error regime.

                                                                                                                                          The controls are reported in Remark 4.7 and Appendix D.10 of The loss does not see the basis, but Adam does[singh2026lossbasis]. The source reports no tuned reproducibility baseline for the ScaledGD control. The diagram is not a convergence theorem.

                                                                                                                                          Optimizer geometry in training interventions [ftip-00BS]

                                                                                                                                            Equivalent parameterizations need not follow equivalent training trajectories. Their evolution also depends on the update geometry; the scalar loss and represented starting object do not always suffice.

                                                                                                                                            Definition. Objective-equivalent parameterizations [ftip-00BT]

                                                                                                                                              Two factor pairs \((U,V)\) and \((\widetilde U,\widetilde V)\) are objective-equivalent for \(L=f\circ W\) when

                                                                                                                                              \[ W(U,V)=W(\widetilde U,\widetilde V). \]

                                                                                                                                              They then have the same scalar loss for every objective depending only on that represented matrix. Orthogonal gauge-related pairs are objective-equivalent by Lemma [ftip-00AY]; objective equivalence need not imply membership in the same orthogonal orbit.

                                                                                                                                              Definition. Parameterization-stable training intervention [ftip-00BU]

                                                                                                                                                Fix a factorized objective, an update algorithm, hyperparameter schedule, randomness coupling, stopping rule, and initial optimizer state. The resulting training intervention is parameterization-stable on an orthogonal orbit when every two gauge-related initial states have identical represented trajectories:

                                                                                                                                                \[ \widetilde U_t\widetilde V_t^{\mathsf T}=U_tV_t^{\mathsf T} \qquad \hbox {at every compared step }t. \]

                                                                                                                                                This is a local stability property of a fully declared intervention. It does not say that arbitrary objective-equivalent pairs outside the orbit must agree.

                                                                                                                                                proposition. Gauge equivariance gives parameterization stability [ftip-00BV]

                                                                                                                                                  Every gauge-equivariant optimizer is parameterization-stable on each orthogonal orbit, provided the remaining protocol coordinates are coupled as in Definition [ftip-00BU].

                                                                                                                                                  Proof. 

                                                                                                                                                    Definition [ftip-00B3] gives \((\widetilde U_t,\widetilde V_t)=(U_tQ,V_tQ)\). Hence

                                                                                                                                                    \[ \widetilde U_t\widetilde V_t^{\mathsf T} =U_tQQ^{\mathsf T}V_t^{\mathsf T}=U_tV_t^{\mathsf T}. \]

                                                                                                                                                    This is a sufficient condition on a declared orbit, not a claim that equivariance is necessary for every possible equality of represented trajectories.

                                                                                                                                                    proposition. A first-step defect falsifies a loss-only protocol specification [ftip-00BW]

                                                                                                                                                      Suppose two orthogonally gauge-related initializations have equal represented object and scalar objective, but a declared update produces different represented objects after one step. Then the training intervention is not determined by the represented initialization and scalar objective alone.

                                                                                                                                                      Proof. 

                                                                                                                                                        If those two objects determined the intervention, equal inputs would give the same represented next state. The witnessed inequality contradicts that factorization.

                                                                                                                                                        Proposition proposition [ftip-00BE] and Example [ftip-00BF] instantiate its premise for one factored quadratic and zero-state Adam. It proves underspecification, not a capability difference.

                                                                                                                                                        Example. The orthogonal gauge of an attention head [ftip-00BX]

                                                                                                                                                          For the row-vector attention weights of Definition [ftip-000P], let \(W^Q,W^K\in \mathbb R^{d\times d_h}\) and \(Q_h=XW^Q\), \(K_h=XW^K\). Attention logits depend on these weights through

                                                                                                                                                          \[ Q_hK_h^{\mathsf T}=XW^Q(W^K)^{\mathsf T}X^{\mathsf T}. \]

                                                                                                                                                          For \(A\in \mathrm O(d_h)\), the right action \((W^Q,W^K)\mapsto (W^QA,W^KA)\) preserves the middle product.

                                                                                                                                                          Section 6 of The loss does not see the basis, but Adam does[singh2026lossbasis] uses column-vector weights and the invariant \(W_Q^{\mathsf T}W_K\). Transposition gives the row-vector invariant \(W^Q(W^K)^{\mathsf T}\), with the action on the right as displayed.

                                                                                                                                                          Remark. What the attention-twin experiment observes [ftip-00BY]

                                                                                                                                                            Section 6 and Table 3 of The loss does not see the basis, but Adam does[singh2026lossbasis] compare two function-equivalent initializations of a two-layer, four-head transformer on modular addition. The reported relative logit distance after one Adam step is \(3.6\times 10^{-3}\), versus \(2.9\times 10^{-7}\) for a same-basis noise twin; the final per-head invariant distance is reported as 56 percent.

                                                                                                                                                            The exact first-step defect has an algebraic explanation, but later drift, scale robustness, and downstream behavior are empirical. The tested small transformers, precision, optimizer allocation, seeds, and task do not establish a frontier-wide law.

                                                                                                                                                            Example. One loss quotient and two optimizer trajectories [ftip-00BZ]

                                                                                                                                                              A loss-only description collapses a gauge orbit to one represented starting point. A basis-sensitive optimizer can split that quotient back into different represented trajectories.

                                                                                                                                                              The right-hand split is possible only because the centre box omits the factor basis and optimizer geometry. An FTIP admissible protocol in Definition [ftip-005J] must retain those intervention coordinates when they affect the executable result.

                                                                                                                                                              Remark. Product symmetry and the scope of optimizer conclusions [ftip-00C0]

                                                                                                                                                                The relevant optimizer results appear in version 1 of [⧉], submitted 5 August 2026: Sections 3--4 and 6, Appendices B.1--B.6 and B.9, with limitations in Section 11.

                                                                                                                                                                The full product symmetry acts on the factors by

                                                                                                                                                                \[ (U,V)\longmapsto (UA,VA^{-\mathsf T}). \]

                                                                                                                                                                It does not use \((UA,VA)\). For row-vector attention weights, the corresponding action uses right multiplication, as in Example [ftip-00BX].

                                                                                                                                                                The general rectangular preconditioner representation need not have the square-case uniqueness. A probability-one full-rank claim additionally needs a law for the gradient; a generic full-rank observation does not specify one.

                                                                                                                                                                The coordinatewise theorem is a zero-state, memoryless obstruction. The clock theorem is continuous-time and excludes scalar-Adam's momentum. Experimental optimizer and attention results remain observations reported by the source. Equivariance is not identified with low-rank recovery.

                                                                                                                                                                Remark. Continuous-flow conclusions and discrete optimization [ftip-00C1]

                                                                                                                                                                  The factored-model results prove that update geometry can be a genuine intervention coordinate even when objective and represented initialization are fixed. They do not prove transformer or RLVR convergence, practical optimizer superiority, a scaling law, or any change in reliable capability. Different represented trajectories are not by themselves an acquisition witness of Definition [ftip-005X].

                                                                                                                                                                  The memoryless theorem does not describe Adam's full state, and the continuous-flow result does not supply discrete rates or finite-step bounds. The source also notes that fixed-rank LoRA changes the learned-rank mechanism; no LoRA transfer is made here.

                                                                                                                                                                  A dynamic-compute comparison also depends on active research state, persistent harness state, retained history, evaluation configuration, and descendant and retry costs. One realized chain and an archive-best envelope are different outcomes, even at the same declared budget.