proposition. Coordinatewise equivariance forces linearity [ftip-00BB]
✍️source

    Let \(k\geq 2\) and let a memoryless update have the fixed entrywise form \(\Phi (G)_{ij}=\phi (G_{ij})\). If \(\Phi (GQ)=\Phi (G)Q\) for every \(G\) and \(Q\in \mathrm O(k)\), then \(\phi (x)=cx\) for some \(c\in \mathbb R\).

    Proof. 

      It suffices to use one row and rotations in the first two coordinates. Equivariance at zero gives \(\phi (0)=0\). Rotating \((x,0)\) through an angle whose cosine is \(a\in [-1,1]\) gives \(\phi (ax)=a\phi (x)\). For nonzero \(x,y\), compare each with a third number of at least their absolute magnitudes; the ratio \(\phi (x)/x\) is constant.

      Proposition 4.3 and its proof are in Appendix B.4 of The loss does not see the basis, but Adam does[singh2026lossbasis]. No continuity assumption is needed.