Lemma. Gradients transform covariantly along an orthogonal orbit [ftip-00B1]
✍️source

    For \(Q\in \mathrm O(k)\),

    \[ \nabla _U L(UQ,VQ)=\nabla _U L(U,V)Q, \qquad \nabla _V L(UQ,VQ)=\nabla _V L(U,V)Q. \]

    Proof. 

      Put \(G=\nabla f(W)\). Then \(\nabla _U L=GV\) and \(\nabla _V L=G^{\mathsf T}U\). The represented matrix, and hence \(G\), is unchanged along the orbit. Substitution gives both identities.

      See Lemma 4.1 and its proof in Appendix B.2 of The loss does not see the basis, but Adam does[singh2026lossbasis].