Lemma. The orthogonal gauge is the maximal isometric subgroup [ftip-00B0]
✍️source

    The action \(\gamma _A\) preserves \(\lVert U\rVert _F^2+\lVert V\rVert _F^2\) for every pair \((U,V)\) if and only if \(A\in \mathrm O(k)\).

    Proof. 

      If the action is an isometry, take \(V=0\) and let one row of \(U\) be an arbitrary \(u\in \mathbb R^{1\times k}\). Then \(\lVert uA\rVert _2=\lVert u\rVert _2\) for every \(u\), hence \(AA^{\mathsf T}=I\). Conversely, right multiplication by an orthogonal matrix preserves both Frobenius norms.

      See Lemma 3.2 and its proof in Appendix B.1 of The loss does not see the basis, but Adam does[singh2026lossbasis].