Example. A tokenizer round trip and normalization failure [ftip-000F]
Example. A tokenizer round trip and normalization failure [ftip-000F]
A four-stage toy tokenizer separates token-level round-trip stability from recovery of the original string.
Here underscores display spaces. Declare \(N(A\_\_B)=a\_b\), \(T(a\_b)=(a,b)\), and \(D(a,b)=a\_b\). For \(s=A\_\_B\), \[ D(T(N(s)))=a\_b\neq s, \qquad T(N(D(T(N(s)))))=(a,b). \] The token sequence is stable after normalization although the original case and repeated space are not recovered.
The source-grounded tokenizer interface is Definition [ftip-000E]. The displayed normalization rule belongs only to this finite construction; no such lossy transformation is attributed to SentencePiece.