Let \(\Sigma \) be a finite character or byte alphabet and let
\(\mathcal V\) be the vocabulary of Definition [ftip-000D]. A
tokenizer is a specified map
\(\tau :\Sigma ^*\to \mathcal V^*\). A detokenizer is a specified map
\(\delta :\mathcal V^*\to \Sigma ^*\).
A tokenizer package also fixes its text normalization rule \(N\). When the
package promises normalized round trips, the relevant condition is
\(\delta (\tau (s))=N(s)\) for text \(s\in \Sigma ^*\); it is not the claim that
\(\tau \) and \(\delta \) are inverse bijections on all sequences. SentencePiece
keeps the segmentation model and detokenization convention together, which is
why both maps belong to the model interface.
Byte-pair encoding gives another construction of \(\tau \). Learned merge
operations turn an initial symbol sequence into subword units; see
[sennrich2016neural, §3.2]. The learned merges and base alphabet are
therefore part of the tokenizer specification.