Definition. Tokenizer and detokenizer [kudo2018sentencepiece, secs. 3.1 and 3.5] [ftip-000E]
Definition. Tokenizer and detokenizer [kudo2018sentencepiece, secs. 3.1 and 3.5] [ftip-000E]
Let \(\Sigma \) be a finite character or byte alphabet and let \(\mathcal V\) be the vocabulary of Definition [ftip-000D]. A tokenizer is a specified map \(\tau :\Sigma ^*\to \mathcal V^*\). A detokenizer is a specified map \(\delta :\mathcal V^*\to \Sigma ^*\).
A tokenizer package also fixes its text normalization rule \(N\). When the package promises normalized round trips, the relevant condition is \(\delta (\tau (s))=N(s)\) for text \(s\in \Sigma ^*\); it is not the claim that \(\tau \) and \(\delta \) are inverse bijections on all sequences. SentencePiece keeps the segmentation model and detokenization convention together, which is why both maps belong to the model interface.
Byte-pair encoding gives another construction of \(\tau \). Learned merge operations turn an initial symbol sequence into subword units; see [sennrich2016neural, §3.2]. The learned merges and base alphabet are therefore part of the tokenizer specification.