For a finite set \(X\), write \(|X|\) for its cardinality and \(X^*\) for the set of finite sequences with entries in \(X\). A map \(f:X\to Y\) sends \(x\in X\) to \(f(x)\in Y\); the inverse image of \(A\subseteq Y\) is \(f^{-1}(A)=\{x\in X:f(x)\in A\}\).

Write \[ \Delta (X)=\left \{p:X\to [0,1]:\sum _{x\in X}p(x)=1\right \} \] for the probability simplex on \(X\). If \(p\in \Delta (X)\) and \(Z:X\to \mathbb R\), then \[ \mathbb E_{x\sim p}[Z(x)]=\sum _{x\in X}p(x)Z(x). \] A random variable is a map from the sample set to its value set. Random variables \(Z\) and \(W\) are independent under \(p\) when \(p(Z=z,W=w)=p(Z=z)p(W=w)\) for every pair of values \(z,w\).

For nonfinite interaction spaces, probability laws and conditional kernels require the measurable structure described in Remark 3.

Using the notation of Notation 1, a finite probability law on \(X\) is an element \(p\in \Delta (X)\). For an event \(A\subseteq X\), its probability is \[ p(A)=\sum _{x\in A}p(x). \] For \(p(A)>0\), the conditional law on \(B\subseteq X\) is \(p(B\mid A)=p(A\cap B)/p(A)\).

A finite vocabulary and a bounded token sequence admit probability laws expressed as ordinary sums, with explicit normalization and conditioning. The finite-sequence interface matches [phuong2022formal, sec. 3, Sequence modelling].

States, observations, and tool outputs need not be finite. In that setting \(\Delta (X)\) is replaced by probability measures on a declared measurable space and conditional laws are probability kernels. Results on finite spaces need additional measurable-space hypotheses before they apply in this setting.

A finite vocabulary is a nonempty finite set \(\mathcal V\) of token identifiers. A token identifier is an atomic symbol for the sequence model; it need not be a word or a character.

When termination is represented in-band, choose a distinguished token \(\mathtt {eos}\in \mathcal V\). Other control tokens, if present, are named as elements of the same vocabulary rather than assumed implicitly.

Let \(\Sigma \) be a finite character or byte alphabet and let \(\mathcal V\) be the vocabulary of Definition 4. A tokenizer is a specified map \(\tau :\Sigma ^*\to \mathcal V^*\). A detokenizer is a specified map \(\delta :\mathcal V^*\to \Sigma ^*\).

A tokenizer package also fixes its text normalization rule \(N\). When the package promises normalized round trips, the relevant condition is \(\delta (\tau (s))=N(s)\) for text \(s\in \Sigma ^*\); it is not the claim that \(\tau \) and \(\delta \) are inverse bijections on all sequences. SentencePiece keeps the segmentation model and detokenization convention together, which is why both maps belong to the model interface.

Byte-pair encoding gives another construction of \(\tau \). Learned merge operations turn an initial symbol sequence into subword units; see [sennrich2016neural, §3.2]. The learned merges and base alphabet are therefore part of the tokenizer specification.

A four-stage toy tokenizer separates token-level round-trip stability from recovery of the original string.

Here underscores display spaces. Declare \(N(A\_\_B)=a\_b\), \(T(a\_b)=(a,b)\), and \(D(a,b)=a\_b\). For \(s=A\_\_B\), \[ D(T(N(s)))=a\_b\neq s, \qquad T(N(D(T(N(s)))))=(a,b). \] The token sequence is stable after normalization although the original case and repeated space are not recovered.

The source-grounded tokenizer interface is Definition 5. The displayed normalization rule belongs only to this finite construction; no such lossy transformation is attributed to SentencePiece.

Let \(\mathcal V\) be the vocabulary of Definition 4. A token sequence of length \(n\geq 0\) is written \(x_{1:n}=(x_1,\ldots ,x_n)\in \mathcal V^n\); positions are one-based. Its length is \(|x_{1:n}|=n\), its prefix before position \(t\) is \(x_{<t}=x_{1:t-1}\), and \(x_{<1}\) is the empty sequence. Juxtaposition \(xy\) denotes concatenation.

If \(\mathtt {eos}\) is distinguished, a completed generated continuation \(y_{1:m}\) ends with \(y_m=\mathtt {eos}\) and contains no earlier \(\mathtt {eos}\). A fixed length limit can also stop generation; the stopping mechanism must state which convention it uses.

Let \(\Theta \) be a parameter set. With the probability notation of Notation 1 and the sequence convention of Convention 7, a conditional next-token law is a family of maps \[ \pi _\theta :\mathcal V^*\longrightarrow \Delta (\mathcal V), \qquad \theta \in \Theta . \] For a prefix \(x_{<t}\) and token \(a\in \mathcal V\), the number \(\pi _\theta (a\mid x_{<t})\) is the probability assigned to choosing \(a\) at position \(t\).

This is the observable probabilistic interface. It does not yet specify the neural computation that produces the distribution.

Fix the next-token law of Definition 8, a prompt \(x\in \mathcal V^*\), and a continuation \(y_{1:m}\in \mathcal V^m\). Its autoregressive continuation probability is \[ P_\theta (y_{1:m}\mid x) =\prod _{t=1}^{m} \pi _\theta \left (y_t\mid x y_{<t}\right ). \] Each factor conditions only on the prompt and earlier continuation tokens.

If generation stops on \(\mathtt {eos}\), the law of a completed continuation is restricted to the sequences specified in Convention 7. If generation instead stops at a length limit, the terminal event and its probability must be recorded separately.

Along one branch of a three-symbol prefix tree, conditional token laws determine both continuation probability and negative log-likelihood.

Let the vocabulary be \(\{a,b,\mathtt {EOS}\}\) and let \(p_i\) be the displayed conditional probability at step \(i\). The continuation \((b,a,\mathtt {EOS})\) has \[ \Pr (b,a,\mathtt {EOS}\mid a) =p_1p_2p_3, \qquad -\log \Pr =-\sum _{i=1}^{3}\log p_i. \]

The multiplication follows the causal product in [phuong2022formal, Section 3]. Its scope is one fixed finite continuation, leaving both the decoding algorithm and the law over all variable-length outputs unspecified.