Definition. Finite vocabulary [phuong2022formal, sec. 3, Sequence modelling] [ftip-000D]
✍️source

    A finite vocabulary is a nonempty finite set \(\mathcal V\) of token identifiers. A token identifier is an atomic symbol for the sequence model; it need not be a word or a character.

    When termination is represented in-band, choose a distinguished token \(\mathtt {eos}\in \mathcal V\). Other control tokens, if present, are named as elements of the same vocabulary rather than assumed implicitly.