Convention. Token sequences, prefixes, positions, and end-of-sequence [ftip-000G]
Convention. Token sequences, prefixes, positions, and end-of-sequence [ftip-000G]
Let \(\mathcal V\) be the vocabulary of Definition [ftip-000D]. A token sequence of length \(n\geq 0\) is written \(x_{1:n}=(x_1,\ldots ,x_n)\in \mathcal V^n\); positions are one-based. Its length is \(|x_{1:n}|=n\), its prefix before position \(t\) is \(x_{<t}=x_{1:t-1}\), and \(x_{<1}\) is the empty sequence. Juxtaposition \(xy\) denotes concatenation.
If \(\mathtt {eos}\) is distinguished, a completed generated continuation \(y_{1:m}\) ends with \(y_m=\mathtt {eos}\) and contains no earlier \(\mathtt {eos}\). A fixed length limit can also stop generation; the stopping mechanism must state which convention it uses.