Definition. Autoregressive continuation law [phuong2022formal, sec. 3, Sequence modelling] [ftip-000I]
✍️source

    Fix the next-token law of Definition [ftip-000H], a prompt \(x\in \mathcal V^*\), and a continuation \(y_{1:m}\in \mathcal V^m\). Its autoregressive continuation probability is \[ P_\theta (y_{1:m}\mid x) =\prod _{t=1}^{m} \pi _\theta \left (y_t\mid x y_{<t}\right ). \] Each factor conditions only on the prompt and earlier continuation tokens.

    If generation stops on \(\mathtt {eos}\), the law of a completed continuation is restricted to the sequences specified in Convention [ftip-000G]. If generation instead stops at a length limit, the terminal event and its probability must be recorded separately.