Let \(\mathcal S,\mathcal O,\mathcal A\) be declared measurable spaces of environment states, public observations, and agent actions. Time is discrete, \(t\in \{0,1,\ldots ,T_{\max }\}\), where \(T_{\max }<\infty \) is a hard horizon. Random states, observations, and actions are \(S_t,O_t,A_t\); lowercase \(s_t,o_t,a_t\) denote realized values. The same convention makes \(h_t\) a realization of \(H_t\) and \(z\) a realization of \(Z\).

The public history available before action \(A_t\) is \(H_t\); the full environment trajectory is \(Z\); and the stopping time is \(\tau _{\mathrm {stop}}\in \{0,1,\ldots ,T_{\max }\}\). Probability kernels are written \(K(dy\mid x)\); on finite spaces this means the probability law \(K(y\mid x)\) and reduces to the notation of Notation [ftip-000A].

With the spaces in Notation 1, an initial-state law \(\rho _0(ds_0,do_0)\) is a probability law on \(\mathcal S\times \mathcal O\). It jointly samples the hidden initial state \(S_0\) and the first public observation \(O_0\).

Allowing a joint law covers both a deterministic rendering \(O_0=\operatorname {obs}(S_0)\) and a noisy initial observation. A fixed task instance can be included as a coordinate of \(O_0\); its sampling law is then the task law of Definition [ftip-001Q].

A transition-observation kernel at time \(t\) is a probability kernel \[ K_t(ds_{t+1},do_{t+1}\mid s_t,a_t) \] from the current hidden state and action to the next hidden state and public observation. After \(A_t=a_t\), the environment draws \((S_{t+1},O_{t+1})\) from this kernel.

The factorization \(K_t(ds',do'\mid s,a)=P_t(ds'\mid s,a)\,Q_t(do'\mid s',a)\) for declared kernels \(P_t\) and \(Q_t\) recovers the usual separate state-transition and observation kernels when such a factorization is declared. The joint-kernel form does not assume it.

The kernel in Definition 3 describes both tools and broader environments. A deterministic tool has a measurable update map \(F_t(s,a)=(s',o')\) and the point-mass kernel concentrated at \(F_t(s,a)\). A stochastic environment permits several next state-observation pairs for the same input.

This distinction matters for replay. Re-executing a deterministic tool call under an unchanged state reproduces its observation, whereas a stored rollout from a stochastic environment is one sample from a conditional law. Any analysis that replaces one with the other needs a coupling or concentration assumption.

At time \(t\), the public history is the alternating sequence \[ H_t=(O_0,A_0,O_1,A_1,\ldots ,A_{t-1},O_t). \] Thus \(H_0=(O_0)\). It contains exactly the observations received and actions taken before the next action \(A_t\); it does not contain the hidden state \(S_t\) unless that state was itself revealed as an observation.

The history in Definition 5 contains the agent's available information. The hidden state sequence \((S_0,\ldots ,S_t)\), future observations, evaluator-only tests from Remark [ftip-001V], and future random seeds are absent unless the environment has already exposed them through \(O_0,\ldots ,O_t\).

This boundary determines which policies are implementable. It also separates legitimate memory from leakage: a summary of earlier public observations may be stored in the history, while a hidden reference answer may not be supplied under the same declaration.

A non-anticipating policy is a family of probability kernels \[ \pi _t(da\mid H_t), \qquad t=0,1,\ldots ,T_{\max }-1, \] from public histories in Definition 5 to the action space \(\mathcal A\). At time \(t\), the action may depend on \(H_t\) and fresh policy randomness, but not on a later observation or an unobserved state.

A decoder-only language model becomes such a policy only after an inference protocol serializes \(H_t\) into tokens, decodes tokens, and parses them as an action. The next-token law in Definition [ftip-000H] alone does not specify those maps.

A stopping rule is a random time \(\tau _{\mathrm {stop}}\in \{0,1,\ldots ,T_{\max }\}\) such that, for each \(t\), the decision whether \(\tau _{\mathrm {stop}}=t\) is determined by the public history \(H_t\) and any declared stopping randomness available by time \(t\). It cannot inspect future observations.

Stopping can be triggered by a terminal environment observation, an agent action, a resource limit, or the hard horizon. The trigger and the owner of the decision are part of the rule.

If \(\tau _{\mathrm {stop}}=n\), the stopped trajectory is \[ Z=(S_0,O_0,A_0,S_1,O_1,A_1,\ldots , A_{n-1},S_n,O_n). \] It contains the environment states as well as the public action-observation record. Its public projection is the history \(H_n\) of Definition 5.

For \(n=0\), the trajectory is \((S_0,O_0)\) and contains no action. A terminal utility may depend on the full trajectory when evaluated inside the environment, even though the policy is restricted to its public projection.

For a task \(\mathsf T=(\mathcal X_{\mathsf T},\mathcal O_{\mathsf T}, \operatorname {Adm}_{\mathsf T})\) from Definition [ftip-001O], an interactive realization is

For each \(x\in \mathcal X_{\mathsf T}\), the law \(\rho _0^x\) is on \(\mathcal S_{\mathsf T}\times \mathcal O^{\rm obs}_{\mathsf T}\), and \(K_t^x\) maps a state and action to the next state-observation law for \(0\leq t<T_{\max }\). The outcome map sends a stopped trajectory \(Z\) to \(\operatorname {out}_{\mathsf T}(x,Z)\in \mathcal O_{\mathsf T}\) and satisfies \(\operatorname {Adm}_{\mathsf T}(x,\operatorname {out}_{\mathsf T}(x,Z))\). For one fixed realization, write its three spaces as the unadorned \(\mathcal S,\mathcal O,\mathcal A\) of Notation 1.

The agent policy and stopping rule are not environment fields. They are combined with this realization only when an interaction law is formed.

Section 3.1 of [sutton2018reinforcement] specifies the agent--environment interaction. Section 2 of [kaelbling1998planning] separates hidden state from observation. We combine those ingredients with the task interface of Definition [ftip-001O] so each task instance selects an initial law and transition kernels.

The outcome-extraction map is an additional interface choice; neither cited source supplies it in this task-indexed form. This map connects a stopped interaction to the candidate outcome that the independent evaluator will score.

A deterministic three-action episode records each observation before the next policy decision and ends with an explicit terminal observation.

Let \(H_0=(\text {door locked})\). After action \(A_0=\text {take key}\), the environment returns \(O_1=\text {key held}\), so \(H_1=(H_0,A_0,O_1)\). Repeating the same update with unlock and door open, then choosing \(A_2=\text {enter}\), produces \(O_3=\text {terminal success}\) and \(H_3=(H_2,A_2,O_3)\). No later observation is available to an earlier action choice.

The alternating action--observation interface is the finite-history form of the agent setup in [kaelbling1998planning, Section 2]. The trace checks chronology and termination; uncertain state, partial observability, and language-valued actions lie beyond this deterministic instance.

A prefix tree represents token emission as a finite interaction stopped either by an end token or by a hard length cap.

Take vocabulary \(\{a,\mathtt {EOS}\}\). The terminal histories are an immediate end token, an end token after \(a\), and a length-capped history \(aa\). Their probabilities are products of the conditional token laws and sum to one. The last history terminates because the declared maximum length is reached, not because the policy emitted \(\mathtt {EOS}\).

The causal token product is given in [phuong2022formal, Section 3]. The finite representation distinguishes two stopping mechanisms. Token steps remain distinct from tool calls, and unbounded generation is outside the construction.

Sutton and Barto's agent--environment interface in [sutton2018reinforcement, sec. 3.1] supplies alternating states, actions, and rewards. Kaelbling et al. separate hidden state and public observation in [kaelbling1998planning, Section 2]. A stopped interaction is determined by its initial law, transition--observation kernel, policy, and public stopping rule.

The induced interaction law uses four independently declared objects: the initial law \(\rho _0\) in Definition 2, the environment kernels \(K_t\) in Definition 3, the policy \(\pi _t\) in Definition 7, and the stopping rule in Definition 8. On finite spaces, the probability of a length-\(n\) trajectory before applying its stopping indicator factors as \[ \rho _0(s_0,o_0) \prod _{t=0}^{n-1} \pi _t(a_t\mid h_t) K_t(s_{t+1},o_{t+1}\mid s_t,a_t). \] The event \(\{\tau _{\mathrm {stop}}=n\}\) selects the stopped trajectories.

This factorization supports later expectations, likelihood ratios, and off-policy reuse. It also exposes an identifiability limit: an outcome distribution by itself generally does not reveal which of the initial law, environment, policy, or stopping rule changed.