Write \(\mathsf T\) for a task interface, \(\mathcal X_{\mathsf T}\) for its instance set, and \(\mathcal O_{\mathsf T}\) for its candidate-outcome set. A task instance is \(x\in \mathcal X_{\mathsf T}\) and a candidate outcome is \(o\in \mathcal O_{\mathsf T}\). The symbol \(\mu _{\mathsf T}\) denotes a probability law on instances. Equip the instance and outcome sets with declared sigma-algebras; \(\mu _{\mathsf T}\) is a probability measure on the instance sigma-algebra.

Fix an evaluation-resource dimension \(m_{\mathrm {eval}}\geq 1\) and write \(\mathcal B_{\mathrm {eval}}=\mathbb R_+^{m_{\mathrm {eval}}}\) with its coordinatewise order. An evaluation inference protocol is written \(\mathsf I\); its declared evaluation budget is \(b_{\mathrm {eval}}\in \mathcal B_{\mathrm {eval}}\). Its inference-seed set is \(\Omega _I\), with realized seed \(\xi \in \Omega _I\). An evaluator's random-seed set is \(\Omega _E\); its realized seed is \(\omega \in \Omega _E\). It returns utility \(u_{\mathsf T}(x,o;\omega )\in \mathbb R^d\). The dimension \(d\geq 1\) is declared by the evaluation: \(d=1\) gives scalar utility, while \(d>1\) retains several outcomes or costs without an implicit weighting. Equip both seed sets with declared sigma-algebras and \(\mathbb R^d\) with its Borel sigma-algebra. Products below carry the product sigma-algebra; finite or countable discrete spaces may use all subsets. A seed space alone does not specify its sampling law.

A task is a triple \[ \mathsf T=(\mathcal X_{\mathsf T},\mathcal O_{\mathsf T}, \operatorname {Adm}_{\mathsf T}), \] where \(\mathcal X_{\mathsf T}\) is an instance set, \(\mathcal O_{\mathsf T}\) is a candidate-outcome set, and \(\operatorname {Adm}_{\mathsf T}(x,o)\) is a declared well-formedness predicate for an instance-outcome pair.

Admissibility says that an outcome can be interpreted for the instance; it does not say that the outcome is correct, useful, or safe. Those judgments belong to the evaluation utility introduced later.

For the instance set in Definition 2, a task law is a probability law \(\mu _{\mathsf T}\) on \(\mathcal X_{\mathsf T}\). An evaluation sample of size \(N\) is a declared joint law of instances \((X_1,\ldots ,X_N)\) whose marginals are \(\mu _{\mathsf T}\); independent sampling is a further assumption, not part of the term ``task law.''

In an episodic control problem this law includes the distribution of starting situations. In a fixed benchmark it may instead be the uniform law on a held-out finite set. Reweighting instances changes the task law even when the instance files are unchanged.

A task and a task law specify admissible instance--outcome pairs and how instances are sampled. Benchmark datasets usually present an input and expect an answer in a parseable format. Reinforcement-learning formulations instead name states, actions, and reward; see [sutton2018reinforcement, Chapter 3]. The admissibility predicate captures parseability, while the law records how instances are sampled. Neither choice already identifies parsing with success.

One-shot inference and an interactive environment can share the same task interface. Claims about an interaction's temporal structure additionally depend on the environment and the observations available at each decision.

Let \(M\) be an executable model artifact of the form in Definition [ftip-001K]. An evaluation inference protocol is a specified procedure \[ \mathsf I(M,x,b_{\mathrm {eval}};\xi )=(o,c) \in \mathcal O_{\mathsf T}\times \mathcal B_{\mathrm {eval}}, \] where \(x\in \mathcal X_{\mathsf T}\), \(b_{\mathrm {eval}}\in \mathcal B_{\mathrm {eval}}\), and \(\xi \in \Omega _I\) use the notation of Notation 1; \(o\) is a candidate outcome and \(c\in \mathcal B_{\mathrm {eval}}\) is the realized resource-use vector. A valid execution satisfies \(c\preceq b_{\mathrm {eval}}\) coordinatewise.

The protocol fixes prompt construction, decoding, sampling temperature, number of candidates, candidate selection, allowed tools, stopping, and any test-time search. None of these choices is determined by the checkpoint alone.

One next-token law supports many evaluation procedures. Greedy decoding, temperature sampling, majority voting, verifier-guided selection, and multi-round tool use can produce different outcome laws from the same checkpoint. The sampling and selection procedures in [shao2024deepseekmath, secs. 3--4] are concrete examples.

Accordingly, a comparison that changes \(b_{\mathrm {eval}}\) or \(\mathsf I\) does not estimate a training-only effect. A potential frontier therefore depends on evaluation-time compute as well as the trained model.

With the notation of Notation 1, an evaluation utility is a declared measurable function \[ u_{\mathsf T}:\mathcal X_{\mathsf T}\times \mathcal O_{\mathsf T}\times \Omega _E\longrightarrow \mathbb R^d, \] where \(\Omega _E\) is the evaluator's measurable seed space. The evaluator applies it only to admissible pairs from Definition 2. A deterministic evaluator is the special case in which the value does not depend on \(\omega \in \Omega _E\).

Binary pass-fail evaluation takes \(d=1\) and values in \(\{0,1\}\). A cumulative return is another scalar case. Vector utility keeps qualities such as correctness, safety, latency, and tool cost distinct until a later rule declares how to compare them.

Whenever an expected utility vector is used, its composite evaluation random variable must be measurable and absolutely integrable in every coordinate under the declared joint law. Thus its expectation lies in \(\mathbb R^d\). Bounded measurable utility is a sufficient specialization; so is finite support for the complete evaluation random variable with finite values on that support. Merely taking a finite real value at each seed is insufficient for an unbounded evaluator. The scalar protocol comparison in Convention [ftip-005D] states its joint law explicitly.

Sutton and Barto define scalar episodic return in [sutton2018reinforcement, Section 3.3]. Benchmarks also use exact-match, test, latency, safety, and cost measurements. A proposed seeded vector utility keeps those coordinates separate, with evaluator randomness included in the declared law.

The vector is not a claim that its coordinates share a natural scale. Scalarization, Pareto order, and worst-group summaries are introduced only after the attainable evaluation set is defined.

One fixed policy can receive different measured success rates when the evaluation horizon changes.

Partition complete trajectories into those that finish within the short horizon, those requiring additional tokens, and failures. The short protocol counts only the first class; the longer protocol may count the first two. Thus the measured score depends on the declared horizon even though the policy is unchanged.

Making an evaluation horizon explicit follows finite-horizon policy evaluation in [sutton2018reinforcement, Chapter 3]. Only truncation is varied here. Larger budgets need not help once decoding rules, tool costs, or selection errors are allowed to change with the budget.

The inference protocol receives only the information designated as public. The public instance \(x\) may include a prompt, tools, and a response schema. Evaluator-only information may include a reference answer, hidden tests, a simulator state, or the evaluator seed \(\omega \). The protocol in Definition 5 receives the former and not the latter.

If evaluator-only information influences training, decoding, or candidate selection, the experiment no longer measures performance under the declared information interface. A theorem that assumes an independent evaluator must name the sigma-field or finite data record from which the protocol is excluded.