Convention. Fixed evaluation interface for protocol comparison [ftip-005D]
AGENTDRAFTED
Fix an evaluation task \(\mathsf T_{\rm ev}\) from Definition [ftip-001O] and a
measurable scalar evaluation utility
\(u_{\mathsf T_{\rm ev}}:\mathcal X_{\mathsf T_{\rm ev}}\times
\mathcal O_{\mathsf T_{\rm ev}}\times \Omega _E\to \mathbb R\), the \(d=1\)
case of Definition [ftip-001T]. Fix its task law
\(Q_{\rm ev}:=\mu _{\mathsf T_{\rm ev}}\), an inference protocol
\(\mathsf I_{\rm ev}\) from Definition [ftip-001R], and an inference budget
\(b\in \mathcal B_{\mathrm {eval}}\). Use the inference-seed space \(\Omega _I\)
and evaluator-seed space \(\Omega _E\) from Notation [ftip-001N]. A declared
probability kernel on these measurable spaces
\(\Lambda _{\rm ev}(d\xi ,d\omega \mid x)\) assigns inference and evaluator
randomness to each instance. Together with \(Q_{\rm ev}\), it gives the joint
evaluation law
\(\nu _{\rm ev}(dx,d\xi ,d\omega )
=Q_{\rm ev}(dx)\Lambda _{\rm ev}(d\xi ,d\omega \mid x)\), which is fixed
independently of training.
Declare a measurable space \(\mathcal M\) of executable artifacts. For
\(M\in \mathcal M\), define the aliases
\[
\mathsf {Eval}_b(M,x;\xi )
:=\operatorname {pr}_1\left (\mathsf I_{\rm ev}(M,x,b;\xi )\right ),
\qquad
U(x,o;\omega ):=u_{\mathsf T_{\rm ev}}(x,o;\omega ).
\]
Require \(\mathsf {Eval}_b\) to be measurable in \((M,x,\xi )\) and to return
admissible outcomes on admitted runs.
Each post-training protocol \(P\) has a probability space
\((\Omega _P,\Sigma _P,\mathbb P_P)\) for its randomness and a measurable
artifact map \(M_P:\Omega _P\to \mathcal M\) produced from the common base
artifact \(M_0\). The no-training protocol is \(P_0\). The complete evaluation
law for \(P\) is the product \(\mathbb P_P\otimes \nu _{\rm ev}\), expressing
independence between protocol randomness and the fixed evaluation draw.
For \(\zeta \in \Omega _P\), write
\[
Z_P(\zeta ,x,\xi ,\omega )
=U\left (x,\mathsf {Eval}_b(M_P(\zeta ),x;\xi );\omega \right ).
\]
This composite is measurable. Every protocol compared through expected
performance, including the baseline \(P_0\), must satisfy
\[
\int |Z_P|\,d(\mathbb P_P\otimes \nu _{\rm ev})<\infty .
\]
This is the evaluation domain; finite pointwise utility does not replace the
absolute-integrability condition. Fix also a scalar success threshold
\(u_*\in \mathbb R\).
