Convention. Fixed evaluation interface for protocol comparison [ftip-005D]

✍️sourceAGENTDRAFTED

Fix an evaluation task \(\mathsf T_{\rm ev}\) from Definition [ftip-001O] and a measurable scalar evaluation utility \(u_{\mathsf T_{\rm ev}}:\mathcal X_{\mathsf T_{\rm ev}}\times \mathcal O_{\mathsf T_{\rm ev}}\times \Omega _E\to \mathbb R\), the \(d=1\) case of Definition [ftip-001T]. Fix its task law \(Q_{\rm ev}:=\mu _{\mathsf T_{\rm ev}}\), an inference protocol \(\mathsf I_{\rm ev}\) from Definition [ftip-001R], and an inference budget \(b\in \mathcal B_{\mathrm {eval}}\). Use the inference-seed space \(\Omega _I\) and evaluator-seed space \(\Omega _E\) from Notation [ftip-001N]. A declared probability kernel on these measurable spaces \(\Lambda _{\rm ev}(d\xi ,d\omega \mid x)\) assigns inference and evaluator randomness to each instance. Together with \(Q_{\rm ev}\), it gives the joint evaluation law \(\nu _{\rm ev}(dx,d\xi ,d\omega ) =Q_{\rm ev}(dx)\Lambda _{\rm ev}(d\xi ,d\omega \mid x)\), which is fixed independently of training.

Declare a measurable space \(\mathcal M\) of executable artifacts. For \(M\in \mathcal M\), define the aliases \[ \mathsf {Eval}_b(M,x;\xi ) :=\operatorname {pr}_1\left (\mathsf I_{\rm ev}(M,x,b;\xi )\right ), \qquad U(x,o;\omega ):=u_{\mathsf T_{\rm ev}}(x,o;\omega ). \] Require \(\mathsf {Eval}_b\) to be measurable in \((M,x,\xi )\) and to return admissible outcomes on admitted runs.

Each post-training protocol \(P\) has a probability space \((\Omega _P,\Sigma _P,\mathbb P_P)\) for its randomness and a measurable artifact map \(M_P:\Omega _P\to \mathcal M\) produced from the common base artifact \(M_0\). The no-training protocol is \(P_0\). The complete evaluation law for \(P\) is the product \(\mathbb P_P\otimes \nu _{\rm ev}\), expressing independence between protocol randomness and the fixed evaluation draw. For \(\zeta \in \Omega _P\), write \[ Z_P(\zeta ,x,\xi ,\omega ) =U\left (x,\mathsf {Eval}_b(M_P(\zeta ),x;\xi );\omega \right ). \] This composite is measurable. Every protocol compared through expected performance, including the baseline \(P_0\), must satisfy \[ \int |Z_P|\,d(\mathbb P_P\otimes \nu _{\rm ev})<\infty . \] This is the evaluation domain; finite pointwise utility does not replace the absolute-integrability condition. Fix also a scalar success threshold \(u_*\in \mathbb R\).