Measurement and contamination controls [ftip-00GS]
✍️sourceAGENTDRAFTED
Measurement and contamination controls [ftip-00GS]
✍️sourceAGENTDRAFTED
Convention 1. Evidence required for a reliability claim [ftip-00GT]AGENTDRAFTED
Convention 1. Evidence required for a reliability claim [ftip-00GT]AGENTDRAFTED
Interpreting a reliability claim requires a named estimand, task and evaluation laws, artifact revision, inference protocol, evaluator version, sample rule, and failure policy. Missing coordinates are unknown rather than implicitly held fixed.
Definition 2. Contamination record [ftip-00GU]AGENTDRAFTED
Definition 2. Contamination record [ftip-00GU]AGENTDRAFTED
For a candidate evaluation item \(z\), a contamination record is
\[c(z)=(source,split,exposure,overlap,decision)\]The fields identify source provenance, split membership, prior agent exposure, overlap evidence, and the admission decision; an absent field is an explicit unknown.
Definition 3. A contamination predicate [ftip-00GV]AGENTDRAFTED
Definition 3. A contamination predicate [ftip-00GV]AGENTDRAFTED
Given a contamination record \(c(z)\), write \(\operatorname {cont}(z)=1\) when its declared overlap or exposure rule rejects \(z\), and write \(\operatorname {cont}(z)=0\) when the rule clears it. The predicate belongs to the protocol, not to a model’s score.
Definition 4. An uncontaminated evaluation slice [ftip-00GW]AGENTDRAFTED
Definition 4. An uncontaminated evaluation slice [ftip-00GW]AGENTDRAFTED
For an evaluation law \(\mathsf Q\), an admitted slice is \(\mathcal Z_0=\{z:\operatorname {cont}(z)=0\}\). Its reported score is conditioned on the declared slice law; removing contaminated items does not preserve the original estimand unless the law is unchanged by construction.
Definition 5. A paired comparison design [ftip-00GX]AGENTDRAFTED
Definition 5. A paired comparison design [ftip-00GX]AGENTDRAFTED
A paired design fixes one evaluation law, one item order, and one sampling kernel for two artifacts, then records the paired difference from Definition [ftip-00FE]. Any item-level exclusion is applied before seeing the paired outcomes or is declared as an adaptive rule.
Lemma 6. Pairing preserves the declared marginal target [ftip-00GY]AGENTDRAFTED
Lemma 6. Pairing preserves the declared marginal target [ftip-00GY]AGENTDRAFTED
Under a fixed nonadaptive slice and the iid coupling of Definition [ftip-00FE], the paired estimator remains unbiased for the two marginal scores in Theorem [ftip-00FF].
Proof.
Proof.
Condition on the fixed slice. The coupling has the stated marginals, so the proof of Theorem [ftip-00FF] applies without changing either expectation.
Example 7. A contamination decision can change the estimand [ftip-00GZ]AGENTDRAFTED
Example 7. A contamination decision can change the estimand [ftip-00GZ]AGENTDRAFTED
If one paired run scores all ten items but a second run removes two items after inspecting outputs, the two means answer different questions. A report must expose the removal rule and the resulting slice law.
Remark 8. Scope of measurement methodology [ftip-00H0]AGENTDRAFTED
Remark 8. Scope of measurement methodology [ftip-00H0]AGENTDRAFTED
The measurement and paired-comparison guidance in [jarmak2026reliable], Part I, is a methodology source. It motivates contamination ledgers and matched comparisons; it supplies no universal contamination detector or statistical guarantee.