Interpreting a reliability claim requires a named estimand, task and evaluation laws, artifact revision, inference protocol, evaluator version, sample rule, and failure policy. Missing coordinates are unknown rather than implicitly held fixed.

For a candidate evaluation item \(z\), a contamination record is

The fields identify source provenance, split membership, prior agent exposure, overlap evidence, and the admission decision; an absent field is an explicit unknown.

Given a contamination record \(c(z)\), write \(\operatorname {cont}(z)=1\) when its declared overlap or exposure rule rejects \(z\), and write \(\operatorname {cont}(z)=0\) when the rule clears it. The predicate belongs to the protocol, not to a model’s score.

For an evaluation law \(\mathsf Q\), an admitted slice is \(\mathcal Z_0=\{z:\operatorname {cont}(z)=0\}\). Its reported score is conditioned on the declared slice law; removing contaminated items does not preserve the original estimand unless the law is unchanged by construction.

A paired design fixes one evaluation law, one item order, and one sampling kernel for two artifacts, then records the paired difference from Definition [ftip-00FE]. Any item-level exclusion is applied before seeing the paired outcomes or is declared as an adaptive rule.

Under a fixed nonadaptive slice and the iid coupling of Definition [ftip-00FE], the paired estimator remains unbiased for the two marginal scores in Theorem [ftip-00FF].

If one paired run scores all ten items but a second run removes two items after inspecting outputs, the two means answer different questions. A report must expose the removal rule and the resulting slice law.

The measurement and paired-comparison guidance in [jarmak2026reliable], Part I, is a methodology source. It motivates contamination ledgers and matched comparisons; it supplies no universal contamination detector or statistical guarantee.

A grading rubric is \(R=(\mathcal Z,\mathcal L,g,v)\): outcome space, finite label set, scoring map \(g:\mathcal Z\to \mathcal L\), and evaluator version \(v\). The rubric declares which labels count as success and which observations are abstentions.

For rubric \(R\), a grader response law is a kernel \(K(\ell \mid z)\) on \(\mathcal L\) given outcome \(z\). A deterministic grader is the point-mass case; stochastic or model-based graders must retain their version and sampling coordinates.

A calibration set is a finite set \(\mathcal C\subseteq \mathcal Z\) with an adjudicated label \(y^star(z)\) for each \(z\in \mathcal C\). The adjudication source, disagreement policy, and release version are part of the set’s provenance.

For grader kernel \(K\) and calibration set \(\mathcal C\), the agreement rate is \(A(K,\mathcal C)=|\mathcal C|^{-1}\sum _{z\in \mathcal C} K(y^star(z)\mid z)\). It is a calibration-set statistic, not a population accuracy claim.

If \(|\mathcal C|=n\geq 1\) and \(A(K,\mathcal C)\geq 1-\eta \), then the empirical disagreement mass \(1-A(K,\mathcal C)\) is at most \(\eta \).

A grader that agrees on 99 of 100 easy cases but disagrees on the one safety case has overall agreement \(99/101\approx 0.9802\). Its agreement within the safety subgroup is zero. Any positive safety-subgroup agreement threshold therefore rejects this grader despite its high overall agreement. A calibration report stratifies by failure-relevant labels.

A grader release gate accepts \(K\) only when its calibration version, agreement thresholds, abstention handling, and stratified failure checks are recorded. A failed gate blocks interpretation of downstream score changes.

Part II of [jarmak2026reliable] organizes grading and reviewer calibration practices. The finite agreement quantities do not establish a universal threshold or blinded-human reliability theorem.

An execution gate is a predicate \(g_i(e_i)\) over a recorded stage result \(e_i\). It names its owner, required inputs, pass/fail/unknown outputs, and the artifact versions to which the result applies.

An ordered gate chain is \((g_1,\ldots ,g_k)\) with stage outputs \(e_1,\ldots ,e_k\); gate \(g_i\) may execute only after its declared prerequisites and records a monotone status in \(\{pass,fail,unknown\}\).

For a finite chain, define release status as \(G=\bigwedge _{i=1}^k g_i(e_i)\). If any gate is false, then \(G\) is false.

A durable execution record is

It is append-only, versioned, and sufficient to locate each gate input and output without relying on worker-local memory.

A replay contract for \(d\) fixes the executable revision, environment image, input artifact hashes, seed law, and event order. A replay is faithful only when those coordinates are available and the resulting events satisfy the recorded schema.

If the execution map is deterministic in the coordinates fixed by Definition 3.5, replaying the same inputs, revision, environment, and seed produces the same event trace.

A reproducible run can deterministically reproduce a wrong patch or a misconfigured evaluator. Replay establishes trace identity under its contract; it does not establish that the trace met the intended utility or safety goal.

For durable record \(d\), let \(H(d)\) be a cryptographic digest of its canonical serialized fields. A replay request fails closed when the supplied record digest or any referenced input hash differs from the declared value.

To bind a replay result to an audit decision, the replay trace must be included in the audited record and covered by its digest. A matching digest for other fields does not bind that trace to the decision.

Part III of [jarmak2026reliable] motivates containment, durable execution, and recovery records. Deterministic replay and matching digests establish execution consistency; they do not ensure correctness.

An audit-before-commit record is \(a=(d,scope,checks,reviewer,decision)\). It binds the durable execution record \(d\) to the audited scope, check list, review identity, and a decision in \(\{commit,hold,reject\}\).

A commit decision is admissible only if every required check in an Definition 4.1 record is pass, the audited digest equals the proposed record digest, and the decision is \(commit\).

An audit is independent for scope \(S\) when its inputs are frozen before review, its reviewer or checker is distinct from the proposing action, and its decision cannot rewrite \(S\) without a new record. Independence is a protocol condition, not a claim of infallibility.

A run may pass unit tests while an independent audit finds that the evaluator version in \(d\) differs from the declared version. The audit holds the commit and records the mismatch; replay alone would not have caught it.

Classify a failed run by its earliest violated contract: measurement failure (M), grading failure (G), execution/state failure (E), audit failure (A), or transfer failure (T). A single run may receive secondary labels, but the primary label is the earliest failed gate in recorded order.

For a finite ordered gate chain with at least one failed gate, the earliest failed index is unique and therefore determines one primary taxonomy label.

A remediation record maps a primary label from Definition 4.5 to an owner, corrective action, re-test scope, and closure condition. Closure requires a new durable record and cannot mutate the failed record in place.

Parts III and V of [jarmak2026reliable] motivate durable execution, review, and accountability practices. The audit taxonomy and conjunction gates are proposed operational specifications; they do not establish that an audit is complete or that a committed artifact is correct.

The gates above make evidence auditable. They do not identify the causal effect of an intervention when task law, artifact, grader, budget, or harness state changes together. Those coordinates must be controlled in the matched evaluation of Definition [ftip-00GF].

Let a deterministic run train and evaluate on one leaked item. Its digest, event trace, and replay are all identical across reruns, yet the measurement is contaminated under Definition 1.3. Replayability and contamination control are independent obligations.

A grader can meet its threshold on \(\mathcal C\) in Lemma 2.5 while every deployment item lies outside that calibration domain and receives an unvalidated label. Agreement on \(\mathcal C\) alone does not transport to a new law.

If an audit plan has finite nonnegative stage costs and its declared sum is at most budget \(B\), then the plan is budget-admissible; adding any positive stage beyond the remaining slack makes it inadmissible.

With budget \(B=100\), a plan may allocate 50 units to execution, 30 to grading calibration, and 20 to audit. A proposed 10-unit extra audit must be funded by reducing another stage or the admission gate fails.

A passing protocol establishes that declared evidence contracts, hashes, checks, and budgets were satisfied for the recorded run. It does not establish capability acquisition, broad generalization, or absence of unobserved faults.