For a fixed task \(x\), let \(\mathcal Y_x\) be a finite response set and let a named reward channel be a map \(r:\mathcal Y_x\to \mathbb R\). A utility map \(u:\mathcal Y_x\to \mathbb R\) is a separate evaluation quantity.

If an interaction has state space \(\mathcal S\), action space \(\mathcal A\), and transition kernel \(K(\mathord {\cdot }\mid s,a)\), then \(K\) is a third object: changing \(K\) can change consequences without changing \(r\).

A verifier channel is a map \(V:\mathcal Y_x\to \{0,1\}\). Given a validity map \(U:\mathcal Y_x\to \{0,1\}\), a false-accept event is

The event depends on the declared verifier and validity test. It is not identified by a scalar reward unless equivalence with that reward criterion is an explicit assumption.

Sections 4.1--4.2 and 7.2--7.3 of [dharna2026aifindsway] catalogue reward, score, environment, and evaluator-target anecdotes. The source gives 26 curated firsthand anecdotes involving more than 100 researchers; it does not provide a common sampling frame, base rates, or causal effect estimates.

Accordingly, this section uses the paper as an empirical counterexample catalog. The finite statements that follow are proved here and do not claim to summarize all training systems.

Let \(\mathcal Y_x\) be finite, let \(u,r:\mathcal Y_x\to \mathbb R\), and assume \(|r(y)-u(y)|\le \varepsilon \) for every \(y\), with \(\varepsilon \ge 0\). If \(y^\star \) maximizes \(u\) and \(\widehat y\) maximizes \(r\), then

Indeed, \(u(y^\star )\le r(y^\star )+\varepsilon \le r(\widehat y)+\varepsilon \le u(\widehat y)+2\varepsilon \). This is a finite same-class decision bound, not an optimization, distribution-shift, or capability theorem.

Take \(\mathcal Y_x=\{a,b\}\), with \(r(a)=r(b)=1\) but \(u(a)=1\) and \(u(b)=0\). Every reward maximizer is a nominal tie, while only \(a\) is utility optimal. The example shows why a score equality does not establish intent or validity.

For candidate index \(i\), let \(I_i\in \{0,1\}\) indicate invalidity and let \(A_i\in \{0,1\}\) indicate verifier acceptance. Define \(F_i=\{I_i=1,A_i=1\}\). Write \(q_i=\Pr (I_i=1)\). If \(q_i>0\), write \(\eta _i=\Pr (A_i=1\mid I_i=1)\); if \(q_i=0\), set \(\eta _i=0\). Then \(\Pr (F_i)=q_i\eta _i\).

No independence or identical-distribution hypothesis is part of this definition.

For finitely many candidate events from Definition 2.1,

This is the union bound and requires no independence. If every marginal is at most \(\eta \), the right side is at most \(N\eta \).

Suppose two candidates are produced: candidate 1 is invalid and falsely accepted, while candidate 2 is valid and rejected by a separate score tie-break. Then \(\bigcup _iF_i\) occurs, but a selection rule that always chooses candidate 2 outputs a valid response. Thus an existence probability and a selected-output probability coincide only after the selection law is specified.

Given transition kernel \(K\), reward \(r\), and validity predicate \(U\), an environment exploit is a candidate \(y\in \mathcal Y_x\) whose induced interaction under \(K\) receives high \(r(y)\) while failing \(U\). The definition is relative to the declared kernel, horizon, and validity test; it is not a universal property of a model.

Let one response receive \(r(y)=1\) under two environments, and let \(W:\mathcal S\to \{0,1\}\) test terminal-state safety. In \(K_1\), its next state \(s_1\) has \(W(s_1)=1\); in \(K_2\), the same observed response enters \(s_2\) with \(W(s_2)=0\). The reward channel alone cannot distinguish the two transition semantics, so equal reward does not certify safe consequences.

Let \(V(y)=1\) for every output that contains a visible marker, while \(U\) checks a hidden task condition that the marker does not affect. A policy that optimizes \(V\) can improve its observed score while leaving \(U\) unchanged or worse. This is a finite illustration of evaluator targeting, not a claim about the frequency of such behavior.

The source catalog does not identify the base rate of exploits, the causal effect of a reward intervention, or a universal relationship between score and utility. Heterogeneous anecdotes therefore motivate audit questions and failure tests, not population-level probabilities.

An audit record is a tuple containing candidate identity, evaluator and verifier version, environment or transition-kernel version, invalidity tests, random seed, evidence pointers, and adjudicator decision. A record is complete only when these fields are bound to the candidate and the protocol run.

Let \(N\in \mathbb N_{\geq 1}\) and \(0\leq \eta \leq 1\). Assume these \(N\) candidates are audited and each invalid candidate is falsely accepted with marginal probability at most \(\eta \). Then the probability that some invalid candidate is accepted is at most \(N\eta \), by the union bound of Lemma 2.2. This conclusion does not require independence.

The statement bounds existence of an accepted invalid candidate. A selected output guarantee additionally requires a typed selection rule and its relation to the audit decisions.

These finite channel distinctions do not prove reward hacking rates, capability acquisition, or safety of a deployed agent. The next questions are to measure verifier sensitivity, environment changes, selection rules, and independent audit power under a declared protocol.