Example. Counterexample: Small average proxy error fails after distribution shift [ftip-007J]
Example. Counterexample: Small average proxy error fails after distribution shift [ftip-007J]
Let the finite decision set be \(\mathcal Y_0=\{y_{\rm safe},y_{\rm exploit}\}\). Define its utility and proxy utility by the following table.
\[ \begin {array}{c|cc} &y_{\rm safe}&y_{\rm exploit}\\ \hline U&1&0\\ \widehat U&1&2. \end {array} \]For \(0<\delta <1\), let a reference law \(P_\delta \) assign probability \(\delta \) to \(y_{\rm exploit}\). Its mean absolute proxy error is
\[ \mathbb E_{P_\delta }\lvert \widehat U-U\rvert =2\delta , \]which tends to zero with \(\delta \). Nevertheless, proxy maximization selects \(y_{\rm exploit}\) and incurs utility regret \(1\). Under the shifted law concentrated on that selected outcome, the mean error is \(2\). Thus no vanishing selection-regret bound can depend only on average error under the reference law.
This two-outcome construction isolates the distribution-shift warning in Remark [ftip-0061]. It does not claim that a particular learned verifier has this error profile; Example [ftip-0060] gives a source-linked checker instance with the same proxy-versus-utility separation.