Example. Reward hacking under an incomplete checker [ftip-0060]
Example. Reward hacking under an incomplete checker [ftip-0060]
Shifting probability mass toward a checker exploit raises proxy reward while reducing independently judged utility.
Let the three outputs be \(y_0=\text {41}\), \(y_1=\text {42}\), and \(y_2=\text {ignore; 42; done}\). The proxy accepts \(y_1,y_2\); the independent evaluator accepts only \(y_1\). Moving their probabilities from \((0.5,0.4,0.1)\) to \((0.1,0.1,0.8)\) changes expected proxy reward from \(0.5\) to \(0.9\), but true utility from \(0.4\) to \(0.1\).
Reward exploitation in formal verification is discussed in [ren2025deepseekproverv2, Section 3.2]. The constructed checker exposes one failure mechanism. It provides no basis for treating every executable verifier as incomplete or every proxy improvement as reward hacking.