Example. High aggregate agreement can hide a subgroup failure [ftip-00H7]
Example. High aggregate agreement can hide a subgroup failure [ftip-00H7]
A grader that agrees on 99 of 100 easy cases but disagrees on the one safety case has overall agreement \(99/101\approx 0.9802\). Its agreement within the safety subgroup is zero. Any positive safety-subgroup agreement threshold therefore rejects this grader despite its high overall agreement. A calibration report stratifies by failure-relevant labels.