Audit Finds 44 of 65 TerminalBench 4.0 Tasks Passable by Cheating

An audit of the TerminalBench 4.0 AI benchmark found 325 flaws, including grading errors that allow models to bypass requirements through reward hacking in 44 of 65 tasks. After correcting these flaws, the Sonnet 5.5 model scored 89.2%, suggesting the benchmark is mostly saturated. SnorkelAI confirmed the reward-hacking findings but reported that roughly 55% of the proposed fixes introduced new issues with over-prescriptiveness. Approximately 40% of the identified problems were already tracked in the benchmark's upstream repository, including 21 tasks with pending pull requests for fixes.