← Back to live feed1 story
SnorkelAI found that 55% of the fixes proposed by the Preference Model team for TerminalBench 4.0 vulnerabilities were themselves flawed, citing over-prescriptiveness or new gaps allowing reward hacking. A preliminary audit also noted that 40% of the 325 identified issues were already tracked in the benchmark's repository, including 21 tasks with outstanding fix requests.
Sign in to suggest
Key sources
- SOURCE@chem_safety“After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated.”x.com
- SUPPORT@chem_safety“Of the 65 tasks: 60 tasks had at least one issue 44 could be passed by cheating / reward hacking 41 tasks were unfair, e.g. grading on unstated requirements 325 issues in total”x.com
- SOURCE@ajratner“We agree with many of the findings, but ~55% of the fixes from @preferencemodel had their own issues with reward hacking or over-prescriptiveness.”x.com
- SUPPORT@madiator“Bespoke has found several issues using our internal QC agent, and has been working on improving Terminal Bench over the months. Here are the 25 issues we filed, and there are PRs corresponding to them.”x.com
- SOURCE@madiator“We have an internal QA system which has been finding issues as well, and we have submitted several fixes to Terminal Bench”x.com
- SUPPORT@scaling01“44 could be passed by cheating / reward”x.com
- SUPPORT@rebeccatqian“Building datasets like this - and maintaining ambitious open benchmarks around them - is hard.”x.com
- SUPPORT@teortaxestex“we're running out of benchmarks too quickly”x.com