● ToolsNov 8, 2026 · 01:548 sources
Hermes Agent Leads Reliability Ranking on Updated AI Coding Harness Leaderboard
From marketbrief.now
AI;DR
Researcher Joel Niklaus added a consistency metric to the Harness Leaderboard to track how often coding-agent wrappers solve the same task on three consecutive attempts. Hermes Agent was identified as the most reliable overall, achieving a 71.3% solve rate on the DeepSeek-V4.1-Flash model. The updated data shows that reliability for unattended AI work remains low, with the top-ranked mini-swe-agent solving only 33% of tasks consistently across three open models.
Read originalSign in to suggest
Sources
+ 6 more
- SUPPORT@ofirpress“mini-swe-agent is 130 lines of code and it keeps on winning in harness comparisons”x.com
- SUPPORT@teknium“Hermes supremacy tweet #9138”x.com
- SUPPORT@joelniklaus“Hermes Agent leads on DeepSeek-V4.1-Flash (71.3%), and over all three models it solves the most tasks on every attempt”x.com
- SUPPORT@joelniklaus“Pi compacts at 112k tokens (it assumes a 128k window for models it doesn't know), while mini-swe-agent never compacts”x.com
- SUPPORT@joelniklaus“OpenCode 1.19.0 just makes far more calls, about 200 per trial vs 75 for Pi”x.com
- SUPPORT@thom_wolf“same open model, different harness => very different results”x.com