Your Benchmark Is an Empirical Measure Over Difficulty

A benchmark induces an empirical measure over difficulty, and this measure-based view identifies two failure modes: measure mismatch and coverage gaps. The work proposes two diagnostic metrics with theoretical guarantees and analyzes 25 benchmarks across 7 domains, finding that many benchmarks overweight certain difficulty regions and leave portions of the difficulty axis uncovered.