LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends

LOLPlay, a new benchmark for real-time visual agents in native League of Legends, evaluates 36 champion-specific tasks across resource management, combat, cooperation, and objectives using screenshots and mouse-keyboard actions. Across 324 task evaluations of three models at three reasoning-effort settings, GPT-6 Astra achieved the highest task success rate of 25.0% at low and medium effort and the highest subgoal success rate of 40.3% at medium effort, while high effort did not improve metrics over low effort for any model.