RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

RuleTrek is a multi-domain benchmark introduced to evaluate agentic rule reasoning in complex scenarios where scenario information is distributed across multiple files. It uses a rule-grounded construction pipeline to extract structured rule graphs and generate consistent, solvable sub-scenarios, assessing Rule Identification, Rule Deduction, and End-to-End Rule Reasoning. Experiments on 11 frontier LLMs show the highest Rule Identification and Rule Deduction scores are 62.47% and 60.43%, respectively, while End-to-End scores range from 26.97% to 55.29%.