The Evolution of Agent Benchmarks as Task Pools Scale: The Role of Evidence Selection and Sequential Races
This work explains why increasing the number of tasks can hinder the evolution of agent benchmarks: the problem is poor evidence selection, not data volume itself. The proposed Successive Scale Racing iteratively selects diverse task combinations and compares candidates at low cost; across three benchmarks, it increased the share of successful runs as the pool grew from 40 to 200, while baseline methods degraded, and reduced validation runs by about 60%.








