Zero2Repo Benchmark: The Limits of Autonomous Project Building from an Empty Directory

13 November 20263 views

The study proposes evaluating coding agents on a full repository build: the input is a specification, an interface contract, and a blank environment; validation uses hidden tests without an LLM judge. Even leading agents rarely pass every check, and most failures stem from individual missed requirements.

Zero2Repo Benchmark: The Limits of Autonomous Project Building from an Empty Directory

What Zero2Repo evaluates

The benchmark evaluates whether a coding agent can create a repository from scratch. The agent receives a product requirements document, an interface contract, and an empty workspace. The agent must deliver a complete repository in the ecosystem native to the project.

The evaluation covers building the entire project. Scoring is based on the finished repository.

  • Input: a product requirements document, an interface contract, and an empty workspace.
  • Output: a complete repository in the ecosystem native to the project.
  • Evaluation focus: the ability to create a repository from scratch.

Takeaway: the benchmark tests autonomous building from an empty directory. Practical criterion: use it to evaluate agents on end-to-end project-building tasks.

How tasks are created

The pipeline creates tasks in a language-independent way. It takes real open-source projects with pinned versions, then transforms them into behavioral specifications, reproducible environments, and hidden acceptance tests.

Stages:

  • Selecting real open-source projects with pinned versions.
  • Transforming them into behavioral specifications.
  • Preparing reproducible environments.
  • Creating hidden acceptance tests.

Validation:

  • The reference implementation from the original project passes the tests.
  • Adversarial validation confirms that the tests reject incorrect implementations.

Takeaway: each task is based on a reference implementation and tests that have passed adversarial validation. Practical criterion: check that the tests are validated, not just that the task is described.

How agents are evaluated

The benchmark evaluates production coding agents in isolated containers. Acceptance tests remain hidden until the solution is explicitly submitted. The reward is binary: it is awarded only if all tests pass. The benchmark does not use an LLM judge.

Criteria:

  • An isolated container for the agent.
  • Tests hidden until the solution is submitted.
  • A binary reward for passing all tests.
  • No LLM judge.

Takeaway: passing some tests does not earn a reward. Practical criterion: for comparing agents, use an evaluation where success requires passing all hidden tests.

Languages and coverage in the current release

The pipeline and test system do not assume a specific programming language. The current release includes tasks in Python, TypeScript, Go, and C++.

Release languages:

  • Python.
  • TypeScript.
  • Go.
  • C++.

Takeaway: the benchmark is not tied to a single language, but the current task set is limited to four languages. Practical criterion: match the project's language to the languages covered in the release.

Limits revealed by the results

The limits are visible in a set of 11 tasks. These tasks come from repositories that frontier models have likely encountered during training.

What the results show:

  • The strongest agent solves 10 of 11 tasks.
  • Every failed submission passes 90–99% of the hidden tests.
  • For the two strongest agents, 67–100% of failed tests are associated with one omission or a rare rule in the specification.
  • Failures are related to precision in following the specification, not to missing subsystems.
  • The authors identify each such failure as a specific target for improvement.

Takeaway: the limits of autonomous project building emerge in retaining rare rules and complete specifications. Practical criterion: check how well an agent retains rare specification rules, not just whether it builds the subsystems.

Frequently asked questions

Zero2Repo Benchmark: The Limits of Autonomous Project Building from an Empty Directory