Zero2Repo Benchmark: The Limits of Autonomous Project Building from an Empty Directory
The study proposes evaluating coding agents on a full repository build: the input is a specification, an interface contract, and a blank environment; validation uses hidden tests without an LLM judge. Even leading agents rarely pass every check, and most failures stem from individual missed requirements.

What Zero2Repo evaluates
The benchmark evaluates whether a coding agent can create a repository from scratch. The agent receives a product requirements document, an interface contract, and an empty workspace. The agent must deliver a complete repository in the ecosystem native to the project.

The evaluation covers building the entire project. Scoring is based on the finished repository.
- Input: a product requirements document, an interface contract, and an empty workspace.
- Output: a complete repository in the ecosystem native to the project.
- Evaluation focus: the ability to create a repository from scratch.
Takeaway: the benchmark tests autonomous building from an empty directory. Practical criterion: use it to evaluate agents on end-to-end project-building tasks.
How tasks are created
The pipeline creates tasks in a language-independent way. It takes real open-source projects with pinned versions, then transforms them into behavioral specifications, reproducible environments, and hidden acceptance tests.
Stages:
- Selecting real open-source projects with pinned versions.
- Transforming them into behavioral specifications.
- Preparing reproducible environments.
- Creating hidden acceptance tests.
Validation:
- The reference implementation from the original project passes the tests.
- Adversarial validation confirms that the tests reject incorrect implementations.
Takeaway: each task is based on a reference implementation and tests that have passed adversarial validation. Practical criterion: check that the tests are validated, not just that the task is described.
How agents are evaluated
The benchmark evaluates production coding agents in isolated containers. Acceptance tests remain hidden until the solution is explicitly submitted. The reward is binary: it is awarded only if all tests pass. The benchmark does not use an LLM judge.
Criteria:
- An isolated container for the agent.
- Tests hidden until the solution is submitted.
- A binary reward for passing all tests.
- No LLM judge.
Takeaway: passing some tests does not earn a reward. Practical criterion: for comparing agents, use an evaluation where success requires passing all hidden tests.
Languages and coverage in the current release
The pipeline and test system do not assume a specific programming language. The current release includes tasks in Python, TypeScript, Go, and C++.
Release languages:
- Python.
- TypeScript.
- Go.
- C++.
Takeaway: the benchmark is not tied to a single language, but the current task set is limited to four languages. Practical criterion: match the project's language to the languages covered in the release.
Limits revealed by the results
The limits are visible in a set of 11 tasks. These tasks come from repositories that frontier models have likely encountered during training.
What the results show:
- The strongest agent solves 10 of 11 tasks.
- Every failed submission passes 90–99% of the hidden tests.
- For the two strongest agents, 67–100% of failed tests are associated with one omission or a rare rule in the specification.
- Failures are related to precision in following the specification, not to missing subsystems.
- The authors identify each such failure as a specific target for improvement.
Takeaway: the limits of autonomous project building emerge in retaining rare rules and complete specifications. Practical criterion: check how well an agent retains rare specification rules, not just whether it builds the subsystems.
Frequently asked questions
Related materials
All materials
FedSocket: Exchangeable Models with Guaranteed Executability on the Recipient Side under Multimodal Heterogeneity
FedSocket is proposed as an approach in which the shared model must be executable on the recipient’s side, while a common prediction interface Q connects heterogeneous private models. The paper describes membership-aware aggregation, forwarding only Q’s parameters along with counters, and experiments on six datasets demonstrating improved accuracy with missing modalities and the advantage of Joint inference over independent ensembles.
13 November 2026
CyberArena: Head-to-Head AI Agent Matches Reveal the Limits of Static Cyber Benchmarks
The benchmark places two agents in sandboxes with continuously changing services and, over a series of rounds, aggregates their actions into multidimensional scores of offensive and defensive skills. Across 466 matches with 12 state-of-the-art models, it distinguished capabilities where one-sided tests hit a ceiling, including multi-step planning and backdoor creation.
13 November 2026
How Robust Are Detectors to Paraphrased Machine-Generated Text? What the Tests Revealed
This article compiles the results of tests on tools that rewrite generated passages in a more natural style. After processing, detectability drops noticeably, while the remaining false positives mean a detector’s verdict cannot be considered proof.
13 November 2026