ReproAgent: AI agent builds working code from a research paper using an "implementation contract"

17 September 202618 views

Recreating a repository from a publication is hindered by a fragmented specification: some information is lost along the long chain of the agent's reasoning, while other details are simply not mentioned in the article's text because they fall under commonly accepted framework defaults. ReproAgent addresses this through a four-stage cycle and a persistent contract that links requirements from the paper with evidence from related open repositories; on the PaperBench Code-Dev benchmark, the approach achieved the best average result among scaffolds using the same base model.

ReproAgent: AI agent builds working code from a research paper using an "implementation contract"

From Article to Repository: Where the Chain Breaks

A scientific publication is not a build instruction. Even a detailed text with formulas and figures does not contain everything needed to run the code and obtain the same numbers. It is precisely this gap that the work titled ReproAgent: Contract-Guided Paper-to-Code Reproduction investigates — it formalizes the task of paper-to-code reproduction: a scientific AI agent must turn a paper into an executable repository that preserves the method, the experimental protocol, and the artifacts.

According to the authors, the reason for the difficulty is not the "weakness" of models but the fragmentation of the specification. The explicit part — algorithms, metrics, the composition of artifacts — is present in the paper, but over long agent trajectories these details gradually get lost, washed out of the working context. The implicit part — framework defaults, conventions inherited from related work — is not mentioned in the text at all: for the paper's authors it is so obvious that it needs no words. Such details cannot be reconstructed "from general reasoning," and without them the code either fails to run or produces different results.

Formal details of the preprint: arXiv:2608.24291, primary category — cs.AI, with cs.SE additionally listed; submitted on August 25, 2026. The authors are Xue Hu, Zewei Pan, Zhongyuan Wang, Zhou Liu, Zeli Su, and Wentao Zhang; the submitter is Xue Hu. The work was accepted to Findings of EMNLP 2026, DOI: 10.48550/arXiv.2608.24291.

The Implementation Contract as an Anchor for the Agent

The main idea of ReproAgent is not to let the agent "just read the paper and write code." Instead, a persistent implementation contract is built — an artifact that lives throughout the entire process and survives long trajectories, unlike the source text in the context.

The contract is populated from two independent channels:

  • the implementation-requirement channel — turns fragments of the paper into concrete obligations for the code: what exactly must be implemented, which metrics must be computed, which data must be loaded;
  • the reference-evidence channel — extracts substantive and structural hints from related repositories, that is, pulls in those very implicit conventions that are absent from the text.

Then both streams converge: they are tied to work packages, and then projected into contracts at the level of individual files. This projection matters because it is at the file level that the agent actually has to act, and it is precisely there that the connection between "the idea from the paper" and "the line in the code" is usually lost.

The point of the construction is that the obligation comes first and generation comes second. The agent does not try to recall the needed detail at the moment of writing code — it checks against an already fixed requirement.

Four Stages and the Role of Repair

The work is described as a pipeline of four consecutive stages that form the acronym Prepare — Plan — Generate — Repair.

  1. Prepare — preparation: parsing the paper and related materials, initial population of the contract.
  2. Plan — planning: breaking the task into work packages and tying obligations and evidence to them.
  3. Generate — generating code from file-level contracts.
  4. Repair — repair: the contract is reused to understand what exactly diverged from the requirement, rather than patching errors at random.

Note the asymmetry: the contract is needed not only at the input but also at the very end — when fixing errors. This is perhaps the most practical part of the design. Most agentic pipelines are strong at the writing stage and weak at the debugging stage, because by the time the first error appears the original specification has already been blurred. Here it remains at hand.

Evaluation: PaperBench Code-Dev

The authors tested the approach on PaperBench Code-Dev — a benchmark where an agent must reproduce code from scientific papers. The result: the highest average score among scaffolds with the same backbone. Importantly, the effect is observed with two different base models — Claude Sonnet 4.5 and Gemini 3 Flash.

What these terms mean. Backbone is the base model, the agent's "brain." Scaffold is the harness around it: rules, memory, step order, tools. Comparison with the same backbone is the correct way to show that the gain comes specifically from the architecture rather than from a stronger model. ReproAgent is precisely a contribution to the class of scaffolds.

Additionally, the authors conducted ablations by channel: if the requirement channel or the evidence channel is disabled, end-to-end quality drops. Plus case studies of individual papers showing exactly what contribution each channel makes on concrete examples. Such double control — an aggregate measurement plus a qualitative analysis — makes the conclusions more convincing than a single number in a table.

According to the authors, the code and experimental artifacts are publicly available.

What This Changes in Practice

The value of the work lies not only in the benchmark result but in the very formulation of the problem. "Fragmentation of the specification" is a good explanation of why agents confidently write code that looks correct but does not reproduce the results. Explicit details get lost as the trajectory grows longer; implicit ones are absent from the start.

The implementation contract offers a workaround: move obligations out of the fragile context into a separate, durable object and refer to it at all stages, including debugging. This technique looks transferable — similar "fixed requirements" could be useful in other tasks where an agent needs to hold the initial conditions for a long time: migrations, infrastructure reproduction, long engineering tasks.

The limitations are also clear from the logic of the method: the quality of the contract depends directly on the quality of requirement extraction and on how suitable the related repositories found are. If there is no public code on the topic, the evidence channel works blind. But at least one of the authors' claims already sounds convincing: reproducibility of scientific work is a problem about specification, not about the speed of code generation.

Frequently asked questions

Related materials

All materials
Validating Candidates Costs More Than Generating Them: Honeypots Against LLM Vulnerability Hunting
News

Validating Candidates Costs More Than Generating Them: Honeypots Against LLM Vulnerability Hunting

The study examines the cost asymmetry in agentic vulnerability discovery: generating candidate versions is cheap, while confirming them through reachability analysis, execution, and PoC development consumes most of the budget, making verification a potential standalone target for defense. The proposed RedHerring—repository-adapted decoys with a hidden certificate—reduced the number of real vulnerabilities found by 38.7–60.4% at equal cost across 33 OSS-Fuzz projects; even after the agent was warned, the reduction remained at 37.2%.

9 November 2026
Checking LLM Hallucinations via SQL-Based Source Representation Without Fine-Tuning
News

Checking LLM Hallucinations via SQL-Based Source Representation Without Fine-Tuning

The study examines whether an LLM can apply a basic symbolic skill—building a SQL database from source documents—to ground its outputs in sources without supervision when detecting hallucinations. On RAGTruth and DiaHalu, this neuro-symbolic pipeline proves more accurate than direct prediction and comparable to state-of-the-art methods, without requiring domain-specific fine-tuning.

9 November 2026
MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals
News

MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals

A diagnostic suite of tasks spanning 13 undergraduate and graduate mathematics topics evaluates machine-proving systems in Lean 4 not only by their final results, but also by their subject-matter mastery, the quality of their informal reasoning, and their robustness to rephrased conditions. A component-by-component analysis shows that translating a problem into formal form remains the main obstacle, and that success depends heavily on the area of mathematics and how the problem is framed.

9 November 2026
ReproAgent: AI agent builds working code from a research paper using an "implementation contract"