149 edits from real labs: a new benchmark tests whether LLMs can rewrite protocols

15 September 202626 views

BenchBench-Protocol challenges models not with abstract questions but with reconstructed episodes from practice: a scientist adapted a published methodology for their experiment — and the model must retrace their line of reasoning. Of the nine systems tested, the best result was 59.2% on the rubric, while the rest fell within the 34–47% range.

149 edits from real labs: a new benchmark tests whether LLMs can rewrite protocols

A protocol no one follows to the letter

A lab protocol is rarely followed literally. Between publication and the actual experiment there is almost always adaptation: a different cell line, a substituted reagent, fewer samples, someone else's instrument, a trimmed budget. For a person at the wet bench this is routine, but a treacherous kind of routine — any edit drags a chain of consequences behind it. Change the reagent volume at an early step, and you have to remember how that will play out at the wash stage and how it squares with what was already done before. It is precisely this ability — to hold the entire protocol in your head and modify it meaningfully — that the new task set tests.

The study came out under the title BenchBench-Protocol in the arXiv preprint arXiv:2608.23898v1 (cs.AI), submitted on August 24, 2026. The authors are Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone, and Nithin Parsan. It runs 21 pages with 15 figures, so the material is more methodological than promotional.

How 149 tasks were assembled from draft edits

The mechanics of BenchBench-Protocol are unusual. The authors did not sit down to write questions — they took an existing trail of work. Every published protocol had a version that a scientist had edited for their specific experiment. The difference between those two documents became the task: the model is shown the original protocol and the conditions of the new experiment, and it has to propose a correct modification.

An important detail follows directly from this approach: the task does not have an abstract "correct answer" but a weighted rubric. That is because the edit made by a real researcher is known — it can be broken down into elements and scored by how closely the model's proposal matches the real decision in meaning rather than in wording. This removes the perennial problem of biomedical tests, where a model can give a reasonable answer that does not match the reference and score zero.

The foundation turned out to be solid: 96 original protocols from nine areas of wet-lab biology. Only those tasks that passed expert review with high ratings made it into the final set — the rest were filtered out. The result is those very 149 assignments.

Why this is not just another set of expert-written questions

The authors spell out the context separately. In recent years biomedical benchmarks have indeed shifted toward open-ended tasks scored by rubric — that is progress. But the tasks themselves are still most often invented by experts: they sit down and write questions based on their own experience. The limitation of this approach is that the expert answers a question they formulated themselves, implicitly tuning it to their own sense of what is difficult.

Here the logic is reversed. A task arises where a real person has already run into a real problem and spent time solving it. This does not guarantee perfectly clean data, but it does guarantee that the problem was not invented for the sake of a nice phrasing.

Who managed and who did not

Nine models took part in the testing — both closed and open. The spread turned out to be noticeable, though not dramatic.

The best result came from Claude Opus 5 — 59.2% of the normalized rubric score. The other models fell within a range of 34.1% to 47.1%. Put simply, no one reached the leader, but there is no one-and-a-half-times collapse either: all the models can do something, they just do it differently and not always all the way through.

A separate line in the report is the saturation check. The authors gave the models ten attempts each and picked the best one (best-of-10). Even under such a generous scheme the benchmark did not "collapse": the ceiling was not reached. For an evaluation this is good news — it means the task set will not go stale after the next generation of models arrives.

What follows from this in practice

The first conclusion is practical and slightly sobering. A language model cannot yet be fully trusted with protocol adaptation. Even the best result of 59.2% means that roughly four times out of ten the proposal requires human intervention. The model works as a draft, as a generator of options, as a way to quickly check what you may have overlooked. But it does not bear final responsibility for the test tube.

The second conclusion is methodological, and it is more interesting than the first. The authors see their work not only as an evaluation of reasoning in the wet lab but also as proof of a more general idea: real experiments are an underrated source of benchmark tasks. If a lab already has a history of edits, then it also has material for testing models — material, moreover, that cannot be invented at a desk.

The third conclusion concerns where to go next. The gap between 59.2% and 47.1% does not look like a chasm given the difficulty of the task itself. It is more of a hint that the bottleneck is not the volume of a model's knowledge about biology but its ability to build a chain: to account for previous decisions and predict where they will lead in the following steps. It is this quality, rather than fact memorization, that the set of 149 edits is devoted to.

Frequently asked questions

Related materials

All materials
Validating Candidates Costs More Than Generating Them: Honeypots Against LLM Vulnerability Hunting
News

Validating Candidates Costs More Than Generating Them: Honeypots Against LLM Vulnerability Hunting

The study examines the cost asymmetry in agentic vulnerability discovery: generating candidate versions is cheap, while confirming them through reachability analysis, execution, and PoC development consumes most of the budget, making verification a potential standalone target for defense. The proposed RedHerring—repository-adapted decoys with a hidden certificate—reduced the number of real vulnerabilities found by 38.7–60.4% at equal cost across 33 OSS-Fuzz projects; even after the agent was warned, the reduction remained at 37.2%.

9 November 2026
Checking LLM Hallucinations via SQL-Based Source Representation Without Fine-Tuning
News

Checking LLM Hallucinations via SQL-Based Source Representation Without Fine-Tuning

The study examines whether an LLM can apply a basic symbolic skill—building a SQL database from source documents—to ground its outputs in sources without supervision when detecting hallucinations. On RAGTruth and DiaHalu, this neuro-symbolic pipeline proves more accurate than direct prediction and comparable to state-of-the-art methods, without requiring domain-specific fine-tuning.

9 November 2026
MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals
News

MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals

A diagnostic suite of tasks spanning 13 undergraduate and graduate mathematics topics evaluates machine-proving systems in Lean 4 not only by their final results, but also by their subject-matter mastery, the quality of their informal reasoning, and their robustness to rephrased conditions. A component-by-component analysis shows that translating a problem into formal form remains the main obstacle, and that success depends heavily on the area of mathematics and how the problem is framed.

9 November 2026
149 edits from real labs: a new benchmark tests whether LLMs can rewrite protocols