Short and to the point
Can a modern language model replace a live reviewer at a machine learning conference? Researchers tested this on real ICLR materials — one of the most prestigious venues in the field. In a paper published on arXiv (number 2608.03659), Abraham Camelo-Guerrero and Jairo Diaz-Rodriguez compared how three well-known models handle the evaluation of scientific submissions against human decisions.
The conclusion is twofold: the algorithms do a decent job of separating "accepted" papers from rejected ones, but they completely miss finer distinctions, such as the difference between an oral presentation and a poster. At the same time, each model shows its own signature: one gives generous scores, another is overly strict with strong papers. Machine reviewers also focus on completely different weaknesses than human experts.
In short, using LLMs as a rough filter is possible, but fully trusting them with the final decision is premature for now. Below, we break down the details of the experiment and its findings.
How the experiment is set up
Data and conditions
The authors took 300 submissions to ICLR 2026 and balanced the sample across three categories: oral presentations, posters, and rejected papers — a hundred of each. This approach guarantees that the models are not simply guessing but are actually forced to distinguish all three classes.
Each model — GPT-5.4, Gemini 3.1 Pro Preview, and Claude Opus 4.6 — received the same set of instructions and rating scale. An important detail: information about the human decision was removed from the submission texts in advance. This is necessary to prevent answer leakage and to honestly test the model's ability to make an independent judgment.
Comparison methodology
The researchers looked at the results from three angles. First, they checked how accurately the models' predictions match the final decisions — both in a broad sense (accepted or not) and in detail (oral vs. poster). Second, they studied how the models use the recommendation scale: for example, whether they inflate scores. Third, they compared which weaknesses in the papers are found by humans and which by algorithms.
What the results showed
Distinguishing the fates of submissions
The encouraging news: all three LLMs confidently distinguish accepted papers from rejected ones. This means that model reviews carry a useful signal even without training on conference-specific data. However, the success ends there.
None of the models managed to reproduce the distinction between oral and poster that was obvious in human judgments. For the algorithms, accepted papers looked like a homogeneous group, even though there is a noticeable hierarchy between an oral presentation and a poster. This is an important gap: the fine-grained quality criteria that distinguish outstanding research from merely good work are not yet captured by machines.
Differences across model providers
The models' behavior turned out to be heavily tied to the developer. Gemini 3.1 Pro Preview acted like a classic "lenient" reviewer: its scores were consistently above average. Interestingly, GPT-5.4 and Claude Opus 4.6 were much closer to humans when it came to rejected and poster papers, but at the same time showed increased strictness toward oral submissions.
This bias can be interpreted in different ways. Perhaps the models "nitpick" strong papers because of elevated expectations: it is easier to find a vulnerable spot in a good text than in an obviously weak one. Or perhaps the attention architecture simply does not know how to scale praise: for it, there is no big difference between "good" and "excellent."
Topical divergences
The most curious part is what machine reviewers pay attention to. All three models often pointed to the lack of basic comparative experiments — a classic criticism of scientific work. Human experts, meanwhile, much more often raised questions of computational efficiency: how many resources the proposed method requires, how practical it is.
This is not just a difference in style. It reflects a difference in priorities. Humans think about real-world application, about the cost of training and deployment, while models evaluate the text more as a formal document. So if you use an LLM for a preliminary check of a paper, you should be prepared that some important comments will simply never reach you.
Conclusions
The main lesson of this study: a model's accuracy at the "accepted/rejected" level does not yet indicate its competence in detailed evaluation. The ability to distinguish obviously weak papers from the rest is a fairly coarse filter that does not replace thoughtful reviewing.
The practical application suggests itself: LLMs can be used as primary assistants that mass-filter clearly unsuitable submissions or flag obvious formal flaws. But decisions about the fate of solid papers should remain with humans — if only because the human scale of values (priority of efficiency, assessment of novelty, field context) still differs noticeably from the machine's.
In the long run, it might make sense to calibrate models for specific communities and even for specific reviewers. But until that happens, relying on automated review in scientific communication is at least premature.
