Omanic: a step-by-step breakdown of exactly where LLM logic breaks down

20 September 202621 views

A new open benchmark from Omanic proposes evaluating a model's reasoning abilities not only by its final answer but also by each intermediate step. It is built on roughly 10,300 synthetic training examples and 967 test questions labeled manually by experts.

Omanic: a step-by-step breakdown of exactly where LLM logic breaks down

The final answer as the only metric — and what it hides

Most benchmarks for language models are structured like a school test: there's a question, there's a reference answer, there's a match check. Computing such a metric is convenient, and comparing models is too. But this scheme has a built-in flaw: it evaluates the result, not the path to it.

Imagine a task where you need to connect four facts. The model gets confused on the second, guesses correctly by chance on the fourth — and gets the point. Or the opposite: three links are built flawlessly, but the final answer differed by a single phrasing — and there's no point. In both cases, the final number lies about what actually happened inside. This is especially painful for multi-step questions: there, it's not a single skill being tested, but a sequence of them — finding information, holding onto it, connecting it with what follows, and not losing it along the way.

Diagnosing "exactly where it broke" is not an academic nitpick. What you fix depends on it: the data corpus, the training strategy, or the wording of the prompt. Final accuracy cannot answer that question by definition.

What Omanic is

It is precisely this blind spot that the work "Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models" closes (arXiv:2603.16654, listed under cs.CL, also cross-listed to cs.AI and cs.LG, accepted to EMNLP 2026 Findings). The author team is large and international: Xiaojie Gu, Sherry T. Tong, Aosong Feng, Sophia Simeng Han, Jinghui Lu, Yingjian Chen, Yusuke Iwasawa, Yutaka Matsuo, Chanjun Park, Rex Ying, Irene Li. The first version was posted on March 17, 2026, followed by two revisions — in May and in August.

Omanic is an open-domain benchmark with four reasoning steps. The word "open-domain" is key here: the model doesn't pick an option from a ready-made list and doesn't dig through a single pre-prepared document. The information has to be found independently, and only then built into a chain. This setup is closer to real-world scenarios, where the answer almost never lies in a single paragraph.

Two corpora instead of one

Inside Omanic live two different datasets, and they shouldn't be confused.

The first is OmanicSynth, 10,296 machine-generated examples. This is training material: it can be used as supervision, and it was precisely on it that the authors tested whether the ability to reason step by step is transferable.

The second is OmanicBench, 967 examples that underwent expert review and come with manual annotation. This is the evaluation part, and it is substantially smaller — which makes sense, since human annotation at the step level is expensive.

This split is not a formality. It allows separating "the model was trained" from "the model actually can": the training corpus is large and machine-made, the validation one is compact and vetted.

How the step-by-step annotation is structured

The main difference between Omanic and conventional QA sets is that each question is broken down into components. Single-step sub-questions, intermediate answers, structured graph topologies — that is, a scheme of exactly how the facts are connected to each other.

This turns evaluation from a single point into a set of checkpoints. Now you can look not only at "did the model answer correctly or not," but also at "at exactly which transition did it go off the rails." The graph structure matters here also because different types of connections are learned with varying ease by models: pulling out a property of an object is one thing, tracing a chain of four consecutive dependencies is quite another.

Three diagnoses visible only at the steps

Experiments with closed and open models yielded three observations that the final metric simply cannot show.

The bottleneck at later steps

The authors call the first finding the later-hop bottleneck. It's about the main losses accumulating not at the beginning, but closer to the end of the chain. Models get through the first and second transitions relatively confidently, but on the third and fourth they start to fall apart.

The explanation suggests itself: the longer the chain, the more intermediate entities need to be held in working state simultaneously. At later steps, the model has to operate not on the original question but on the results of its own previous inferences — and any inaccuracy there multiplies. The practical takeaway is unpleasant for those who like building long multi-stage pipelines: adding a fifth and sixth step can cost more than it seems.

The "floor" of factual knowledge

The second phenomenon is the factual knowledge floor. Some errors are not related to logic at all: the model simply doesn't have the needed fact and, instead of honestly stopping, keeps building reasoning on empty ground.

This is an important distinction. When a model errs in its inference — that's a question about reasoning. When it doesn't know the initial fact — that's a question about data and about whether it can admit ignorance. The latter is hard to cure: neither step-by-step reasoning nor a smarter prompt will help if there's simply no support under the first link.

An error that travels along the chain

The third diagnosis is error propagation along the chain. An early inaccuracy doesn't stay local: it gets picked up by subsequent steps and turns into a confident, smoothly worded, but incorrect final answer.

Here lies a trap for evaluation. A model that erred once and then reasoned consistently, and a model that fell apart at every step, look identical in the final metric — both got it wrong. Step-by-step analysis distinguishes them, and this is exactly the case where diagnostics are worth more than the final score.

Transfer learning: 7.41 points across six benchmarks

A separate question is whether this annotation can be used not only for measurement but also for training. The authors checked: fine-tuning on OmanicSynth yielded improvement on six third-party benchmarks dedicated to reasoning and mathematics, with an average gain of 7.41 points.

A figure that should be read correctly. It's not "the model got smarter at everything" — it's "a skill honed on decomposed chains transfers to other tasks of a similar type." Which in itself says something about the nature of the benchmark: it's not about memorizing specific questions, but about training a procedure — how to break a task into steps and maintain the connection between them.

What this means for practice

A few takeaways that follow from what's described.

Evaluate by steps, not by the outcome. If your system produces multi-step answers, a single final metric is not enough — it will average failures of different natures into one indistinct number.

Separate "I don't know" from "I inferred incorrectly." These are two different defects with different remedies, but in conventional reporting they merge.

Be careful with long chains. Later steps are the most vulnerable spot. Sometimes it's wiser to split a task into several independent subtasks than to drive one long chain to the end.

Decomposition is also a training signal. The result with transfer to six third-party sets shows that step-by-step annotation works not only as a ruler but also as training material.

What Omanic is not

It's useful to mark the boundaries. Omanic is not a universal assessment of a model's intelligence and not a test of everything at once. It's a tool for a specific task: to show at which link of reasoning a failure occurs in multi-step open-domain questions.

The size of the evaluation part — 967 examples — is also worth keeping in mind: the set is vetted, but not gigantic. The training part is an order of magnitude larger and machine-created, which provides scale but doesn't replace manual verification.

And most importantly: a diagnosis by itself cures nothing. Knowing that a model stumbles at later steps is a starting point, not a solution. After that comes ordinary work: where to add data, where to simplify the chain, where to teach the model to stop and say "I don't know this."

The authors have released the data and code openly; the work is available via DOI 10.48550/arXiv.2603.16654.

Frequently asked questions

Related materials

All materials
Omanic: a step-by-step breakdown of exactly where LLM logic breaks down