Why break down pipeline errors at all
Evaluating a RAG system with a single number is convenient, but almost useless. An end-to-end metric answers the question "did the answer match expectations," not the question "why did it or didn't it." Meanwhile, RAG is not a monolithic block but a chain: first something is retrieved from the database, then the system decides whether what was found is enough, and only then does the generator write the text. A failure at any link produces an equally "wrong" answer at the output, even though the causes of two such failures can be directly opposite.
This is precisely the problem tackled by the paper The RAT: A Unified Bayesian Model for RAG Evaluation (arXiv:2608.24753, cs.CL section, submitted August 25, 2026). The authors — Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak, and Jan Deriu — propose not measuring the pipeline as a whole, but describing it as a probabilistic model where each stage is a separate random variable with its own connections. Evaluation of the system then becomes an inference problem: which values of the latent variables most plausibly explain the observed answers and annotations.
What exactly the Bayesian framework models
The key idea is factorization along the information flow. Variables are introduced not mechanically "by pipeline component," but in the order in which information actually moves through the system: retrieval success influences how the generator should behave, and the generator's behavior together with the retrieval outcome determines the final correctness of the answer. This structure makes dependencies explicit rather than hidden inside an aggregated score.
This is how the The RAT model gets three substantive layers that are usually merged into a single metric.
Task success and generator success are different questions
The first layer is task success: did the user ultimately get the correct answer. The second is generator success: did the generator behave appropriately given what it received as input. Formally, the second is conditional quality: how adequate the model's behavior is to the given retrieval outcome, rather than in general.
The difference is fundamental. A system can produce a correct answer despite poor retrieval — the generator guessed from parametric memory. Formally this is task success, but the behavior was incorrect: the system answered without relying on sources, and on another query this habit will result in a hallucination. And vice versa: retrieval found everything needed, the generator answered carefully — yet the result is still wrong because the question was misunderstood. An end-to-end metric does not distinguish these two cases; a conditional one does.
Abstention is not a failure but a decision
The third element of the model is abstention behavior. In a RAG context, refusing to answer is sometimes the only correct reaction: if there is nothing relevant in the database, an honest "I don't know" is better than confident fabrication. But the same refusal becomes an error when the needed document was retrieved and the system failed to use it.
That is why abstention cannot be evaluated in isolation from the retrieval outcome — only conditionally. The The RAT model accounts for this directly: it looks at whether the fact of refusal or answering is consistent with what was actually in the context. This view is also useful for product decisions: if the system abstains en masse when retrieval succeeds, the problem is not in the generator but in how the context is conveyed to it.
Why marginal metrics deceive
The authors applied the framework to 27 configurations — three datasets, three retrievers, and three generators, iterated through all combinations. The full sweep is not an end in itself here: it shows how the decomposition behaves when exactly one link changes while the others stay the same.
The result this was all for: conditional decomposition reveals noticeable behavioral differences between systems that look equivalent by marginal metrics. Two pipelines can show the same share of correct answers and still diverge by tens of percent in where exactly they fail — in retrieval, in the abstention policy, or in generation. For someone choosing a configuration for a product, these are different systems; for someone looking at a single number in a table, they are the same.
Where to spend the annotation budget
A separate part of the work is devoted to allocating annotations. Annotation is expensive, and it is natural to ask: if people can be asked only one question about each example, which question should be chosen?
The authors' answer: annotations about retrieval success are more informative than annotations about task success if the goal is to judge the system's policy adherence. The reason is not convenience but the structure of the model: retrieval annotation lands on a variable at the beginning of the causal chain, so it also "illuminates" the downstream layers. Annotation of final success is about a variable at the output, and it says far less about what happened inside. The authors give an information-theoretic explanation for this asymmetric effect.
The practical takeaway is simple: if the budget is limited, annotators should be asked not "is the answer correct" but "was the needed item found." The former sounds nicer in a report; the latter is more useful for diagnosis.
LLM-as-a-judge as a noisy observation
The third part is extending the model to automatic evaluations. The The RAT scheme allows LLM-as-a-judge verdicts to be incorporated not as a replacement for humans but as calibrated noisy observations: the judge has its own probability of erring, and it is estimated within the same probabilistic model.
This removes the artificial opposition of "expensive human annotation versus cheap automatic annotation." In a single model, one can hold a small corpus of expert judgments that set the scale and calibration, and a large stream of automatic evaluations that refine the parameters. The judge stops being an oracle and becomes just another source of signal — with a known and, importantly, measurable error.
What to take into your own practice
- Don't treat the pipeline as a single node. As soon as the report has a separate metric for retrieval, a separate one for generator behavior, and a separate one for abstention, debates about "what broke" turn into diagnosis rather than a guessing game.
- Evaluate behavior conditionally. Answer correctness and behavior appropriateness given the context are different questions, and a good outcome does not justify a bad process.
- Build reports by slices, not by averages. Two systems with the same end-to-end quality may require completely different fixes.
- Choose what to annotate. If human resources are limited, annotating early stages yields more information about the system as a whole.
- Keep automatic evaluations on a leash. An LLM-as-a-judge is useful as an observation with an error model, not as the ultimate truth.
The Bayesian view is valuable here not for the mathematics as such but for the discipline of thought: it forces you to name in advance which quantities are latent, which are observable, and how one relates to another. After that, any table of results stops being a set of numbers and becomes a description of how the system makes decisions.
