The problem: memory exists, but the error is hard to find
Persistent memory is considered one of the most important parts of an LLM agent: it is what allows it to work across sessions, recall context, adjust its representations, and adapt to the user. However, a real memory system is not a single component but an entire pipeline of several stages: data intake, retrieval of relevant fragments, filtering, and finally, use during response generation. When the result is incorrect, end-to-end metrics merely state the fact of failure. They do not explain at which stage something went wrong—and this turns memory refinement into blind parameter tuning.
Existing evaluation approaches exacerbate the problem. They typically report average accuracy across a test set, but omit the statistical significance of changes, do not check whether improvements on some slices came at the cost of regressions on others, and leave behind no diagnostic traces from which the chain of reasoning could be reconstructed. As a result, a team can spend weeks on experiments and still never learn which specific intervention led to the improvement or whether something else broke along the way.
The D²ACCI protocol
To address this problem, the researchers proposed D²ACCI—a dual-loop protocol for iterative memory development. The idea is that any change to the system must pass through an external diagnostic "gate": it either advances the change, marks it as experimental, or rejects it outright—based on paired evidence, monitoring of protected slices, and traces that allow the failure to be localized to a specific stage.
The key novelty of the protocol is the DCR metric, which measures "graded observability": the degree to which errors remain localizable given the available data. That is, it evaluates not only the quality of the response but also whether the experiment logs make it possible to understand where exactly the problem arose. For reproducibility, there is also the D²ACCI-Eval artifact, which allows the gate to be re-run on the same data and results to be cross-checked.
How it works in practice
The protocol was applied to the MemStack memory system. It was evaluated on three public benchmarks at once: accuracy reached 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. But far more interesting than the numbers themselves is what was discovered through paired ablations.
Five experiments showed that three components—supplement extraction, session-memory retrieval, and Forget Guard—yield a statistically significant gain of +1.9 to +3.7 percentage points (p ≤ .003 in all cases). The remaining changes were indistinguishable on the aggregate metric. For example, the choice between BM25 and RRF was kept as a managed feature flag—and this difference is simply invisible under standard evaluation, yet it becomes noticeable at the trace level.
Why this changes the development approach
Separately, the authors conducted a diagnostic audit: they compared expert agreement on the root cause of failure when re-labeling based only on results versus using enriched traces. The second option significantly improved the consistency of conclusions. DCR@3, meanwhile, demonstrated a striking difference: diagnostic artifacts achieved 98–100% localizability, whereas logs with results only—0%.
The conclusion is fairly transparent: to reliably improve memory systems, you need evidence that can be traced, statistically substantiated, and checked for regressions. Without this, it is impossible to distinguish a genuine improvement from random luck, or a successful fix from a problem that has simply "moved" from one pipeline stage to another. D²ACCI is an attempt to turn such diagnostics from a manual, labor-intensive process into a standard part of the development cycle.
