Freshness vs. usefulness: CausalCache rethinks screenshot selection rules in GUI agent memory

14 September 202611 views

Instead of allocating all visual slots to the most recent actions, the method evaluates the entire history and returns archived frames to those events where they will yield the most benefit. On OSWorld-Verified, enabling old screenshots adds about 13 percentage points to success relative to memory from summaries alone, and zero-shot transfer to MobileWorld raises the result from 30.2% to 36.8%.

Freshness vs. usefulness: CausalCache rethinks screenshot selection rules in GUI agent memory

The budget that always goes to fresh frames

A long-lived GUI agent is built like a creature with a poor memory for images. It can drag its textual action history along almost indefinitely — summarization weighs little. Screenshots, however, are expensive: only a handful of images fit into the model's active window. Hence the task the authors call budgeted fidelity restoration. Every event is kept as a compressed description, but a fixed budget B decides which events get their archived pixels back.

The baseline Recent-B answers this question in the simplest way: all visual slots go to the most recent events. The logic of "the fresher, the more useful" looks natural, but it has a blind spot — it never checks whether a recent screen is really more important than one from twenty steps ago. Sometimes the user switched to another window, configured something there, came back — and the thing most needed for the next action was left far back in the history.

A different selection principle

CausalCache is a method described in the paper arXiv:2608.22577 (Jiaxuan Luo, Zhanfeng Liao, Jiayao Teng, Yuan Wang; v1 dated August 23, 2026, v2 update dated August 25, nine pages and four figures). Instead of the rule "everything to the most recent," it evaluates the entire history and substitutes an older event exactly when its predicted usefulness outweighs that of a recent candidate. The question is posed not as "what happened just now" but as "what will be useful at the next step."

An adapter that knows its place

The second half of the design is a history-gated key/value adapter. It touches only the tokens of restored historical images and switches off entirely when there are no such images in the context. This is an important detail: processing of the current screen does not degrade just because the system is capable of retrieving old frames in principle.

The selector and the adapter are trained with matched-budget interventions on desktop trajectories, and then tested zero-shot on mobile, that is, without any fine-tuning for the new platform.

What the measurements showed

Desktop: there is a gain, but not right away

On OSWorld-Verified, enabling historical screenshots raises success by roughly 13 percentage points compared to memory that has only textual summarizations. That sounds impressive, but the nuances start after that.

At the official limit of 15 steps, CausalCache and the simple Recent-4 are statistically indistinguishable. That is, on short episodes the whole endeavor does not pay off: there the history simply does not have time to accumulate anything valuable, and a novelty-greedy selection works no worse. In the 30-step diagnostic, however, a gap appears — 46.7% versus 42.4%, a gain of 4.3 points.

Mobile: transfer without retraining

Zero-shot on 117 MobileWorld tasks gives 36.8% versus 30.2% for Recent-4. A method trained on desktop trajectories also wins on a phone — this is exactly the result that says it is not about fitting to a specific interface.

More interesting is where exactly the gain settles. It is concentrated on the predefined cross-app memory-candidate split: 30.6% versus 19.4%, that is, +11.2 points. In the single-app controls there was no difference at all — 43.6% versus 42.4%. The picture is consistent: the advantage shows up where a task requires recalling the state of one application while acting in another.

What follows from this

The authors' formulation goes like this: choosing which past events to give pixels back to is more effective than spending a fixed visual budget entirely on recency. It is hard to argue with that after the gain turned out to be concentrated precisely in tasks involving switching between applications.

But an honest interpretation requires two caveats. First: on short horizons there is no difference, which means the method is not a free improvement but a bet on long episodes, where accumulated history makes sense at all. Second: the zero difference in the single-app controls implies that selection by usefulness is not universal — it solves a specific class of problems where the agent needs to bring a long-gone screen back into context.

A practical takeaway for those building agents: recency is a convenient but not the only criterion. As soon as the image budget becomes a bottleneck, it is worth asking what exactly you are putting into it — the latest frames out of inertia, or the ones that will actually be useful at the next step.

Frequently asked questions

Related materials

All materials
Freshness vs. usefulness: CausalCache rethinks screenshot selection rules in GUI agent memory