Why the "Control Layer" Is Dangerous for LLM Agents
Modern language models rarely work alone. More often, they are deployed inside an agent harness — a layer that gives the model access to tools, extensions, long-term memory, permissions, and external actions. It is this layer that decides what the model can and cannot do.
The problem is that existing security benchmarks usually test only individual attack scenarios or a limited set of operating conditions. This makes it hard to understand at which exact stage of the harness's operation a failure occurs: during setup, at the moment of a tool call, or during recovery after an incident. Researchers proposed HarnessRisk — a benchmark that looks at agent harness security as a whole, across all stages of its lifecycle.
Six Phases of the Harness Lifecycle
The authors of the study broke down harness security into six operational phases:
- Harness Configuration — setting up parameters and access rights;
- Capability Extension — connecting new capabilities and extensions;
- Runtime Operation — executing tasks in real time;
- State Persistence — saving and restoring state;
- Action Control — controlling actions and tools;
- Incident Recovery — responding to failures and recovering after incidents.
Each phase has its own area of responsibility, and vulnerabilities can manifest in any of them. It is precisely this division that makes it possible to compare how different harnesses handle attacks under identical conditions.
What's Inside HarnessRisk: Scenarios and Metrics
The benchmark contains 128 isolated scenarios. Each scenario is cleverly designed: the model is given a safe user task, but inside an untrusted workflow artifact, a hostile instruction is hidden. The model must accomplish the legitimate goal without succumbing to the attack embedded in the artifact.
Each trajectory is evaluated across four metrics:
- Utility — how well the original task was completed;
- Attack Success Rate — the share of successfully executed attacks;
- Persistence — how long the attack's effect lasts;
- Detection — how effectively the system notices risks.
This approach makes it possible to see not just "was it compromised or not," but also the cost of security to model utility.
What the Testing Showed
Experiments were conducted on three harnesses, six language models, and 14 model-harness combinations. The results were uneven: attack success rates ranged from 12.6% to 80.9%, while Utility stayed within a range of 75.0% to 97.6%. In simple terms, the same model can be nearly invulnerable in one configuration and outright weak in another.
The most vulnerable phase across all three harnesses is Harness Configuration. Attacks often succeed not because of complex exploits, but because within a "permitted" workflow, parameters that affect security can be changed. This is an alarming signal: even a correctly configured harness can allow an attacker to change the rules of the game.
A curious result also relates to risk detection. Some configurations detect attacks in more than 90% of runs, yet still allow a significant share of successful attacks. In other words, the model "understands" that something dangerous is happening, but does not turn that understanding into safe behavior. Awareness of a threat by itself does not protect the system.
Conclusions: Security Cannot Be Measured "on Average"
The main lesson of HarnessRisk is that the security of agent systems depends heavily on the specific model-harness combination. Averaged assessments like "the model is safe" or "the harness is safe" say little unless the configuration in which they operate is taken into account.
Agents need to be evaluated across multiple harness responsibility areas at once — from initial configuration to post-incident recovery. And separately, it is necessary to check how the model behaves under an attack embedded in a routine work artifact. Only this way can weak spots be noticed that remain out of sight in classic benchmarks.
