Models that work simultaneously with images and text have learned to perform impressively well: they recognize objects, answer questions about pictures, and transfer knowledge to new tasks. However, behind this success often hides a non-obvious weakness: the model memorizes not the true properties of an object, but the accompanying details that serve as "shortcuts." Adversarial prompt tuning is one way to make models more robust, but it also has a pitfall. Let's break down why a model learns the wrong things and how a new approach proposes separating useful and deceptive features.
Why the model learns the wrong things
When a vision-language model is fine-tuned on data with noise or adversarial examples, it seems logical to expect robustness to grow uniformly. In practice, the opposite happens: on already-seen classes, the model shows excellent results, but on new, unseen classes, when adversarial attacks appear, quality drops sharply. This phenomenon is called robust overfitting.
The reason is that the model begins to rely on pseudo-robust features — sets of random patterns that survive an attack but do not reflect the essence of the object. For example, the model may memorize the background texture, characteristic shadows, or specific noise present in the training examples. Such features pass the adversarial perturbation check but do not work on new data. The model overfits not to the content but to shortcut cues — and the longer the training, the stronger the degradation on unseen classes.
How to separate features: the idea of ADAPT
The authors of the proposal start from an unusual philosophy: instead of teaching the model only the right features, you also need to explicitly teach it what not to learn. This is how the ADAPT (Adversarial Disentangled Prompt Tuning) method came about.
The key idea is to use two types of prompts simultaneously:
- Target prompt — responsible for the model extracting robust, generalizable features.
- Decoy prompts — a special pool of additional prompts that deliberately "absorb" pseudo-robust patterns.
During training, the decoy prompts compete with each other: each one tries to capture a certain type of random shortcut. The target prompt is then forcibly made orthogonal to the decoys in the embedding space. This way, robust features are separated from deceptive ones: what the decoys have already "learned" cannot be used by the target prompt.
The mechanics of dual prompts
In practice, this looks like a division of labor. The decoy prompts are not needed for the final prediction — their job is to act as a "trash bin" for inherited distortions. The target prompt, meanwhile, is trained under a constraint: its vector must not align with the direction of the decoys. Orthogonality in the embedding space means that the features used by the decoys do not influence the target prompt's decision. As a result, the model is forced to seek deeper, causal patterns.
Why separation works
Orthogonality is not just a technical trick but a way to provide a theoretical guarantee. Analysis of the method shows that a special loss function limits the impact of shifts in pseudo-robust features on predictions for unseen classes.
In short: even if the decoys collect a lot of extra information, that information cannot "leak" into the target prompt. Consequently, distribution shifts that occur when moving from seen to unseen classes have less impact on the model's error. This approach yields more predictable behavior on adversarial examples and does not sacrifice the robustness of some classes for the sake of others.
What the experiments showed
Practical tests confirm that ADAPT noticeably improves the target prompt's robustness on unseen classes compared to existing adversarial tuning methods. The model does not just memorize protection against specific attacks but forms cleaner features that work on new material.
Of course, completely eliminating false patterns is not yet possible — the task of separating causality from correlation remains one of the hardest in machine learning. But the approach of "teaching what not to learn" provides a practical guideline: sometimes the best way to teach a model is to show it what to ignore.
The main takeaway for practitioners: when tuning vision-language models, it is worth monitoring not only the growth of accuracy on training data but also which features the model relies on. If you add "traps" for random patterns to the training process — for example, through dual prompts with orthogonality — you can get not only a more reliable but also a more honest model.
