Choosing an encoder by the wrong metric
A frozen encoder is usually brought into a project based on one thing: how well a lightweight head — a linear layer, a small MLP, something along those lines — trains on top of its representations. If a simple head can extract the needed feature, the encoder is considered a good one. But this evaluation answers the question "is it convenient to read something out of these embeddings," not the question "does the geometry of the space itself separate the feature." These are two different questions, and confusing them is costly: the head masks the weakness of the representation as long as the tasks are simple and there are few classes.
The balance problem deserves separate mention. As long as the classes are roughly equal, the difference between "the encoder can" and "the head pulls it out" is almost invisible. As soon as one class becomes rare, nearest-neighbor-based metrics start to systematically lie.
Why nearest-neighbor discordance lies
The classic discordance metric is simple: for each point, neighbors are found in the embedding space, and if the nearest neighbor carries the opposite label, the case is recorded as discordant. Within a single class, the geometry works wonderfully — no complaints about the neighbors.
The problem lies in the asymmetry of the procedure. When neighbors for positive and negative examples are drawn from sets of different sizes, the nearest neighbor with the opposite label wins not because the geometry is built that way, but because its class simply lies more densely in the neighborhood. In other words, class prevalence leaks into the result — a statistic of the dataset, not a property of the representation. The effect is unpleasant: a completely uninformed encoder starts to look blind, when in fact it is exactly as useless as a fair coin, and no worse.
What exactly CANDOR changes
CANDOR — Chance-Calibrated Discordance in Frozen Foundation Encoders — redefines the counting procedure itself. Neighbor banks here are drawn at equal size, and the construction of the metric is symmetric with respect to label permutation: swap the positives and negatives, and the value does not change. As a consequence, the chance level is fixed at exactly one half — 50%. Not "roughly around half, depends on the dataset," but a strict reference point at 1/2.
This is not a cosmetic fix. It was precisely the floating reference point that prevented comparing encoders with each other and across datasets: where the imbalance was stronger, chance guessing looked worse than it actually is, and the model received undeserved credit. When the reference point is pinned down, the comparison becomes fair.
And one more detail, important for practice: since the reference point is fixed, CANDOR can be computed before training any head whatsoever. No fine-tuning, no hyperparameter tuning — the metric describes the representations themselves and tells you in advance which findings a particular frozen encoder supports poorly.
Validation at scale
The authors ran the evaluation broadly: 22 encoders, 20 datasets from 7 subject domains, a total of 605,443 images. Such a scope was needed precisely to separate the effect of the metric from the peculiarities of a single task.
The result overturns the conclusion the old procedure led to. After the correction, representation collapse turns out to be below the chance level almost everywhere. Read it this way: none of the tested encoders is blind — information about the feature is present in the representations — but all of them are weak. The phrasing "all weak, none blind" sounds softer than the previous one, and at the same time describes reality far more accurately.
Clinical example: a strong head with weak geometry
The most vivid illustration is medical imaging. The best model for the chest reads pneumothorax with an AUROC of 84.5. A solid number, and it is usually what people look at when choosing a model. But the same model places 18.4% of positive cases closer to an image with the opposite label than to its own image with the same finding — within the same hospital, that is, with no domain shift as an explanation.
The gap between "the head solves the task well" and "the geometry confuses positives and negatives" is exactly that discrepancy. The head compensates for a weak representation; the CANDOR metric shows that there is a lot to compensate for.
Next comes the comparison across domains. The same encoder distinguishes bird species at a level of 4.5 (that is, almost perfectly), while it leaves chest findings at 42.8 and glaucoma at 49.8. The last number sits at the chance level, and in essence — worse than random weights. One encoder, one training run, three completely different qualities of geometry.
A deficit of selection, not of information
A natural question follows: so the representations are hopeless? Not quite. Chance bounds the normalized margin of any Lipschitz head — that is, a head with bounded sensitivity to input perturbation. However, within a set of eleven heads there is one that is right in all cases except 2.8%, whereas a single head misses in 35.9%. Information about the feature is present in the representation — it is simply not equally accessible to different reading devices. So the deficit is related to selection, not to a lack of information.
An interesting association turned up elsewhere too: erasure retention is linked to collapse. But no link was found with training objective, model scale, release recency, or finding size. Put simply, "grab a bigger and newer model" does not solve the problem — it is not about size or age.
What follows from this in practice
Three practical takeaways.
- Compute CANDOR before training the head. It is a cheap check that shows whether an encoder supports a particular rare finding, and lets you filter out an unsuitable backbone before you spend time on tuning.
- Don't confuse the head's AUROC with the quality of the representation. 84.5 with 18.4% discordant positives is a normal, not rare, situation, and it is worth measuring separately.
- Don't expect scale and novelty to fix everything. No associations were found with size, training objective, or recency; the weakness of frozen encoders looks systemic.
The main value of this story is not even the metric itself, but the overturned conclusion. Previously it seemed that encoders were "blind" to certain classes. After chance calibration, something else becomes visible: they see almost everywhere, but everywhere weakly. This is a far less dramatic and far more workable picture — with it, it is clear what to fix and in what order.
