CANDOR: a metric that separates encoder weakness from simple class imbalance

14 September 202630 views

Frozen encoders are usually compared by the quality of a lightweight head on top of their features, while the geometry of the representations itself remains unassessed. The new CANDOR measure equalizes neighbor banks in size, so that the reference point coincides with a fair coin, and then the picture changes: there are no blind models, but markedly weak ones exist almost everywhere.

CANDOR: a metric that separates encoder weakness from simple class imbalance

Choosing an encoder by the wrong metric

A frozen encoder is usually brought into a project based on one thing: how well a lightweight head — a linear layer, a small MLP, something along those lines — trains on top of its representations. If a simple head can extract the needed feature, the encoder is considered a good one. But this evaluation answers the question "is it convenient to read something out of these embeddings," not the question "does the geometry of the space itself separate the feature." These are two different questions, and confusing them is costly: the head masks the weakness of the representation as long as the tasks are simple and there are few classes.

The balance problem deserves separate mention. As long as the classes are roughly equal, the difference between "the encoder can" and "the head pulls it out" is almost invisible. As soon as one class becomes rare, nearest-neighbor-based metrics start to systematically lie.

Why nearest-neighbor discordance lies

The classic discordance metric is simple: for each point, neighbors are found in the embedding space, and if the nearest neighbor carries the opposite label, the case is recorded as discordant. Within a single class, the geometry works wonderfully — no complaints about the neighbors.

The problem lies in the asymmetry of the procedure. When neighbors for positive and negative examples are drawn from sets of different sizes, the nearest neighbor with the opposite label wins not because the geometry is built that way, but because its class simply lies more densely in the neighborhood. In other words, class prevalence leaks into the result — a statistic of the dataset, not a property of the representation. The effect is unpleasant: a completely uninformed encoder starts to look blind, when in fact it is exactly as useless as a fair coin, and no worse.

What exactly CANDOR changes

CANDOR — Chance-Calibrated Discordance in Frozen Foundation Encoders — redefines the counting procedure itself. Neighbor banks here are drawn at equal size, and the construction of the metric is symmetric with respect to label permutation: swap the positives and negatives, and the value does not change. As a consequence, the chance level is fixed at exactly one half — 50%. Not "roughly around half, depends on the dataset," but a strict reference point at 1/2.

This is not a cosmetic fix. It was precisely the floating reference point that prevented comparing encoders with each other and across datasets: where the imbalance was stronger, chance guessing looked worse than it actually is, and the model received undeserved credit. When the reference point is pinned down, the comparison becomes fair.

And one more detail, important for practice: since the reference point is fixed, CANDOR can be computed before training any head whatsoever. No fine-tuning, no hyperparameter tuning — the metric describes the representations themselves and tells you in advance which findings a particular frozen encoder supports poorly.

Validation at scale

The authors ran the evaluation broadly: 22 encoders, 20 datasets from 7 subject domains, a total of 605,443 images. Such a scope was needed precisely to separate the effect of the metric from the peculiarities of a single task.

The result overturns the conclusion the old procedure led to. After the correction, representation collapse turns out to be below the chance level almost everywhere. Read it this way: none of the tested encoders is blind — information about the feature is present in the representations — but all of them are weak. The phrasing "all weak, none blind" sounds softer than the previous one, and at the same time describes reality far more accurately.

Clinical example: a strong head with weak geometry

The most vivid illustration is medical imaging. The best model for the chest reads pneumothorax with an AUROC of 84.5. A solid number, and it is usually what people look at when choosing a model. But the same model places 18.4% of positive cases closer to an image with the opposite label than to its own image with the same finding — within the same hospital, that is, with no domain shift as an explanation.

The gap between "the head solves the task well" and "the geometry confuses positives and negatives" is exactly that discrepancy. The head compensates for a weak representation; the CANDOR metric shows that there is a lot to compensate for.

Next comes the comparison across domains. The same encoder distinguishes bird species at a level of 4.5 (that is, almost perfectly), while it leaves chest findings at 42.8 and glaucoma at 49.8. The last number sits at the chance level, and in essence — worse than random weights. One encoder, one training run, three completely different qualities of geometry.

A deficit of selection, not of information

A natural question follows: so the representations are hopeless? Not quite. Chance bounds the normalized margin of any Lipschitz head — that is, a head with bounded sensitivity to input perturbation. However, within a set of eleven heads there is one that is right in all cases except 2.8%, whereas a single head misses in 35.9%. Information about the feature is present in the representation — it is simply not equally accessible to different reading devices. So the deficit is related to selection, not to a lack of information.

An interesting association turned up elsewhere too: erasure retention is linked to collapse. But no link was found with training objective, model scale, release recency, or finding size. Put simply, "grab a bigger and newer model" does not solve the problem — it is not about size or age.

What follows from this in practice

Three practical takeaways.

  • Compute CANDOR before training the head. It is a cheap check that shows whether an encoder supports a particular rare finding, and lets you filter out an unsuitable backbone before you spend time on tuning.
  • Don't confuse the head's AUROC with the quality of the representation. 84.5 with 18.4% discordant positives is a normal, not rare, situation, and it is worth measuring separately.
  • Don't expect scale and novelty to fix everything. No associations were found with size, training objective, or recency; the weakness of frozen encoders looks systemic.

The main value of this story is not even the metric itself, but the overturned conclusion. Previously it seemed that encoders were "blind" to certain classes. After chance calibration, something else becomes visible: they see almost everywhere, but everywhere weakly. This is a far less dramatic and far more workable picture — with it, it is clear what to fix and in what order.

Frequently asked questions

Related materials

All materials
Checking LLM Hallucinations via SQL-Based Source Representation Without Fine-Tuning
News

Checking LLM Hallucinations via SQL-Based Source Representation Without Fine-Tuning

The study examines whether an LLM can apply a basic symbolic skill—building a SQL database from source documents—to ground its outputs in sources without supervision when detecting hallucinations. On RAGTruth and DiaHalu, this neuro-symbolic pipeline proves more accurate than direct prediction and comparable to state-of-the-art methods, without requiring domain-specific fine-tuning.

9 November 2026
MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals
News

MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals

A diagnostic suite of tasks spanning 13 undergraduate and graduate mathematics topics evaluates machine-proving systems in Lean 4 not only by their final results, but also by their subject-matter mastery, the quality of their informal reasoning, and their robustness to rephrased conditions. A component-by-component analysis shows that translating a problem into formal form remains the main obstacle, and that success depends heavily on the area of mathematics and how the problem is framed.

9 November 2026
AffordanceWAM: Affordances as an intermediary between scene prediction and manipulator control
Research

AffordanceWAM: Affordances as an intermediary between scene prediction and manipulator control

The work describes a generative model that jointly predicts future RGB observations, scalar fields, affordance heatmaps, and continuous robot commands, using human videos to broaden interaction diversity without action annotations or retargeting. On RoboCasa, CALVIN ABC→D, and real-world tasks, the approach delivers consistent gains over RGB-only and robot-data-only baselines.

9 November 2026
CANDOR: a metric that separates encoder weakness from simple class imbalance