The CHOIR Method: Free-Form Listing Reveals Divergent Model Positions Within Collective Systems

29 October 20262 views

The authors propose CHOIR, an approach in which ensembles repeatedly request ranked lists of answers and then compare the salience of merged concepts across models, prompt phrasings, and role conditions. The study shows that high surface similarity among open-ended answers does not necessarily indicate consensus: base-model identity remains the most stable distinguishing factor, while persona settings shift the concepts expressed.

The CHOIR Method: Free-Form Listing Reveals Divergent Model Positions Within Collective Systems

What CHOIR offers

CHOIR stands for Collective Hierarchically-Ordered Inquiry Responses. The approach adapts free-list elicitation for LLM ensembles. Its goal is to uncover the positions where models within an ensemble diverge. The surface-level agreement between responses hides these differences.

The method works with lists, rather than ready-made formulations. Repeated prompts provide material about which topics the models raise on their own. This material is then brought into a comparable form.

Practical criterion: use CHOIR when ensemble responses look alike but you need to understand exactly where the models differ.

Method workflow

The method consists of three steps.

  1. Repeatedly request ranked lists from ensemble models.
  2. Combine list items into concepts at the prompt level.
  3. Measure the salience of concepts across models, prompt variants, and persona conditions.

Each step adds a dimension. Lists provide raw material, combining them makes it comparable, and measuring salience reveals differences. Without mapping items to concepts, comparison remains at the level of individual words.

Practical criterion: evaluate the method by whether it preserves differences after combining items into concepts.

Evaluation datasets

DatasetWhat it isWhat it showed
Infinity-Chat 100An external prompt set from a recent study on the homogeneity of open-ended model responsesReproduces high surface-level agreement: 93/100 prompts are above chance. Distinguishes narrow prompts from broad ones, which reveal hidden response depth
Targeted diagnostic set27 questions designed to reveal differences at the mechanism levelServes for targeted checks of divergences between models

The datasets serve different purposes. The external set shows the overall picture of agreement while also distinguishing broad prompts from narrow ones. The targeted set is designed to identify differences at the mechanism level.

Selection criterion: use the external prompt set for an overview of agreement, and the diagnostic set to analyze mechanisms.

What differences can be identified

The findings fall into three groups.

  • Base-model identity remains the strongest recoverable signal. This holds for both targeted checks and the external prompt set.
  • Persona prompts shift the concepts expressed within base-model signatures.
  • A source-blind ranking module identifies rare but stable candidates for further study.

Personas change how concepts are expressed, but do not override the base-model signal. Source-blind ranking works as a filter for further analysis.

Practical criterion: establish the base model first, then assess the shift caused by the persona.

A diagnostic framework instead of a statement of agreement

The authors propose treating the homogeneity of open-ended responses as a diagnostic task. The question then takes a different form: where do models converge, why do they converge, and what else can be uncovered?

  • Where they converge — a map of shared concepts across models.
  • Why they converge — the reasons behind surface-level agreement.
  • What else is visible — candidates identified by source-blind ranking.

The paper was published in Proceedings of the Third Conference on Language Modeling (COLM 2026), 2026. It was submitted to arXiv on September 29, 2026. The authors are Ben Wigler and Maria Tsfasman.

Criterion: response homogeneity is not a final conclusion, but a starting point for diagnosing depth.

Frequently asked questions

Related materials

All materials
AURA-Eval: How to Check Whether an LLM Agent Recognized a Risk and Changed Its Next Step
News

AURA-Eval: How to Check Whether an LLM Agent Recognized a Risk and Changed Its Next Step

The framework evaluates LLM agents’ behavior not by their final verdict, but by how they respond to risk at key decision points: tool-call trajectories are interrupted, after which the model must continue reasoning and acting, while a panel of LLM judges assesses risk detection, the nature of the step, and its safety. Across 1,249 tasks drawn from 157 R-Judge trajectories, unsafe actions become more frequent among 20 models when no safe path is available, and removing oversight before execution turns 52.3% of initially safe cases into unsafe ones.

10 November 2026
Chern–Simons Reservoir Memory: Autonomous Connectivity Dynamics and Density Transport
News

Chern–Simons Reservoir Memory: Autonomous Connectivity Dynamics and Density Transport

The article explores how a reservoir preserves the history of input signals both in its density distribution and in its geometric degrees of freedom. A comparison of four mechanisms shows that current creates a separate connected component that alters the subsequent density response, while accounting for all measured features yields a small improvement in decoding pulse order.

10 November 2026
STAR: Adaptive Normalization of Temporal Context and Service Graphs for End-to-End Incident Management
Research

STAR: Adaptive Normalization of Temporal Context and Service Graphs for End-to-End Incident Management

The STAR framework trains representations from metrics, logs, and traces, using learnable normalization that accounts for multiscale temporal dynamics and dependency structures between services. On two real-world datasets, this pipeline delivers consistent improvements in anomaly detection, failure ranking, and root cause analysis over state-of-the-art baselines.

10 November 2026
The CHOIR Method: Free-Form Listing Reveals Divergent Model Positions Within Collective Systems