Why RL Hits a Wall with Labels
Reinforcement learning (RL) is today considered one of the most promising ways to improve reasoning in language and vision-language models. The logic is simple: the model tries different strategies, gets a reward for correct actions, and gradually learns to reason better. But there's a catch — for the reward to be meaningful, precise external signals are usually needed: verifiable answers, reference solutions, labeled reasoning chains. All of this is expensive to prepare, and for many tasks such data is becoming increasingly scarce.
The self-rewarding approach partially solves the problem, where the model evaluates its own answers and adjusts its behavior based on that. However, this creates a vicious cycle: if the model already has biases or is used to responding suboptimally, it will reinforce the same errors. Training on one's own feedback quickly leads to uniformity: answers become similar to each other, diversity drops, and in the worst case, collapse occurs — the model fixates on a narrow set of patterns and stops developing.
How Co-RL Works
The authors of the preprint arXiv:2608.17253 propose the Co-RL framework, which breaks this vicious cycle. Instead of a single model learning from its own evaluations, Co-RL simultaneously trains several fully decoupled models — they share no common parameters and do not peek at each other's weights. Each model receives rewards not from itself, but from other members of the cohort (peers). Essentially, the models reason, critique, and evaluate each other, and training is built precisely on these "external" evaluations.
This exchange of feedback fundamentally changes the dynamics. Errors of different models rarely overlap completely — each has its own weaknesses, thinking patterns, and biases. Therefore, when one model evaluates another's answer, the likelihood that it will "confirm" someone else's systematic error is significantly lower. This helps avoid self-reinforcing feedback loops that break the self-rewarding approach.
Diversity as the Key Ingredient
The core idea of Co-RL is that success directly depends on cohort diversity. If all models are identical, they will simply agree with each other and no advantage will be gained. Therefore, the authors propose three levels of heterogeneity:
- Different model families — for example, combining architectures from different vendors so they don't inherit common errors.
- Different sizes — large and compact models see the task differently, and their evaluations complement each other.
- Paraphrased training samples — the same examples are presented in different formulations, reducing the correlation of errors caused by identical task phrasing.
This diversity of the cohort reduces correlated errors, which are dangerous because they accumulate from iteration to iteration. At the same time, behavioral diversity is preserved — models do not collapse into a single reasoning style but continue exploring different approaches.
What the Experiments Showed
The authors tested Co-RL on text and multimodal tasks, comparing it with baseline models and label-free approaches without access to ground-truth labels. On seven text benchmarks for LLMs, the average improvement was 3.0–8.6%, and on four multimodal benchmarks for VLMs — 2.3–7.2%. Moreover, Co-RL not only outperforms similar label-free methods but in some cases matches or even surpasses supervised learning — even though it requires no labeled data at all.
This is an important signal for the industry: perhaps the future lies not in endlessly collecting expensive annotations, but in properly organized interaction between models. The more agents participate in evaluating each other and the more they differ, the more robust and higher-quality the training becomes.
Practical Takeaways
Co-RL looks like a convincing step toward truly self-learning systems. It requires no labeling, does not burn out on its own feedback, and maintains high reasoning quality. The developers note that the framework's code is available to the community — meaning the approach can be reproduced and adapted to your own tasks today.
Of course, questions remain: for example, how to optimally select the cohort composition for a specific task, and whether training several models at once is too expensive. But the idea itself — training models not in isolation but in a collective where each evaluates others — opens up an interesting direction for research. Perhaps it is precisely such mechanisms that bring us closer to systems that learn to reason as naturally as people learn from each other's experience.
