Training large language models with reinforcement often looks crude: the system only gets a "plus" or "minus" for the entire response and has to figure out on its own what went wrong. The SocraticPO method, described in the preprint arXiv:2606.09887 by a team led by Zirui Liu, tries to fix this. Instead of simply telling the model it made an error, the "teacher" naturally hints at the direction of reasoning — while the model itself still learns within the standard RL framework.
Why binary rewards are not enough
In a typical RL loop, a language model generates a chain of reasoning and then receives a scalar outcome reward: for example, for matching the correct answer. Such feedback sets an optimization direction but does not explain at which step the solution went down the wrong path.
As a result, the model often looks for a shortcut: it memorizes patterns that look plausible but do not hold up on new tasks. The outcome is fragile policies that work well on familiar types of examples and break at the slightest change in wording. A binary signal is too poor to teach reasoning.
How SocraticPO works
The key idea is to add natural-language guidance in a Socratic style to the rollout. The process looks like this:
- The student answers on its own. The model tries to solve the task without external help.
- The teacher checks the reasoning. If the answer is correct, the rollout ends in the usual way.
- On error, the teacher diagnoses the attempt. It does not give away the ready solution but briefly points out the problem or asks a guiding question in the spirit of a Socratic dialogue.
- The student continues with an extended context. The model sees its own original chain and the teacher's hint, then tries to reach the correct answer.
This approach is fundamentally different from simply giving hints before the answer: intervention happens only after the model has already demonstrated its "naive" reasoning. This means the teacher can rely on the student's actual mistake rather than on an abstract "here is how it should be done."
Why a "decaying" reward
If a corrected answer obtained with the teacher's help were rewarded as generously as a fully independent one, the model would be tempted to deliberately make mistakes in order to always get a hint. That would turn the teacher into a free source of correct answers.
SocraticPO gets around this problem cleverly: correct answers obtained after intervention bring a reduced reward. The model can use the hint but does not receive the full reward for it. This mechanism reduces dependence on external help and forces the model to use the teacher's guidance for learning rather than as a loophole.
Compatibility with existing methods
One of the important advantages of SocraticPO is minimal intervention in the learning algorithm itself. The framework changes only the rollout generation process, while the standard objective — maximizing expected reward — remains untouched. Thanks to this, the method can be plugged into existing policy-gradient approaches, such as Reinforce++.
Another plus is that the teacher does not need access to the model's internals. Since guidance is passed only through text, a stronger model acting as a "black box" can play the teacher role. There is no need to align token distributions or use the student model's logits.
Experimental results
The authors tested SocraticPO on undergraduate-level science tasks from the SciKnowEval benchmark. Here, the method outperformed strong RL baselines and approaches based on self-distillation, where a model trains on its own corrected answers.
Ablations show that two mandatory components work together. Targeted textual hints alone noticeably improve results, while reward decay adds an important safeguard against overfitting to the "helper." Removing either one causes performance to drop.
What this means in practice
SocraticPO is interesting because it makes RL feedback more similar to a natural learning process. The model is not just penalized for an incorrect answer — it is shown the direction in which to think.
At the same time, this "human touch" does not require completely rewriting the RL infrastructure: it is enough to replace the rollout process and leave the rest of the setup intact. The ability to use strong black boxes as teachers is especially useful as the gap between available open and closed models grows.
The question remains how the framework will behave on more complex multi-step tasks that require several rounds of dialogue. But even now, the idea of adding not a score but substantive error explanation to RL looks like a natural next step toward more robust reasoning models.
