A small but pointed study argues that the way AI systems learn from AI feedback is systematically skewed — and that a calibration fix can substantially reduce the damage.
When an LLM agent improves its behavior through feedback from an evaluator model, any biases in that evaluator quietly bake themselves into the agent's strategy over time. Researchers call this "evaluator preference coupling." Prior work identified and measured the problem; this paper is the first to test whether probability calibration can actually fix it. Using DeepSeek-V4-Pro as the learning agent and GLM5.2 as the judge, the team ran a controlled experiment comparing standard binary training — simple win/loss signals — against confidence-calibrated updates that weight outcomes by probability. Calibration reduced the coupling coefficient by 20-49% and cut Jensen-Shannon divergence by 45-67%.
The finding matters because LLM-as-judge pipelines are everywhere now. As labs increasingly use AI evaluators to train or steer other AI systems, bias compounding becomes a structural risk — one that can be hard to detect because the student model has no ground truth to push back against. A lightweight calibration step inserted into the feedback loop could prevent a class of alignment drift that would otherwise be invisible until it surfaces in behavior.
The experiment ran with only five subjects, which is a notable limitation the authors acknowledge. The calibrated TTRL protocol is public, but production teams should treat this as a promising early signal, not a solved problem.